Page-configured gesture control methods, electronic devices, and readable storage media
By obtaining configuration information and changes in the user's hand position to determine the target control item, the problem of the inability to dynamically adapt gesture control content is solved, and flexible adaptation between gesture control and the page is achieved.
Patent Information
- Application Number
- CN202611611444.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, gesture control content cannot be dynamically adapted to the current page, resulting in the use of the same fixed control items on different pages, which cannot meet the needs of dynamically changing operations.
By obtaining configuration information, matching the set of control items according to the current page identifier, generating a directional overlay, and determining the target control item based on the spatial position change of the user's hand, the corresponding control command is executed.
It enables dynamic adaptation of gesture control content to the current page, avoiding the use of the same fixed control items on different pages, and improving the flexibility and applicability of gesture control.
Smart Images

Figure CN123114271A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of human-computer interaction technology, and in particular to a gesture control method, electronic device, and readable storage medium based on page configuration. Background Technology
[0002] With the development of intelligent display devices, smart terminals, and human-computer interaction technologies, non-contact gesture control is gradually being applied to scenarios such as page browsing, audio-visual control, and function selection.
[0003] In existing technologies, cameras are typically used to capture image frames containing the user's hand, and these frames are processed using a gesture classification model to obtain the corresponding gesture category. The device typically pre-defines the mapping between gesture categories and control functions, and directly executes the corresponding fixed function based on the gesture category output by the gesture classification model. However, since recognizable gestures and their corresponding functions do not change with the current display page or business scenario, the fixed gesture function mapping is difficult to adapt to the current page when different pages have different executable operations.
[0004] There is currently no effective solution for how to make gesture-controlled content dynamically adapt to the current page in related technologies. Summary of the Invention
[0005] Therefore, it is necessary to provide a page-configured gesture control method, electronic device, and readable storage medium that enables gesture control content to dynamically adapt to the current page, addressing the aforementioned technical problems.
[0006] Firstly, this application provides a gesture control method based on page configuration, including:
[0007] Obtain configuration information, which includes at least one page identifier and a set of control items corresponding to each page identifier; the set of control items includes multiple control items.
[0008] Based on the page identifier of the current page, the corresponding set of control items is obtained from the configuration information, and a directional overlay is generated based on the set of control items;
[0009] Based on the spatial position change information of the user's hand, the target control item in the directional floating layer is determined;
[0010] Execute the corresponding control instructions according to the target control item.
[0011] In one embodiment, each of the control items includes location information and display information;
[0012] The step of generating the directional overlay based on the set of control items includes: generating the directional overlay according to the position information and display information of each control item in the set of control items corresponding to the current page.
[0013] In one embodiment, each control item includes position information; determining the target control item in the orientation overlay based on the spatial position change information of the user's hand includes:
[0014] Acquire multiple consecutive image frames, perform hand detection on each image frame, and obtain the basic gesture recognition result and hand position information corresponding to each image frame;
[0015] When determining the interaction state to enter the selection state based on the basic gesture recognition results corresponding to each image frame, the hand position when entering the selection state is determined as the anchor point position.
[0016] Based on the current hand position in the selected state and the anchor point position, determine the displacement information of the current hand position relative to the anchor point position;
[0017] Based on the displacement information and the position information of each control item in the control item set corresponding to the current page, the target control item is determined from the direction overlay.
[0018] In one embodiment, each control item includes an action identifier;
[0019] The step of executing the corresponding control instruction according to the target control item includes: determining the action processing mode corresponding to the action identifier according to the action identifier corresponding to the target control item; and executing the corresponding control instruction when the action processing mode is the local processing mode.
[0020] In one embodiment, determining the interaction state to enter the selection state based on the basic gesture recognition results corresponding to each of the image frames includes:
[0021] When the first basic gesture is identified based on the basic gesture recognition results of each of the image frames, the interaction state is switched to the ready state;
[0022] Within a preset time after entering the preparatory state, when the recognition result of the basic gesture in the subsequent image frame after the recognition of the first basic gesture is confirmed to be the recognition of the second basic gesture, the interaction state is switched from the preparatory state to the selection state, and the directional overlay is displayed.
[0023] In one embodiment, executing the corresponding control instruction according to the target control item includes:
[0024] Determine whether the target control item meets the preset triggering condition; if so, execute the corresponding control instruction according to the target control item.
[0025] When the target control item remains in the selected state for a duration that reaches a preset dwell time, or when a preset confirmation gesture is recognized during the period when the target control item is in the selected state, it is determined that the preset triggering condition is met.
[0026] In one embodiment, the displacement information includes displacement amount and displacement direction; determining the target control item from the direction overlay based on the displacement information and the position information of each control item in the control item set corresponding to the current page includes:
[0027] Based on the position information of each control item, determine the directional region corresponding to each control item;
[0028] When the displacement is less than the preset dead zone threshold, no control item in the control item set corresponding to the current page will be determined as the target control item.
[0029] When the displacement is greater than or equal to the preset dead zone threshold, the control item corresponding to the direction region where the displacement direction is located is determined as the target control item. When the displacement direction does not reach the preset exit threshold of the direction region corresponding to the target control item, the target control item is maintained. When the displacement direction reaches the preset entry threshold of the adjacent direction region, the control item corresponding to the adjacent direction region is determined as the new target control item.
[0030] In one embodiment, the first basic gesture is an extended palm gesture, and the second basic gesture is a clenched fist gesture;
[0031] The confirmation of the first basic gesture includes: when the first basic gesture is confirmed to be recognized based on the basic gesture recognition results corresponding to a first preset number of consecutive image frames, the hand movement speed is determined based on the hand position information corresponding to adjacent image frames in the first preset number of consecutive image frames, and when the hand movement speed meets a preset low speed condition, the first basic gesture is confirmed to be recognized.
[0032] The confirmation of recognizing the second basic gesture includes: after determining that the first basic gesture has been recognized, when the second basic gesture is confirmed to be recognized based on the basic gesture recognition results corresponding to a second preset number of consecutive image frames, the second basic gesture is confirmed to be recognized.
[0033] Secondly, this application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0034] Obtain configuration information, which includes at least one page identifier and a set of control items corresponding to each page identifier; the set of control items includes multiple control items.
[0035] Based on the page identifier of the current page, the corresponding set of control items is obtained from the configuration information, and a directional overlay is generated based on the set of control items;
[0036] Based on the spatial position change information of the user's hand, the target control item in the directional floating layer is determined;
[0037] Execute the corresponding control instructions according to the target control item.
[0038] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps:
[0039] Obtain configuration information, which includes at least one page identifier and a set of control items corresponding to each page identifier; the set of control items includes multiple control items.
[0040] Based on the page identifier of the current page, the corresponding set of control items is obtained from the configuration information, and a directional overlay is generated based on the set of control items;
[0041] Based on the spatial position change information of the user's hand, the target control item in the directional floating layer is determined;
[0042] Execute the corresponding control instructions according to the target control item.
[0043] The aforementioned gesture control method, electronic device, and readable storage medium based on page configuration determine the set of control items corresponding to the current page from the configuration information according to the page identifier of the current page, and generate an orientation overlay based on the set of control items, so that the selectable control items in the orientation overlay are associated with the current page; and determine the target control item from the orientation overlay according to the change of hand spatial position and execute the corresponding control instruction, so that the gesture control content can be configured according to the current page, avoiding the use of the same fixed control items for different pages. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1This is an application environment diagram of a gesture control method based on page configuration in one embodiment;
[0046] Figure 2 This is a flowchart illustrating a gesture control method based on page configuration in one embodiment;
[0047] Figure 3 This is a flowchart illustrating the gesture interaction process in an example embodiment.
[0048] Figure 4 This is a schematic diagram illustrating the determination of target control items in the directional floating layer in an example embodiment;
[0049] Figure 5 This is a flowchart illustrating a gesture control method based on page configuration in an example embodiment.
[0050] Figure 6 This is a diagram of the internal structure of an electronic device in an example embodiment. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0052] The gesture control method based on page configuration provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 receives configuration information from server 104. The configuration information includes at least one page identifier and a set of control items corresponding to each page identifier; the control item set includes multiple control items. Based on the page identifier of the current page, the terminal matches the corresponding set of control items from the configuration information and generates a directional overlay based on the control item set. Based on the spatial position change information of the user's hand, the terminal determines the target control item in the directional overlay and executes the corresponding control instruction based on the target control item. Before executing the corresponding control instruction based on the target control item, terminal 102 also generates a trigger event carrying the page identifier of the current page and reports the generated trigger event to server 104. After terminal 102 establishes a long connection with server 104, server 104 actively sends out configuration information. Subsequently, server 104 can resend the information at any time to overwrite and update it. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0053] In one embodiment, such as Figure 2 As shown, a gesture control method based on page configuration is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the method includes the following steps:
[0054] Step 202: Obtain configuration information, which includes at least one page identifier and a set of control items corresponding to each page identifier; the set of control items includes multiple control items.
[0055] The page identifier is a unique identifier for the page.
[0056] Optionally, after receiving the configuration information, the configuration information stored locally is updated as a whole using the received configuration information. Specifically, the original correspondence between page identifiers and control item sets is deleted, and the correspondence between page identifiers and control item sets is re-established based on the received configuration information. It can be understood that for page identifiers not included in the current configuration information, their original control item sets are no longer retained.
[0057] Optionally, if configuration information is received multiple times, the most recently received configuration information shall be used as the current valid configuration; if the set of pages contained in the current configuration information is empty, the correspondence between all page identifiers and control item sets stored locally shall be cleared, so that each page no longer has a set of control items determined based on the configuration information.
[0058] Optionally, when leaving the current page and switching to another page, the directional overlay generated based on the control set corresponding to the original current page is removed. At this time, the page identifier of the new page is re-acquired, and the control set is re-matched and the corresponding directional overlay is assembled. This avoids the control items of the original page from continuing to participate in gesture selection on the new page.
[0059] Optionally, if the page identifier of the current page is not included in the configuration information, either of the following methods can be used: The first is to disable the gesture triggering logic for the current page, determine the control item set corresponding to the current page, and not display the directional overlay; the second is to obtain the control item set corresponding to a preset page, use this control item set as the control item set for the current page, and provide default gesture control functionality. In the case of going back, the page identifier used for subsequent reporting of trigger events is consistent with the page that was undone.
[0060] Step 204: Based on the page identifier of the current page, obtain the corresponding set of control items from the configuration information, and generate a directional overlay based on the set of control items;
[0061] Step 206: Determine the target control item in the orientation overlay based on the spatial position change information of the user's hand;
[0062] Step 208: Execute the corresponding control command according to the target control item.
[0063] Optionally, after determining the target control item in the directional overlay and before executing the corresponding control instruction based on the target control item, the gesture control method based on the page configuration further includes generating a trigger event carrying the page identifier of the current page.
[0064] In the above-described gesture control method based on page configuration, the set of control items corresponding to the current page is determined from the configuration information according to the page identifier of the current page, and an orientation overlay is generated based on the set of control items, so that the optional control items in the orientation overlay are associated with the current page; the target control item is determined from the orientation overlay according to the change of hand spatial position and the corresponding control instruction is executed, so that the gesture control content can be configured according to the current page, avoiding the use of the same fixed control items for different pages.
[0065] In one embodiment, each control item includes location information and display information; generating a directional overlay based on the control item set includes: generating a directional overlay based on the location information and display information of each control item in the control item set corresponding to the current page.
[0066] Among them, the position information is used to characterize the position of the corresponding control item in the orientation selection space.
[0067] Optionally, the information displayed may include, but is not limited to, titles and icons.
[0068] Optionally, a directional overlay corresponding to the current page is generated based on the position and display information of each control item in the control item set, and the control items are arranged in the directional overlay according to their respective directional positions. When the control item set includes four control items, their position information can be configured as top left, top right, bottom left, and bottom right, respectively, so that the four control items are displayed in the four directional positions of the directional overlay. When the control item set includes two control items, their position information can be configured as top and bottom, or left and right, respectively, according to the page's interaction requirements. The number and directional positions of control items corresponding to different page identifiers can be different.
[0069] This embodiment enables each control item to be presented in the directional overlay according to a predetermined position and display content, thereby ensuring that the layout and display content of the directional overlay are consistent with the configuration of the current page, providing a basis for selecting the target control item based on the spatial position change information of the hand.
[0070] In one embodiment, each control item includes position information; determining the target control item in the orientation overlay based on the spatial position change information of the user's hand includes: acquiring multiple consecutive image frames, performing hand detection on each image frame to obtain the basic gesture recognition result and hand position information corresponding to each image frame; when determining that the interaction state enters the selection state based on the basic gesture recognition result corresponding to each image frame, determining the hand position when entering the selection state as the anchor point position; determining the displacement information of the current hand position relative to the anchor point position based on the current hand position and the anchor point position in the selection state; and determining the target control item from the orientation overlay based on the displacement information and the position information of each control item in the control item set corresponding to the current page.
[0071] The displacement information can be represented by the relative displacement (dx, dy).
[0072] Optionally, the anchor point can be located at the wrist key point, the palm key point, or other stable key points. It is understood that the anchor point location can be set according to actual needs.
[0073] Optionally, when a hand is detected in any image frame, multiple hand key points are acquired, and the basic gesture recognition result and hand position information corresponding to each image frame are determined based on these multiple hand key points. Here, hand key points are several detected feature points characterizing the hand structure. These multiple hand key points can cover the wrist, the metacarpophalangeal joints, interphalangeal joints, and fingertips of each finger.
[0074] Optionally, determining the basic gesture recognition result corresponding to each image frame based on multiple hand key points includes determining the hand geometric feature parameters corresponding to the hand based on the relative positional relationship between multiple hand key points; matching the hand geometric feature parameters with preset gesture judgment conditions corresponding to multiple basic gestures respectively; and determining the basic gesture recognition result corresponding to the image frame based on the matching result.
[0075] Optionally, before determining the hand geometric feature parameters based on the relative positional relationships between multiple hand key points and matching these parameters with preset gesture judgment conditions corresponding to each basic gesture, a validity judgment of the hand orientation is first performed. Specifically, using the line connecting the wrist key point to the metacarpophalangeal joint of the middle finger as the hand orientation vector, the angle between the hand orientation vector and the vertically upward direction in the image coordinate system is calculated, and this angle is compared with a preset orientation threshold. When the angle is not greater than the preset orientation threshold, the current hand posture is determined to meet the orientation condition, and basic gesture judgment continues; otherwise, basic gesture judgment is not performed. This filters out tilted transitional postures formed during hand placement (horizontal, inverted, or waving), reducing the possibility of misidentification.
[0076] Optionally, a first-order adaptive low-pass filter is applied to the coordinates of each hand keypoint to obtain smoothed keypoint coordinates used to determine the basic gesture recognition result and hand position information. For the current frame, the smoothing coefficient is calculated using the following formula:
[0077] ;
[0078] in, The calculation formula is as follows:
[0079] ;
[0080] in, The time interval between adjacent frames, cutoff frequency The cutoff frequency increases adaptively with the hand movement speed. The calculation formula is as follows:
[0081] ;
[0082] in, The lowest cutoff frequency, For speed coefficient, This represents the rate of change of the corresponding coordinate components. When the hand moves at low speeds, the cutoff frequency is low, resulting in strong smoothing and suppression of jitter; when the hand moves at high speeds, the cutoff frequency increases, resulting in weaker smoothing and reduced hysteresis. When the hand is detected again after not being detected in consecutive image frames, the filtering history is cleared to avoid hand position drift caused by old smoothing results.
[0083] Optionally, the hand key points are represented by normalized two-dimensional coordinates relative to the current image frame, denoted as (x, y), where x represents the relative position of the key point in the width direction of the image frame, and y represents the relative position of the key point in the height direction of the image frame. The values of x and y are both in the range of 0 to 1.
[0084] Optionally, the geometric feature parameter of the hand is the extension of the target finger. For each target finger in the image frame, the metacarpophalangeal joints, proximal interphalangeal joints, and fingertip key points of the target finger are obtained. The ratio of the distance between the metacarpophalangeal joints and the fingertip key points to the sum of the distances from the metacarpophalangeal joints to the proximal interphalangeal joints and from the proximal interphalangeal joints to the fingertip key points is determined as the extension of the target finger.
[0085] Optionally, to eliminate the influence of hand distance, size, and translation, the distance between the wrist key point and the metacarpophalangeal joint of the middle finger is used as the palm length, and the palm length is used to normalize length parameters such as spacing and distance.
[0086] For the index, middle, ring, and little fingers, corresponding extension degrees are determined. When the extension degrees of all four fingers are not less than a first extension degree threshold, the extension degree of the thumb is not less than a second extension degree threshold, and the sum of the normalized distances between adjacent fingertips is not less than a first distance threshold, the basic gesture recognition result corresponding to the current image frame is determined to be an extended palm gesture. The first extension degree threshold can be 0.97, the second extension degree threshold can be 0.95, and the first distance threshold can be 0.45. It can be understood that the thresholds can be adjusted according to the gesture recognition requirements.
[0087] For the index, middle, ring, and little fingers, corresponding extension degrees are determined. When the extension degree of each of these four fingers is no greater than a third extension degree threshold, the normalized distance from the fingertips to the wrist is no greater than a first distance threshold, and the normalized distance from the thumb tip to the metacarpophalangeal joint of the index finger is no greater than a second distance threshold, the basic gesture recognition result for the current image frame is determined to be a clenched fist gesture. The third extension degree threshold can be 0.55, the first distance threshold can be 0.95, and the second distance threshold can be 0.5. It can be understood that the thresholds can be adjusted according to the gesture recognition requirements.
[0088] Optionally, hand detection can be performed on each image frame using a neural network or a pre-defined hand keypoint detection model. It is understood that the specific method for hand detection on each image frame can be set according to actual needs, as long as it can output hand keypoint or basic gesture recognition results.
[0089] Optionally, based on multiple key points of the hand in the current image frame, when a preset gesture judgment condition is determined to match multiple basic gestures simultaneously, the basic gesture recognition result is determined according to the preset basic gesture priority.
[0090] In this embodiment, by performing hand detection on multiple consecutive image frames, the corresponding basic gesture recognition results and hand position information are obtained. When entering the selection state, the current hand position is used as the anchor point, and the target control item is determined based on the displacement information of the current hand position relative to the anchor point position. This allows the selection of the control item to be based on the relative position change, which can reduce the impact of the initial hand position difference on the selection result.
[0091] In one example embodiment, for the target finger, its metacarpophalangeal joint A, proximal interphalangeal joint B, and fingertip key point C are taken, and the straightness s = |AC| / (|AB|+|BC|). The value range of the straightness s is (0,1], and its physical meaning is the degree of collinearity of the three joints of the finger. The closer the straightness s is to 1, the straighter it is, and the significantly less than 1 indicates bending.
[0092] In one embodiment, each control item includes an action identifier; executing the corresponding control instruction according to the target control item includes: determining the action processing method corresponding to the action identifier based on the action identifier corresponding to the target control item; and executing the corresponding control instruction when the action processing method is a local processing method.
[0093] Optionally, the generated trigger event carries not only the page identifier of the current page, but also the action identifier of the target control.
[0094] Optionally, when the action processing method is an external processing method, an action processing request carrying an action identifier is output so that the corresponding control instruction can be executed according to the action identifier.
[0095] Optionally, the action identifier can be an empty string, which indicates that the location is disabled on the current page and is unavailable in the overlay.
[0096] This embodiment enables the distribution of control commands based on action identifiers.
[0097] In one embodiment, determining when the interaction state enters the selection state based on the basic gesture recognition results corresponding to each image frame includes: when the first basic gesture is recognized based on the basic gesture recognition results of each image frame, switching the interaction state to the preparation state; and within a preset time after entering the preparation state, when the second basic gesture is recognized based on the basic gesture recognition results of subsequent image frames after the first basic gesture, switching the interaction state from the preparation state to the selection state and displaying a direction overlay.
[0098] Optionally, after generating a trigger event, the control interaction state enters a trigger lock state, and new trigger events are prohibited from being generated in the trigger lock state; after entering the trigger lock state, if the preset confirmation gesture is not recognized, and the preset confirmation gesture is recognized again thereafter, the trigger lock state is released.
[0099] This embodiment provides clear gesture timing conditions for entering the selection state, reducing the possibility of misidentification and mistriggering.
[0100] In one embodiment, executing the corresponding control instruction according to the target control item includes: determining whether the target control item meets the preset trigger condition; if so, executing the corresponding control instruction according to the target control item; and determining that the preset trigger condition is met when the target control item is continuously in the selected state for a period of time that reaches the preset dwell time, or when a preset confirmation gesture is recognized during the period when the target control item is in the selected state.
[0101] Optionally, the preset confirmation gesture can be the first basic gesture, the second basic gesture, or the OK gesture. It can be understood that the preset confirmation gesture can be set according to actual needs.
[0102] Optionally, for the middle, ring, and little fingers, corresponding extension degrees are determined respectively. When the extension degrees of these three fingers are all no greater than a fourth extension degree threshold, and the normalized distance between the tips of the thumb and index finger is no greater than a third distance threshold, the basic gesture recognition result corresponding to the current image frame is determined to be the OK gesture. The fourth extension degree threshold can be 0.32, and the third distance threshold can be 0.9. It can be understood that the thresholds can be adjusted according to the gesture recognition requirements.
[0103] This embodiment provides a basis for triggering the target control item.
[0104] In one embodiment, the displacement information includes displacement amount and displacement direction; determining the target control item from the direction overlay based on the displacement information and the position information of each control item in the control item set corresponding to the current page includes: determining the direction region corresponding to each control item based on the position information of each control item; when the displacement amount is less than a preset dead zone threshold, not determining any control item in the control item set corresponding to the current page as the target control item; when the displacement amount is greater than or equal to the preset dead zone threshold, determining the control item corresponding to the direction region where the displacement direction is located as the target control item; maintaining the target control item when the displacement direction does not reach the preset exit threshold of the direction region corresponding to the target control item; and determining the control item corresponding to the adjacent direction region as the new target control item when the displacement direction reaches the preset entry threshold of the adjacent direction region.
[0105] Optionally, regions with displacement less than a preset dead zone threshold are considered dead zones.
[0106] This embodiment can reduce erroneous selections caused by natural jitter and decrease repeated jumps in control terms near the directional boundary, thereby improving the stability of the target control term determination process.
[0107] In one embodiment, the first basic gesture is an extended palm gesture, and the second basic gesture is a clenched fist gesture. Confirming the recognition of the first basic gesture includes: when the first basic gesture is confirmed based on the basic gesture recognition results corresponding to a first preset number of consecutive image frames, determining the hand movement speed based on the hand position information corresponding to adjacent image frames in the first preset number of consecutive image frames; when the hand movement speed meets a preset low-speed condition, confirming the recognition of the first basic gesture. Confirming the recognition of the second basic gesture includes: after confirming the recognition of the first basic gesture, when the second basic gesture is confirmed based on the basic gesture recognition results corresponding to a second preset number of consecutive image frames, confirming the recognition of the second basic gesture.
[0108] It is understandable that basic gestures can be set according to actual needs to control the interaction state from the ready state to the selection state.
[0109] Optionally, before obtaining the basic gesture recognition results corresponding to each image frame, hand detection is performed on each image frame to obtain hand confidence information characterizing the credibility of the hand's presence. When the hand confidence information meets the preset confidence conditions, and the first basic gesture is confirmed to be recognized based on the basic gesture recognition results corresponding to a first preset number of consecutive image frames, and the hand movement speed meets the preset low-speed condition, the first basic gesture is confirmed to be recognized. This can suppress pseudo-outstretched palms that are geometrically similar to outstretched palms due to motion blur during rapid waving. When the second basic gesture is confirmed to be recognized based on the basic gesture recognition results corresponding to a second preset number of consecutive image frames, and the hand confidence information meets the preset confidence conditions, the second basic gesture is confirmed to be recognized. When recognizing a fist gesture, the hand movement speed does not need to meet the preset low-speed condition, which allows for continuous recognition during the fist selection process.
[0110] Optionally, when the recognition results of the basic gestures corresponding to the first preset number of consecutive image frames confirm the recognition of the first basic gesture, or after the recognition of the first basic gesture, when the recognition results of the basic gestures corresponding to the second preset number of consecutive image frames confirm the recognition of the second basic gesture, a trigger flag is set so that the same hand shape is not triggered repeatedly until a frame with a different hand shape appears and then the flag is reset.
[0111] This embodiment can reduce the possibility of false triggering and false identification.
[0112] In one example embodiment, multiple consecutive image frames are acquired, and hand detection is performed on each image frame to obtain the basic gesture recognition result and hand position information corresponding to each image frame.
[0113] When a palm-out gesture is recognized for several consecutive frames, and the hand movement speed meets the preset low-speed condition and the hand confidence information meets the preset confidence condition, the interaction state changes from the idle state to the ready state.
[0114] If no fist gesture is detected for several consecutive frames within a preset time window, but an outstretched hand gesture is detected, the system remains in the ready state. If neither fist nor outstretched hand gesture is detected for several consecutive frames within a preset time window, the system returns to the idle state. If a fist gesture is detected for several consecutive frames within a preset time window, the interactive state enters the selection state, a direction overlay pops up, and the anchor point position is determined based on the hand position information; the normalized position of the wrist can be used as the anchor point position. In the selection state, in subsequent image frames where a fist gesture is detected, the displacement and direction of the current hand position relative to the anchor point position are calculated. If the displacement is less than a preset dead zone threshold, the target control item is not determined; if it is greater than or equal to the preset dead zone threshold, the control item corresponding to the direction region where the displacement direction is located is determined as the target control item, and hysteresis is determined using preset exit and entry thresholds. If the target control item is continuously selected for a preset dwell time, or if an outstretched hand gesture is detected during its selection, a trigger event carrying the current page identifier and the action identifier of the target control item is generated, and the corresponding control command is executed according to the target control item. When a target control item is selected, a release threshold is used to avoid repeated entry and exit at the dead zone boundary; wherein, the release threshold is less than the dead zone threshold.
[0115] After a trigger event is generated, the system enters a trigger lock state and prohibits the generation of new trigger events. If no palm-out gesture is detected within a preset time period, and a palm-out gesture is subsequently recognized again, the trigger lock state is released.
[0116] If the displacement is less than the preset dead zone threshold, and a palm-out gesture is detected at this time, it is considered that the user has not completed the selection of the control item, the interaction is canceled, no trigger event is generated, and the interaction state enters a pause / cooling state. If, while maintaining a fist gesture and making a direction selection, the recognition result switches to a gesture that is neither a fist gesture nor a palm-out gesture, or an unrecognized state, the interaction is also canceled and enters a pause / cooling state.
[0117] If no hand is detected for a preset duration, the control interaction state is reset, the directional overlay is closed, and the current interaction is canceled.
[0118] In one example embodiment, the gesture interaction process is as follows: Figure 3 As shown.
[0119] In one example embodiment, a directional overlay is generated based on the position and display information of each control item in the control item set corresponding to the current page. If the control item set includes four control items, and the position information of the four control items is configured as top left, top right, bottom left, and bottom right respectively, the determination of the target control item is as follows: Figure 4 As shown.
[0120] In one example embodiment, a gesture control method based on page configuration is provided. Figure 5 The flowchart of this method includes the following steps:
[0121] The system retrieves configuration information, including at least one page identifier and a set of control items corresponding to each page identifier. Each control item set includes multiple control items, each containing position information, display information, and an action identifier. Based on the current page's page identifier, the system matches the corresponding set of control items from the configuration information and generates a directional overlay based on the position and display information of each control item in the current page's control item set. The system acquires multiple consecutive image frames, performs hand detection on each frame, and obtains the basic gesture recognition results and hand position information for each frame. When the first basic gesture is recognized based on the basic gesture recognition results of each image frame, the interaction state is switched to a ready state. Within a preset time after entering the ready state, when the second basic gesture is recognized based on the basic gesture recognition results of subsequent image frames after the first basic gesture, the interaction state is switched from the ready state to the selection state, and the selection is displayed. A directional overlay is created, and the hand position when entering the selection state is determined as the anchor point. Based on the current hand position and the anchor point position in the selection state, the displacement information of the current hand position relative to the anchor point position is determined. Based on the displacement information and the position information of each control item in the control item set corresponding to the current page, the target control item is determined from the directional overlay. When the target control item remains in the selected state for a preset dwell time, or when a preset confirmation gesture is recognized during the target control item being in the selected state, a trigger event carrying the page identifier of the current page and the action identifier of the target control item is generated. Based on the action identifier corresponding to the target control item, the action processing method corresponding to the action identifier is determined. When the action processing method is the local processing method, the corresponding control instruction is executed. When the action processing method is the external processing method, an action processing request carrying the action identifier is output for executing the corresponding control instruction based on the action identifier.
[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0123] In one exemplary embodiment, an electronic device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this electronic device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores configuration information and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a page-configured gesture control method.
[0124] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0125] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0126] Obtain configuration information, which includes at least one page identifier and a set of control items corresponding to each page identifier; the set of control items includes multiple control items; based on the page identifier of the current page, match the corresponding set of control items from the configuration information, and generate an orientation overlay based on the set of control items; determine the target control item in the orientation overlay based on the spatial position change information of the user's hand; execute the corresponding control instruction based on the target control item.
[0127] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0128] Obtain configuration information, which includes at least one page identifier and a set of control items corresponding to each page identifier; the set of control items includes multiple control items; based on the page identifier of the current page, match the corresponding set of control items from the configuration information, and generate an orientation overlay based on the set of control items; determine the target control item in the orientation overlay based on the spatial position change information of the user's hand; execute the corresponding control instruction based on the target control item.
[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A gesture control method based on page configuration, characterized in that, include: Obtain configuration information, which includes at least one page identifier and a set of control items corresponding to each page identifier; the set of control items includes multiple control items. Based on the page identifier of the current page, the corresponding set of control items is obtained from the configuration information, and a directional overlay is generated based on the set of control items; Based on the spatial position change information of the user's hand, the target control item in the directional floating layer is determined; Execute the corresponding control instructions according to the target control item.
2. The method according to claim 1, characterized in that, Each of the control items includes location information and display information; The step of generating the directional overlay based on the set of control items includes: generating the directional overlay according to the position information and display information of each control item in the set of control items corresponding to the current page.
3. The method according to claim 1, characterized in that, Each control item includes location information; determining the target control item in the orientation overlay based on the spatial position change information of the user's hand includes: Acquire multiple consecutive image frames, perform hand detection on each image frame, and obtain the basic gesture recognition result and hand position information corresponding to each image frame; When determining the interaction state to enter the selection state based on the basic gesture recognition results corresponding to each image frame, the hand position when entering the selection state is determined as the anchor point position. Based on the current hand position in the selected state and the anchor point position, determine the displacement information of the current hand position relative to the anchor point position; The target control item is determined from the directional overlay based on the displacement information and the position information of each control item in the control item set corresponding to the current page.
4. The method according to claim 1, characterized in that, Each control item includes an action identifier; The step of executing the corresponding control instruction according to the target control item includes: determining the action processing mode corresponding to the action identifier according to the action identifier corresponding to the target control item; and executing the corresponding control instruction when the action processing mode is the local processing mode.
5. The method according to claim 3, characterized in that, When determining the interaction state and entering the selection state based on the basic gesture recognition results corresponding to each of the image frames, the following steps are included: When the first basic gesture is identified based on the basic gesture recognition results of each of the image frames, the interaction state is switched to the ready state; Within a preset time after entering the preparatory state, when the recognition result of the basic gesture in the subsequent image frame after the recognition of the first basic gesture is confirmed to be the recognition of the second basic gesture, the interaction state is switched from the preparatory state to the selection state, and the directional overlay is displayed.
6. The method according to claim 3, characterized in that, The step of executing the corresponding control instruction according to the target control item includes: Determine whether the target control item meets the preset triggering condition; if so, execute the corresponding control instruction according to the target control item. When the target control item remains in the selected state for a duration that reaches a preset dwell time, or when a preset confirmation gesture is recognized during the period when the target control item is in the selected state, it is determined that the preset triggering condition is met.
7. The method according to claim 3, characterized in that, The displacement information includes displacement amount and displacement direction; determining the target control item from the direction overlay based on the displacement information and the position information of each control item in the control item set corresponding to the current page includes: Based on the position information of each control item, determine the directional region corresponding to each control item; When the displacement is less than the preset dead zone threshold, no control item in the control item set corresponding to the current page is determined as the target control item. When the displacement is greater than or equal to the preset dead zone threshold, the control item corresponding to the direction region where the displacement direction is located is determined as the target control item. When the displacement direction does not reach the preset exit threshold of the direction region corresponding to the target control item, the target control item is maintained. When the displacement direction reaches the preset entry threshold of the adjacent direction region, the control item corresponding to the adjacent direction region is determined as the new target control item.
8. The method according to claim 5, characterized in that, The first basic gesture is an extended palm gesture, and the second basic gesture is a clenched fist gesture; The confirmation of the first basic gesture includes: when the first basic gesture is confirmed to be recognized based on the basic gesture recognition results corresponding to a first preset number of consecutive image frames, the hand movement speed is determined based on the hand position information corresponding to adjacent image frames in the first preset number of consecutive image frames, and when the hand movement speed meets a preset low speed condition, the first basic gesture is confirmed to be recognized. The confirmation of recognizing the second basic gesture includes: after determining that the first basic gesture has been recognized, when the second basic gesture is confirmed to be recognized based on the basic gesture recognition results corresponding to a second preset number of consecutive image frames, the second basic gesture is confirmed to be recognized.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.
