Gesture interaction method and system under multiple interfaces, computer equipment and storage medium

Abstract

The invention relates to the technical field of man-machine interaction, and discloses a gesture interaction method and system under multiple interfaces, computer equipment and a storage medium, the interaction method comprises the following steps: monitoring a hand action of a user, and capturing a gesture event; subscribing a target gesture event according to a function requirement of a current display interface, and when the target gesture event is generated, acquiring a gesture function parameter of the target gesture event; executing a corresponding operation in a current display interface according to the target gesture event and the specific gesture parameter; by capturing the gesture event and performing targeted recognition on each interface, the accuracy is improved, gesture conflicts of different interfaces are avoided, each interface is enabled to perform quick operation based on the recognition result, and accurate and quick response aiming at user requirements can be realized.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a gesture interaction method, system, computer equipment and storage medium under multiple interfaces. Background Art

[0002] With the continuous development of electronic devices and interactive technologies, human-computer interaction has become a popular function of smart terminals. However, in the traditional interactive mode, when facing the switching of multiple application interfaces, gestures may be misrecognized or fail to correctly respond to the functional requirements of the corresponding interface, affecting the user experience. Therefore, it is necessary to improve the existing technology.

[0003] The above information is presented as background information only to assist with understanding the present disclosure and no determination or admission is made as to whether any of the above may be used as prior art with respect to the present disclosure. Summary of the invention

[0004] The present invention provides a gesture interaction method, system, computer equipment and storage medium under multiple interfaces to solve the problems existing in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A gesture interaction method under multiple interfaces, comprising:

[0007] Monitor user hand movements and capture gesture events;

[0008] Subscribe to a target gesture event according to the functional requirements of the current display interface, and when the target gesture event is generated, obtain the gesture function parameters of the target gesture event;

[0009] According to the target gesture event and the specific gesture parameter, a corresponding operation is performed in the current display interface.

[0010] Optionally, monitoring the user's hand movements and capturing gesture events includes:

[0011] Monitor the hand movements of the user and obtain basic gesture parameters in the hand movements; the basic gesture parameters include one or more of shape parameters, position parameters, trajectory parameters, time parameters, speed parameters, distance parameters and rotation parameters;

[0012] Based on the gesture basic parameters, determining a start gesture, an intermediate continuous gesture, and an end gesture of the hand motion;

[0013] According to the characteristics of the start gesture, the intermediate continuous gesture and the end gesture, the gesture basic parameters are encapsulated into one or more gesture events.

[0014] Optionally, when encapsulating the basic gesture parameters into one or more gesture events, the method further includes:

[0015] Adding a function tag to the gesture event;

[0016] Subscribing to the target gesture event according to the functional requirements of the current display interface includes:

[0017] Acquire the functional requirements corresponding to the current display interface, and subscribe to gesture events corresponding to the functional tags that match the functional requirements.

[0018] Optionally, when an interface switch is detected, the method further includes:

[0019] Cancel the currently subscribed target gesture event;

[0020] The functional requirements corresponding to the switched display interface are obtained, and the gesture events corresponding to the functional tags matching the functional requirements are subscribed.

[0021] Optionally, the gesture interaction method under multiple interfaces further includes:

[0022] Store the subscription history of gesture events on each display interface;

[0023] When the display interface is not switched for the first time, the subscription status of the display interface is restored according to the subscription history.

[0024] Optionally, performing a corresponding operation in the current display interface according to the target gesture event and the specific gesture parameter includes:

[0025] Determine the operation requirements corresponding to the current display interface according to the pre-set mapping relationship between gesture events and operations;

[0026] Determining the user's operation intention based on the gesture function parameters;

[0027] According to the operation requirements and operation intentions, corresponding operations are performed in the current display interface.

[0028] Optionally, before performing a corresponding operation in the current display interface according to the operation requirement and the operation intention, the method further includes:

[0029] Performing anti-misoperation verification on the hand movement to determine whether the hand movement is valid;

[0030] When the hand movement is valid, the corresponding operation is performed in the current display interface according to the operation requirements and operation intentions.

[0031] Optionally, the performing anti-misoperation verification on the hand action includes:

[0032] Acquiring operation validity parameters of the hand motion, wherein the operation validity parameters include a time parameter and a motion verification parameter of the hand motion, and the motion verification parameter includes a motion change state, a gesture combination formed, and / or a number of repetitions;

[0033] Based on the operation validity parameter, the validity of the hand action is determined.

[0034] Optionally, when performing a corresponding operation in the current display interface, the step further includes:

[0035] Providing feedback information in response to the target gesture event;

[0036] The feedback information includes one or more of visual feedback information, auditory feedback information and tactile feedback information.

[0037] Optionally, the interface includes one or more of a video selection interface, a video playback interface, a music interface, a dialing interface, a map page and a shortcut menu.

[0038] Optionally, executing a corresponding operation in the current display interface includes:

[0039] Selection operation, including displaying a selection indicator;

[0040] Media interaction operations, including starting and stopping the playback of videos and / or music, adjusting the playback progress of videos and / or music, and confirming the selection of videos and / or music;

[0041] Communication operations, including number selection, triggering dialing, and hanging up calls;

[0042] Map operations, including panning and zooming of the entire or partial map;

[0043] Quick menu operations, including quick menu display and interaction.

[0044] The present invention also provides a gesture interaction system under multiple interfaces, which is used to implement the gesture interaction method under multiple interfaces as described in any one of the above items, including:

[0045] Gesture recognition module, used to monitor user hand movements and capture gesture events;

[0046] A subscription management module, used to subscribe to a target gesture event according to the functional requirements of the current display interface, and when the target gesture event is generated, obtain the gesture function parameters of the target gesture event;

[0047] The operation execution module is used to execute a corresponding operation in the current display interface according to the target gesture event and the specific gesture parameter.

[0048] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the gesture interaction method under multiple interfaces as described in any one of the above items are implemented.

[0049] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the gesture interaction method under a multi-interface as described in any one of the above items through the computer program.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention provides a gesture interaction method, system, computer device and storage medium under multiple interfaces, which improve accuracy by capturing gesture events and identifying them in a targeted manner on each interface, avoid gesture conflicts between different interfaces, and enable each interface to operate quickly based on the recognition results, which is conducive to achieving accurate and rapid response to user needs.

[0052] The present invention has other features and advantages, which will be apparent from or will be described in detail in the accompanying drawings and the following detailed description incorporated herein, which together serve to explain the specific principles of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a flow chart of a gesture interaction method under multiple interfaces provided by an embodiment of the present invention;

[0055] Figure 2 It is a schematic diagram of gesture event encapsulation in a gesture interaction method under multiple interfaces provided by an embodiment of the present invention;

[0056] Figure 3 is a flowchart of step S1 in a gesture interaction method under a multi-interface provided by an embodiment of the present invention;

[0057] Figure 4 is another flow chart of step S1 in a gesture interaction method under a multi-interface provided by an embodiment of the present invention;

[0058] Figure 5is a flowchart of step S2 in a gesture interaction method under multiple interfaces provided by an embodiment of the present invention;

[0059] Figure 6 is another flow chart of step S2 in a gesture interaction method under a multi-interface provided by an embodiment of the present invention;

[0060] Figure 7 is another flow chart of step S2 in a gesture interaction method under a multi-interface provided by an embodiment of the present invention;

[0061] Figure 8 is a flowchart of step S3 in a gesture interaction method under multiple interfaces provided by an embodiment of the present invention;

[0062] Fig. 9 is another flow chart of step S3 in a gesture interaction method under a multi-interface provided by an embodiment of the present invention;

[0063] Fig.10 is another flow chart of step S1 in a gesture interaction method under a multi-interface provided by an embodiment of the present invention;

[0064] Fig.11 It is a structural block diagram of a gesture interaction system under multiple interfaces provided by an embodiment of the present invention.

[0065] Reference numerals: 10, gesture recognition module; 20, subscription management module; 30, operation execution module. DETAILED DESCRIPTION

[0066] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0067] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0068] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0069] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0070] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0071] Without further limitations, in this application, the words "include", "comprises", "has" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0072] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0073] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0074] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0075] Please refer to Figure 1 , an embodiment of the present invention provides a gesture interaction method under multiple interfaces, including:

[0076] S1. Monitor user hand movements and capture gesture events.

[0077] In this embodiment, the interface includes one or more of a video selection interface, a video playback interface, a music interface, a dialing interface, a map page, and a shortcut menu.

[0078] Please refer to Figure 2 , Figure 3 Specifically, in this step, monitoring the user's hand movements and capturing gesture events include:

[0079] S11. Monitor the user's hand movements and obtain basic gesture parameters in the hand movements.

[0080] The basic gesture parameters include one or more of shape parameters, position parameters, trajectory parameters, time parameters, speed parameters, distance parameters and rotation parameters;

[0081] In step S11, a variety of sensors, such as cameras, touch screen sensors, etc., can work together to monitor the user's hand movements.

[0082] The following example describes how to obtain each parameter in detail.

[0083] 1) For shape parameter acquisition, the camera’s image recognition technology can be used. When the hand appears in the detectable area, the image is analyzed to identify the outline of the palm and determine whether the palm is open, clenched, or in other specific shapes. This can be achieved using an edge algorithm. In addition, the shape of the fingers can also be determined by analyzing the degree of curvature and relative position of the fingers.

[0084] 2) For position parameter acquisition, the coordinates of the hand are obtained by obtaining the position information of the hand on the screen and the distance between the hand and the screen, thereby obtaining the position parameters. For the palm, the coordinate set of the palm contour can be used for comprehensive judgment.

[0085] 3) For trajectory parameter acquisition, during the hand movement, the movement path of the finger or palm is continuously recorded. For example, the position of the finger touch point is collected at a certain time frequency to form movement trajectory data; for gesture movements in the air, cameras and other devices can be used to capture the trajectory of the hand in three-dimensional space.

[0086] 4) For the acquisition of time parameters, the start time, duration and end time of the hand movement are recorded through a timer; it can be understood that when the start of the hand movement is detected, the timer starts; when the hand movement ends, the timer stops, and accurate time information is obtained in sequence, laying the foundation for subsequent recognition and judgment.

[0087] 5) For speed parameter acquisition, specifically, the speed parameters may include real-time speed values ​​and acceleration values, and the speed is calculated based on the displacement change of the hand movement over a period of time. For example, during finger sliding, the speed of the finger sliding is obtained by calculating the position change at adjacent time points divided by the time interval. The speed parameter can be used to distinguish different operation intentions, such as fast sliding and slow sliding.

[0088] 6) For distance parameter acquisition, in some scenarios where operation control needs to be performed based on distance, such as map zooming, the zoom ratio of the map is determined by calculating the distance between the starting position and the ending position of the palm push and pull.

[0089] 7) Obtaining rotation parameters: For devices that support rotation detection, sensors such as gyroscopes are used to detect the rotation of the hand. For example, when browsing images, the rotation of the image can be achieved by identifying the user's hand rotation.

[0090] S12. Determine the start gesture, the middle continuous gesture and the end gesture of the hand movement based on the basic gesture parameters.

[0091] Please refer to Figure 2 In this step, based on the acquired basic gesture parameters, the starting gesture, the intermediate continuous gesture and the ending gesture of the hand movement are accurately determined, so as to realize the recognition and determination of a gesture event.

[0092] It is understandable that when the parameters of the hand movement meet the preset start gesture characteristics, it is identified as a start gesture. For example, when the palm appears on the screen in a specific shape and position and meets certain time thresholds and position stability conditions, it is considered a start gesture; after the start gesture, the gesture state during the continuous hand movement is an intermediate continuous gesture, and the state of the intermediate continuous gesture is tracked in real time by continuously monitoring the changes in the parameters of the hand movement; when the hand movement stops or a specific change occurs that meets the end condition, such as the palm leaves the screen area, the fingers stop moving, or the gesture parameters meet specific end rules, it is identified as an end gesture.

[0093] S13. Encapsulate basic gesture parameters into one or more gesture events according to the characteristics of the start gesture, the intermediate continuous gesture, and the end gesture.

[0094] Please refer to Figure 4 In this step, when encapsulating the basic gesture parameters into one or more gesture events, it also includes:

[0095] S131. Add a function tag to the gesture event.

[0096] In this step, the basic parameters of the gesture are encapsulated into one or more gesture events according to the characteristics of the start gesture, the intermediate continuous gesture, and the end gesture. For example, a series of continuous finger sliding actions and their related parameters are encapsulated into a "sliding gesture event" to represent a specific operation intention in a certain interface, such as adjusting progress or panning a map.

[0097] When encapsulating gesture events, add function tags to each gesture event so that it can be subscribed and identified according to the interface function requirements. For example, in the video playback interface, the function tag of "gesture V" can be set to "video playback control", indicating that the gesture is used to control the playback or pause of the video; in the music interface, the function tag of "single finger left and right movement" can be "music playback progress adjustment".

[0098] S2. Subscribe to a target gesture event according to the functional requirements of the current display interface, and when a target gesture event is generated, obtain gesture function parameters of the target gesture event.

[0099] It can be understood that, for interfaces with specific interactive function requirements, a specific set of gesture events corresponding to such functions is subscribed to. For example, for interfaces with selection and confirmation function requirements, gesture events used for selection and confirmation operations are subscribed to; for interfaces with playback control function requirements, gesture events related to play, pause, fast forward and rewind operations are subscribed to; for interfaces with interface navigation function requirements, gesture events related to operations such as returning to the parent interface and switching interfaces are subscribed to; for interfaces with parameter adjustment function requirements, gesture events related to parameter adjustment are subscribed to; for interfaces with map operation function requirements, gesture events related to map pan and zoom operations are subscribed to; for interfaces with shortcut function calling requirements, gesture events related to shortcut function calling and selection are subscribed to.

[0100] Please refer to Figure 5 Further, in step S2, subscribing to the target gesture event according to the functional requirements of the current display interface includes:

[0101] S21. Obtain functional requirements corresponding to the current display interface, and subscribe to gesture events corresponding to function tags that match the functional requirements.

[0102] It is understandable that the system analyzes and defines the functions of each display interface in advance, and clarifies the gesture events that may be required. For example, the main function of the video selection interface is to allow users to select videos, and the gesture events involved include palm exhaling dots to trigger the selection operation, OK gesture to confirm the selection, etc.; the video playback interface needs to support related gesture events such as controlling video playback and adjusting progress.

[0103] After obtaining the functional requirements corresponding to the current display interface, the system subscribes to gesture events corresponding to the functional tags that match the functional requirements according to the obtained current interface functional requirements. For example, in the video playback interface, the system will subscribe to gesture events with functional tags such as "video playback control" and "video playback progress adjustment". When these gesture events occur, the system can respond in time and perform corresponding processing.

[0104] Please refer to Figure 6 Furthermore, in this embodiment, when the interface switching is detected, it also includes:

[0105] S22, cancel the currently subscribed target gesture event;

[0106] S23: Acquire the functional requirements corresponding to the switched display interface, and subscribe to gesture events corresponding to the function tags that match the functional requirements.

[0107] It is understandable that when an interface switch is detected, the currently subscribed target gesture event is first canceled to avoid conflicts with gesture events of the old interface in the new interface. For example, when switching from the video playback interface to the music interface, the gesture events such as "video playback control" and "video playback progress adjustment" subscribed to the video playback interface are immediately canceled; then, by obtaining the functional requirements corresponding to the switched display interface, further subscribe to gesture events corresponding to the corresponding functional tags, for example, after switching to the music interface, subscribe to gesture events such as "music playback control" and "music playback progress adjustment" required by the music interface.

[0108] Please refer to Figure 7 Furthermore, the gesture interaction method under multiple interfaces provided by this embodiment further includes:

[0109] S24, storing the subscription history of gesture events on each display interface;

[0110] S25. When the display interface is not switched for the first time, the subscription status of the display interface is restored according to the subscription history.

[0111] In this embodiment, by storing the subscription history of gesture events of each display interface, a database or other suitable storage method is used, and the gesture events subscribed by each interface and their function tags, subscription time and other detailed information are recorded with the interface name as the index. When it is not the first time to switch to a certain display interface, the system will quickly restore the subscription status of the display interface based on the subscription history, and ensure accurate restoration to the previous subscription status by reading the subscription history data and matching and verifying it with the current interface, including re-subscribing to the corresponding gesture events and initializing related parameters, so as to provide a high-quality user experience.

[0112] S3. Execute a corresponding operation in the current display interface according to the target gesture event and the specific gesture parameters.

[0113] Please refer to Figure 8 , the step further comprises:

[0114] S31, determining an operation requirement corresponding to the current display interface according to a preset mapping relationship between gesture events and operations;

[0115] S32, determining the user's operation intention based on the gesture function parameters;

[0116] S33: Execute corresponding operations in the current display interface according to the operation requirements and operation intentions.

[0117] In this step, the operation requirements corresponding to the current display interface are determined according to the pre-set mapping relationship between gesture events and operations. For example, in the video playback interface, "gesture V" is mapped to the video play / pause operation. When the "gesture V" event is detected, the system knows that the video play or pause operation needs to be executed.

[0118] Based on the gesture function parameters, the user's operation intention is further determined. For example, by analyzing the speed, direction, and distance of the finger sliding, it is determined whether the user wants to quickly skip a video or fine-tune the playback progress. On the map page, based on the direction and distance parameters of the finger movement, it is determined whether the user wants to pan or zoom the map, as well as the specific pan direction and zoom ratio.

[0119] Furthermore, before executing the operation, the system will perform anti-misoperation verification on the hand movement to determine whether the hand movement is valid.

[0120] Please refer to Fig. 9 Based on this, before performing the corresponding operation in the current display interface according to the operation requirements and operation intentions, it also includes:

[0121] S321, verify the hand action to prevent misoperation and determine whether the hand action is valid; if so, execute step S33: perform corresponding operations in the current display interface according to the operation requirements and operation intentions.

[0122] In this step, firstly, the operation validity parameters of the hand movement are obtained, the operation validity parameters include the time parameters and movement verification parameters of the hand movement, the movement verification parameters include the movement change state, the gesture combination formed and / or the number of repetitions; then, based on the operation validity parameters, the validity of the hand movement is judged.

[0123] It is understandable that the time parameter is used to determine whether the duration of the gesture meets expectations. For example, some operations require the gesture to last for a certain period of time before it is considered valid; and the action verification parameters include the state of action change, such as whether the speed change and direction change of the gesture meet the normal operation mode, and whether the gesture combination formed is the correct gesture combination. For example, in the dialing interface, the dialing operation needs to be performed in the correct order of numbers and gesture combination, and some operations may require a specific number of repetitions to be considered valid operations. Based on these operation validity parameters, the system comprehensively judges the validity of the hand movement. Only when the hand movement is judged to be valid, the corresponding operation will be performed in the current display interface according to the operation requirements and operation intentions to avoid unnecessary system responses or erroneous operations caused by misoperation. For example, in the shortcut menu operation, if the duration of the fist gesture is less than the specified time or the pressure change is unstable during the fisting process, the system will not call out the shortcut menu to prevent false triggering.

[0124] After passing the anti-misoperation verification, the system performs the corresponding operation in the current display interface according to the determined operation requirements and operation intentions.

[0125] In this embodiment, performing corresponding operations in the current display interface includes:

[0126] Selection operation, including displaying a selection indicator;

[0127] Media interaction operations, including starting and stopping the playback of videos and / or music, adjusting the playback progress of videos and / or music, and confirming the selection of videos and / or music;

[0128] Communication operations, including number selection, triggering dialing, and hanging up calls;

[0129] Map operations, including panning and zooming of the entire or partial map;

[0130] Quick menu operations, including quick menu display and interaction.

[0131] Example:

[0132] In the video selection interface, the selection mark is displayed, such as calling out dots, confirming selection, etc. When the user makes corresponding gestures, the system will perform corresponding operations according to the gesture events, such as displaying a list of videos to choose from. When the user selects a video by moving the palm and making an OK gesture, the selection of the video is confirmed and subsequent loading or playback preparation is performed.

[0133] In the video playback interface, start and stop the video playback, such as playing or pausing with the V gesture, adjusting the playback progress by moving one finger left or right, etc. Accurately control the playback status and progress of the video according to the user's gestures, providing users with a smooth video viewing experience.

[0134] In the music interface, similar playback controls are performed, such as playing or pausing music through specific gestures), adjusting the music progress by moving one finger left and right, and selecting music through palm selection and OK gesture confirmation, etc., to meet the user's various operational needs during music playback.

[0135] In the dialing interface, users can select numbers by moving a single finger to accurately select the dial button, trigger dialing by clicking the button in the air with their finger, and hang up the phone with a thumb gesture, making it convenient for users to perform telephone communication operations.

[0136] On the map page, users can pan the map as a whole or partially by moving their fingers up, down, left, or right, and zoom in and out by pushing and pulling with their palms, allowing users to easily view map information of different areas and scales.

[0137] The shortcut menu operation includes the display of the shortcut menu, such as calling out the shortcut menu by holding the fist for a certain period of time, selecting a function by moving the fist up, down, left, and right, and entering the corresponding function menu by releasing the fist, providing users with a convenient way to quickly access commonly used functions.

[0138] Please refer to Fig.10 , further, in this embodiment, when performing a corresponding operation in the current display interface, it also includes:

[0139] S34. Provide feedback information in response to the target gesture event.

[0140] Specifically, the feedback information includes one or more of visual feedback information, auditory feedback information and tactile feedback information.

[0141] For example, in the video selection interface, when the palm successfully calls out the dot, a short animation effect can appear on the interface, such as the dot flashing or a halo appearing around it, to prompt the user to make a selection. In the music playback interface, when the play or pause operation is performed, the music playback icon on the interface can change its state accordingly, such as changing from a paused icon to a play icon, or vice versa. On the map page, when zooming in or out, a digital prompt of the zoom ratio can be displayed on the edge of the map, allowing users to intuitively understand the zoom level of the map.

[0142] In addition, feedback information can be provided by providing sound prompts. For example, in the video playback interface, when the play button is clicked, a short "beep" sound is emitted to indicate that the operation has been received. In the music interface, when switching songs or performing playback control operations, a short music clip can be played as feedback. In the dialing interface, when the number is successfully dialed, a dial tone is emitted to let the user know that the dialing operation has been completed.

[0143] For devices that support tactile feedback, the device can generate tactile prompts when the user performs certain operations. For example, in the video selection interface, when the user confirms the selection of a video, the device uses an air jet mechanism to spray air at the hand, giving the user a tactile feedback that the operation has been confirmed.

[0144] Please refer to Fig.11 Based on the foregoing embodiments, an embodiment of the present invention provides a gesture interaction system under multiple interfaces, which is used to implement the gesture interaction method under multiple interfaces as described above, including a gesture recognition module 10, a subscription management module 20 and an operation execution module 30.

[0145] The gesture recognition module 10 is used to monitor the user's hand movements and capture gesture events.

[0146] Specifically, when the user makes a hand movement in front of the device screen, the camera first captures the hand image, and the gesture recognition module 10 immediately analyzes the image to extract the shape parameters and position parameters of the palm. If the palm shape meets the preset start gesture features, it is identified as a start gesture. As the palm moves, its trajectory parameters and time parameters are continuously recorded. When the palm movement stops or meets the end conditions, the relevant parameters in the entire hand movement process are encapsulated as a gesture event, and the corresponding function tags are added according to the possible functional role of the gesture in the current interface, such as "video selection trigger" or "map translation operation", etc., and then the gesture event is sent to the subscription management module 20.

[0147] The subscription management module 20 is used to subscribe to a target gesture event according to the functional requirements of the current display interface, and to obtain gesture function parameters of the target gesture event when a target gesture event is generated.

[0148] Specifically, when the system switches to the video playback interface, the subscription management module 20 obtains the functional requirements of the video playback interface from the system configuration, such as the need to support video playback control, progress adjustment and other operations. Then, based on these requirements, the subscription management module 20 filters out gesture events with relevant function tags from the gesture events sent by the gesture recognition module 10 for subscription, such as subscribing to the gesture V event corresponding to the "video playback control" function tag and the finger sliding event corresponding to the "video playback progress adjustment" function tag.

[0149] In addition, when it is detected that the interface is switched to the music interface, the subscription management module 20 immediately cancels the gesture events subscribed in the video playback interface, and instead obtains the functional requirements of the music interface, such as music playback control, progress adjustment, etc., and then subscribes to gesture events that match the functional requirements of the music interface, such as specific gesture events with the "music playback control" function label and finger sliding events with the "music playback progress adjustment" function label. At the same time, the gesture event subscription history of each interface such as the video playback interface and the music interface is stored, so that when the user switches back to these interfaces again, the corresponding subscription status can be quickly restored based on the subscription history.

[0150] The operation execution module 30 is used to execute a corresponding operation in the current display interface according to the target gesture event and the specific gesture parameters.

[0151] Specifically, when the operation execution module 30 receives a gesture V event with a function tag of "video playback control" and its related gesture function parameters sent by the subscription management module 20, it first determines the operation requirement of the video playback or pause corresponding to the gesture in the video playback interface according to the pre-set mapping relationship. Then, based on the gesture function parameters, it further analyzes the user's operation intention, for example, to determine whether it is a simple click to pause or a long press may have other functions, such as fast forward or rewind.

[0152] Before executing the operation, the operation execution module 30 also performs anti-misoperation verification to check the validity of the gesture, such as whether the gesture duration is within a reasonable range, etc. If the verification is passed, the operation execution module 30 will interact with the relevant interface of the video playback application or system to perform the video playback or pause operation, and at the same time trigger corresponding feedback information, so that the user can intuitively feel that the operation has been executed.

[0153] Based on the foregoing embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the gesture interaction method under multiple interfaces as described above are implemented.

[0154] Specifically, when the computer program is executed by the processor, the steps of the gesture interaction method under the multi-interface described above are implemented. The storage medium can be a hard disk, solid-state hard disk, USB flash drive, optical disk, or other medium capable of storing data and programs. When the program in the storage medium is loaded into the computer system and runs, the system can process the gesture interaction according to the steps in the above method, including gesture monitoring, event subscription and processing, operation execution, and feedback information provision, etc., to provide users with a smooth and natural gesture interaction experience in a multi-interface environment.

[0155] Based on the foregoing embodiments, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute any of the above gesture interaction methods under multiple interfaces through the computer program.

[0156] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A gesture interaction method under multiple interfaces, characterized in that: include: Monitor user hand movements and capture gesture events; Subscribe to a target gesture event according to the functional requirements of the current display interface, and when the target gesture event is generated, obtain the gesture function parameters of the target gesture event; According to the target gesture event and the specific gesture parameter, a corresponding operation is performed in the current display interface.

2. The gesture interaction method under multiple interfaces according to claim 1, characterized in that: The monitoring of the user's hand movements and capturing gesture events includes: Monitor the hand movements of the user and obtain basic gesture parameters in the hand movements; the basic gesture parameters include one or more of shape parameters, position parameters, trajectory parameters, time parameters, speed parameters, distance parameters and rotation parameters; Based on the gesture basic parameters, determining a start gesture, an intermediate continuous gesture, and an end gesture of the hand motion; According to the characteristics of the start gesture, the intermediate continuous gesture and the end gesture, the gesture basic parameters are encapsulated into one or more gesture events.

3. The gesture interaction method under multiple interfaces according to claim 2, characterized in that: When the basic gesture parameters are encapsulated into one or more gesture events, it also includes: Adding a function tag to the gesture event; Subscribing to the target gesture event according to the functional requirements of the current display interface includes: Acquire the functional requirements corresponding to the current display interface, and subscribe to gesture events corresponding to the functional tags that match the functional requirements.

4. The gesture interaction method under multiple interfaces according to claim 3, characterized in that: When an interface switch is detected, it also includes: Cancel the currently subscribed target gesture event; The functional requirements corresponding to the switched display interface are obtained, and the gesture events corresponding to the functional tags matching the functional requirements are subscribed.

5. The gesture interaction method under multiple interfaces according to claim 4, characterized in that: Also includes: Store the subscription history of gesture events on each display interface; When the display interface is not switched for the first time, the subscription status of the display interface is restored according to the subscription history.

6. The gesture interaction method under multiple interfaces according to claim 1, characterized in that: The performing a corresponding operation in the current display interface according to the target gesture event and the specific gesture parameter includes: Determine the operation requirements corresponding to the current display interface according to the pre-set mapping relationship between gesture events and operations; Determining the user's operation intention based on the gesture function parameters; According to the operation requirements and operation intentions, corresponding operations are performed in the current display interface.

7. The gesture interaction method under multiple interfaces according to claim 6, characterized in that: Before performing the corresponding operation in the current display interface according to the operation requirement and the operation intention, the method further includes: Performing anti-misoperation verification on the hand movement to determine whether the hand movement is valid; When the hand movement is valid, the corresponding operation is performed in the current display interface according to the operation requirements and operation intentions.

8. The gesture interaction method under multiple interfaces according to claim 6, characterized in that: The verifying the hand movement to prevent misoperation includes: Acquiring operation validity parameters of the hand motion, wherein the operation validity parameters include a time parameter and a motion verification parameter of the hand motion, and the motion verification parameter includes a motion change state, a gesture combination formed, and / or a number of repetitions; Based on the operation validity parameter, the validity of the hand action is determined.

9. The gesture interaction method under multiple interfaces according to claim 1, characterized in that: When performing the corresponding operation in the current display interface, it also includes: Providing feedback information in response to the target gesture event; The feedback information includes one or more of visual feedback information, auditory feedback information and tactile feedback information.

10. The gesture interaction method under multiple interfaces according to claim 1, characterized in that: The interface includes one or more of a video selection interface, a video playback interface, a music interface, a dialing interface, a map page, and a shortcut menu.

11. The gesture interaction method under multiple interfaces according to claim 1, characterized in that: The performing of corresponding operations in the current display interface includes: Selection operation, including displaying a selection indicator; Media interaction operations, including starting and stopping the playback of videos and / or music, adjusting the playback progress of videos and / or music, and confirming the selection of videos and / or music; Communication operations, including number selection, triggering dialing, and hanging up calls; Map operations, including panning and zooming of the entire or partial map; Quick menu operations, including quick menu display and interaction.

12. A gesture interaction system under multiple interfaces, characterized in that: A method for implementing a gesture interaction method under a multi-interface as claimed in any one of claims 1 to 11, comprising: Gesture recognition module, used to monitor user hand movements and capture gesture events; A subscription management module, used to subscribe to a target gesture event according to the functional requirements of the current display interface, and when the target gesture event is generated, obtain the gesture function parameters of the target gesture event; The operation execution module is used to execute a corresponding operation in the current display interface according to the target gesture event and the specific gesture parameter.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the gesture interaction method under multiple interfaces as described in any one of claims 1 to 11 are implemented.

14. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the gesture interaction method under a multi-interface as described in any one of claims 1 to 11 through the computer program.

Images