Gesture recognition method, terminal control method, device, terminal and storage medium

Abstract

This disclosure proposes a gesture recognition method, terminal control method, device, terminal, and storage medium. The gesture recognition method includes: acquiring hand motion information relative to the display screen, and collecting hand images during the acquisition of motion information; identifying hand shape information during the acquisition of motion information based on the hand images; and determining the identified target gesture based on the motion and shape information. By acquiring hand image features and hand motion features, the target gesture is determined by combining hand motion and shape information, enabling interactive operation with the terminal based on the target gesture. This makes the terminal's interactive operation more convenient, effectively improves the terminal's interactive operation efficiency, effectively reduces wear and tear on the terminal's display screen, expands the application scenarios of the terminal's interactive operation, and improves the terminal's interactive operation effect.

Description

Technical Field

[0001] This disclosure relates to the field of terminal technology, and in particular to a gesture recognition method, terminal control method, device, terminal and storage medium. Background Technology

[0002] With the continuous development and upgrading of terminals, their appearance is evolving towards larger sizes and more industrial design (ID) forms. For example, various forms of flexible screens have been developed for terminal displays, such as rollable screens and foldable screens.

[0003] In the actual application of terminals equipped with flexible screens, due to the material and screen size characteristics of flexible screens, the touch-click interaction methods for the display screen in related technologies are more likely to accelerate the wear and tear of the display screen, and the interaction operation is more cumbersome and inconvenient. Summary of the Invention

[0004] This disclosure aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, the purpose of this disclosure is to propose a gesture recognition method, a terminal control method, a device, a terminal, a storage medium, and a computer program product, which makes the interactive operation of the terminal more convenient, effectively improves the interactive operation efficiency of the terminal, effectively reduces the wear and tear on the terminal's display screen, expands the application scenarios of the terminal's interactive operation, and improves the interactive operation effect of the terminal.

[0006] The gesture recognition method proposed in the first aspect of this disclosure includes: acquiring motion information of a hand relative to a display screen, and acquiring a hand image during the acquisition of motion information; identifying the shape information of the hand during the acquisition of motion information based on the hand image; and determining the identified target gesture based on the motion information and the shape information.

[0007] The gesture recognition method proposed in the first aspect of this disclosure acquires hand motion information relative to the display screen and hand images collected during the acquisition of motion information. Based on the hand images, it identifies the hand's shape information during the acquisition of motion information and determines the identified target gesture based on the motion and shape information. By acquiring hand image features and hand motion features, it combines hand motion and shape information to assist in determining the target gesture, enabling interactive operations with the terminal based on the target gesture. This makes the interactive operation method of the terminal more convenient, effectively improves the efficiency of the terminal's interactive operation, effectively reduces the wear and tear on the terminal's display screen, expands the application scenarios of the terminal's interactive operation, and improves the interactive operation effect of the terminal.

[0008] The terminal control method proposed in the second aspect of this disclosure includes: acquiring a recognized target gesture according to the gesture recognition method described above; and controlling the terminal to execute an instruction corresponding to the target gesture according to the target gesture.

[0009] The terminal control method proposed in the second aspect of this disclosure obtains the recognized target gesture according to the gesture recognition method described above, and controls the terminal to execute the instruction corresponding to the target gesture according to the target gesture. It can generate a wide variety of target gestures, which facilitates multi-dimensional control of the terminal display screen based on the target gesture, thereby improving the flexibility, accuracy and comprehensiveness of interactive control.

[0010] The gesture recognition device proposed in the third aspect of this disclosure includes: a first acquisition module, configured to acquire motion information of a hand relative to a display screen, and a hand image acquired during the acquisition of the motion information; a recognition module, configured to recognize the shape information of the hand during the acquisition of the motion information based on the hand image; and a determination module, configured to determine the recognized target gesture based on the motion information and the shape information.

[0011] The gesture recognition device proposed in the third aspect of this disclosure acquires hand motion information relative to the display screen and hand images collected during the acquisition of motion information. Based on the hand images, it identifies the hand's shape information during the acquisition of motion information and determines the identified target gesture based on the motion and shape information. By acquiring hand image features and hand motion features, it combines hand motion and shape information to assist in determining the target gesture, enabling interactive operation with the terminal based on the target gesture. This makes the interactive operation method of the terminal more convenient, effectively improves the efficiency of the terminal's interactive operation, effectively reduces the wear and tear on the terminal's display screen, expands the application scenarios of the terminal's interactive operation, and improves the interactive operation effect of the terminal.

[0012] The fourth aspect of this disclosure discloses a terminal control device, including: a second acquisition module, configured to acquire a recognized target gesture according to the gesture recognition device; and an execution module, configured to control the terminal to execute an instruction corresponding to the target gesture according to the target gesture.

[0013] The terminal control device proposed in the fourth aspect of this disclosure acquires the recognized target gesture according to the gesture recognition method described above, and controls the terminal to execute the instruction corresponding to the target gesture according to the target gesture. It can generate a wide variety of target gestures, which facilitates multi-dimensional control of the terminal display screen based on the target gesture, thereby improving the flexibility, accuracy and comprehensiveness of interactive control.

[0014] A fifth aspect of this disclosure provides a terminal including an under-display sensor, the under-display sensor including a motion information acquisition unit; wherein the motion information acquisition unit is used to acquire motion information of a hand relative to the display screen; the under-display sensor is used to execute the gesture recognition method proposed in the first aspect of this disclosure, or to execute the terminal control method as described in the second aspect.

[0015] A sixth aspect of this disclosure provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the gesture recognition method as proposed in the first aspect of this disclosure, or executes the terminal control method as described in the second aspect.

[0016] A seventh aspect of this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the gesture recognition method as proposed in the first aspect of this disclosure, or performs the terminal control method as described in the second aspect.

[0017] An eighth aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs a gesture recognition method as described in the first aspect of this disclosure, or performs a terminal control method as described in the second aspect.

[0018] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 This is a schematic flowchart of a gesture recognition method proposed in an embodiment of this disclosure;

[0021] Figure 2 This is a schematic flowchart of a gesture recognition method proposed in another embodiment of this disclosure;

[0022] Figure 3 This is a schematic flowchart of a terminal control method proposed in another embodiment of this disclosure;

[0023] Figure 4 This is a schematic diagram of a hand image in an embodiment of this disclosure;

[0024] Figure 5 This is a schematic diagram of the structure of a gesture recognition device according to an embodiment of the present disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of a terminal control device according to another embodiment of this disclosure;

[0026] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present disclosure;

[0027] Figure 8 This is a schematic diagram of another terminal provided in an embodiment of the present disclosure. Detailed Implementation

[0028] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0029] Figure 1 This is a schematic flowchart of a gesture recognition method proposed in one embodiment of the present disclosure.

[0030] This embodiment illustrates the example of a gesture recognition method configured in a gesture recognition device.

[0031] In this embodiment, the gesture recognition method can be configured in a gesture recognition device, which can be set in a terminal. The terminal can be a hardware device with various operating systems, imaging devices, voice acquisition devices, voice processing devices, and voice playback devices, such as a mobile phone, tablet computer, personal digital assistant, or wearable device.

[0032] It should be noted that the execution entity of the embodiments disclosed herein may be, for example, a central processing unit (CPU) in a terminal in terms of hardware, and may be, for example, a related background service in the terminal in terms of software, without limitation.

[0033] The terminal in this embodiment of the disclosure may specifically be a terminal configured with a flexible screen, and there is no limitation thereto.

[0034] like Figure 1 As shown, the terminal control method includes:

[0035] S101: Acquire motion information of the hand relative to the display screen, and capture hand images during the acquisition of motion information.

[0036] The display screen is the terminal's display screen, which can be, for example, a flexible display screen, and there are no restrictions on this.

[0037] Among them, motion information refers to the relative motion information between the hand and the display screen. The motion information of the hand relative to the display screen can be used to describe the actual movement of the hand relative to the display screen, such as the distance of the hand relative to the display screen, the direction of the hand's movement relative to the display screen, or the speed of the hand's movement relative to the display screen, etc., without any limitations.

[0038] In this embodiment of the disclosure, a variety of sensors can be configured for the terminal to assist in obtaining hand motion information relative to the display screen. The various sensors can be, for example, ultrasonic sensors, and a low-power camera device can also be configured. The ultrasonic sensor is used to obtain hand motion information relative to the display screen, and the low-power camera device is used to support image monitoring and processing, so as to realize the acquisition of hand images during the process of obtaining motion information.

[0039] Among them, hand images acquired during the process of acquiring motion information can be combined with motion information to determine the target gesture.

[0040] Among them, the terminal's imaging device can be activated to capture hand images during the acquisition of motion information. Images containing the user's hand can be referred to as hand images.

[0041] For example, an imaging device can capture scene images of the terminal's location, and then send the captured scene images to an image recognition device for analysis. If the scene image includes the user's hand, the image containing the hand can be extracted from the scene image and used as the hand image. Alternatively, the device can first detect whether the hand is close to the display screen based on the hand's motion information relative to the display screen. If it is detected that the hand is close to the display screen, the imaging device can be activated to continuously capture multiple images containing the hand and use them as multiple hand images. There are no restrictions on this.

[0042] In other words, in this embodiment of the present disclosure, the scene image of the scene where the terminal is located can be acquired first based on the imaging device, and then the scene image features (e.g., depth features, color features, brightness features, etc.) of the scene image can be analyzed to detect the actual content contained in the scene image and analyze whether it carries the local hand features of the hand.

[0043] For example, the analyzed scene image features can be compared with the pre-collected hand image features to determine whether the scene image contains a partial hand image. If the scene image features indicate that the scene image contains a partial hand image, the partial hand image can be directly extracted from the scene image and used as the hand image. Then, multiple hand images can be continuously captured in the same way.

[0044] Therefore, by acquiring scene images and extracting scene image features, if a partial hand image containing a hand is determined based on the scene image features, then the partial hand image is used as the hand image, and multiple hand images are continuously captured. This enables the rapid capture of multiple hand images, which are extracted from the scene image. This effectively reduces the consumption of terminal storage resources, facilitates the terminal's gesture recognition by referring to the hand images, ensures recognition efficiency, and can trigger the continuous capture of multiple hand images once a frame of hand image is captured. This can be effectively combined with the actual terminal usage scenario (for example, when a user has interactive operation needs for the terminal, they usually use their hand to approach the terminal screen to perform the corresponding operation), thereby greatly improving the efficiency of hand image acquisition.

[0045] Of course, other possible methods can also be used to obtain multiple hand images, such as capturing scene video and parsing multiple video frames containing hands from the scene video as multiple hand images, without any restrictions.

[0046] The aforementioned imaging device can also be, for example, an infrared imaging device, which, while detecting a hand image, determines whether the hand is a living hand, thereby ensuring the accuracy of terminal control and avoiding false triggering.

[0047] In other words, the embodiments of this disclosure support the acquisition of hand images during the process of acquiring motion information. The motion information and hand images can be combined to determine the target gesture, thereby effectively ensuring the accuracy of control, assisting in accurate learning and modeling to obtain the user's interactive operation intention for the terminal, effectively avoiding misoperation, and improving the interactive operation experience.

[0048] S102: Based on the hand image, identify the shape information of the hand during the process of acquiring motion information.

[0049] After acquiring hand images during the process of obtaining motion information, the shape information of the hand during the process of acquiring motion information can be identified based on the hand images.

[0050] Among them, morphological information can be used to describe the shape of the hand in a hand image, the posture of the hand, etc., without any restrictions.

[0051] This morphological information can be obtained by performing image feature recognition on a single frame of a hand image and analyzing the recognized image features to characterize the hand's morphological dimensions. Alternatively, if the collected hand images are multiple frames, the morphological information of the hand can be determined by continuously recognizing multiple image features. There are no restrictions on this.

[0052] In other words, the morphological information can be obtained by analyzing the morphological changes based on image feature recognition of multiple hand images captured during the acquisition of motion information, without any restrictions.

[0053] S103: Determine the identified target gesture based on motion and morphological information.

[0054] The above-mentioned process involves acquiring hand motion information relative to the display screen, collecting hand images during the acquisition of motion information, and identifying hand shape information during the acquisition of motion information based on the hand images. Then, the target gesture can be determined based on the motion information and shape information.

[0055] It is understood that the aforementioned multiple hand images can be acquired at different time points. Therefore, the hand morphology information in this disclosure can be based on image feature recognition of multiple hand images captured during the acquisition of motion information to analyze the obtained morphological change information (e.g., whether the hand gradually gets bigger, smaller, or the hand size remains constant). Correspondingly, the hand motion information relative to the display screen can also correspond to multiple time points. Thus, the generated target gesture can actually be determined based on the morphological change information and the hand motion information relative to the display screen. This can meet the control requirements in actual terminal application scenarios and help to more accurately identify the user's interactive operation intentions, thereby assisting in achieving accurate and rapid response control of the terminal.

[0056] In this embodiment, by acquiring hand motion information relative to the display screen and hand images collected during the acquisition of motion information, and by identifying hand shape information during the acquisition of motion information based on the hand images, and by determining the identified target gesture based on the motion and shape information, the method combines hand image features and hand motion features to assist in determining the target gesture. This allows for interactive operation with the terminal based on the target gesture, making the interactive operation method of the terminal more convenient, effectively improving the efficiency of the terminal's interactive operation, and effectively reducing the wear and tear on the terminal's display screen. This expands the application scenarios of the terminal's interactive operation and improves the interactive operation effect of the terminal.

[0057] Figure 2 This is a schematic flowchart of a gesture recognition method proposed in another embodiment of this disclosure.

[0058] like Figure 2 As shown, the terminal control method includes:

[0059] S201: Acquire information on the distance change of the hand relative to the display screen, and / or the movement speed information of the hand relative to the display screen, and / or the motion trajectory formed by the hand projected onto a plane parallel to the display screen.

[0060] Among them, the information on the change in distance between the hand and the display screen, and / or the information on the movement speed of the hand relative to the display screen, and / or the motion trajectory formed by the hand projected onto a plane parallel to the display screen, can be used to combine to obtain the motion information of the hand relative to the display screen.

[0061] For example, hand motion information relative to the display screen can be: information on the change in distance between the hand and the display screen, or information on the speed at which the hand moves relative to the display screen, or the motion trajectory formed by the hand projected onto a plane parallel to the display screen.

[0062] For example, hand motion information relative to the display screen could be: information on the change in distance between the hand and the display screen and information on the speed at which the hand moves relative to the display screen.

[0063] For example, hand motion information relative to the display screen can be: information on the change in distance between the hand and the display screen, and the motion trajectory formed by the hand projected onto a plane parallel to the display screen.

[0064] For example, hand motion information relative to the display screen can be: the speed at which the hand moves relative to the display screen and the trajectory of the hand projected onto a plane parallel to the display screen.

[0065] For example, hand motion information relative to the display screen can be: information on the change in distance between the hand and the display screen, information on the speed at which the hand moves relative to the display screen, and the motion trajectory formed by the hand projected onto a plane parallel to the display screen.

[0066] The present disclosure does not limit the above-described combination forms.

[0067] In this embodiment of the disclosure, the information on the change in the distance between the hand and the display screen can be obtained by capturing the change in the distance between the hand and the display screen over a period of time. That is, the distance value between the hand and the display screen at different time points is captured, and then the change of the distance value between multiple time points is analyzed as distance change information (e.g., whether the distance is getting smaller or larger).

[0068] For example, the terminal's ultrasonic sensor can be activated to capture the distance between the hand and the display screen at different times. This distance can be a vertical distance, and there are no restrictions on this.

[0069] In this embodiment of the disclosure, the information on the movement speed of the hand relative to the display screen can be the speed of the hand movement and / or the acceleration of the hand movement. That is, the movement speed information can be the speed of the hand movement, or the acceleration of the hand movement, or a combination of the two, and there is no limitation thereto.

[0070] For example, the terminal's speed detection sensor can be activated to capture the speed of hand movement or the acceleration of hand movement, without any restrictions.

[0071] In this embodiment of the disclosure, obtaining the motion trajectory of the hand projected onto a plane parallel to the display screen can be achieved by: pre-calibrating a spatial rectangular coordinate system for the display screen, wherein the plane of the display screen can correspond to a plane defined by any two coordinate axes in the spatial rectangular coordinate system; if the hand moves at a position with a certain distance relative to the plane of the display screen, a reference plane can be configured for the hand, and the reference plane can be kept parallel to the plane where the display screen is located; then, the three-dimensional motion trajectory of the hand in space can be captured, and the three-dimensional motion trajectory can be projected onto the reference plane to obtain the motion trajectory of the hand projected onto a plane parallel to the display screen, without any limitation.

[0072] In other words, compared to the two-dimensional motion estimation formed by sliding the hand on the display screen surface in related technologies, the embodiments of this disclosure support capturing the three-dimensional motion trajectory of the hand in space, and by recognizing the motion trajectory formed by the hand projected onto a plane parallel to the display screen, it can help to identify more accurate three-dimensional motion information.

[0073] Of course, any other possible method can be used to obtain the motion trajectory formed by the hand projected onto a plane parallel to the display screen. For example, a video of the hand during the hand movement can be captured, and multiple video frames of the hand video can be analyzed for depth to determine the depth information corresponding to the hand. Then, the depth information of the hand in multiple video frames can be referenced to model and learn the motion trajectory formed by the hand projected onto a plane parallel to the display screen. There are no restrictions on this.

[0074] For example, suppose the initial position of the hand is at a point in space, which is a certain distance away from the display screen. This point can be the center of a cube. Imagine the hand moving within a virtual three-dimensional cube. The three-dimensional trajectory of the hand can be along the diagonal inside the cube, moving diagonally upward to the left, diagonally downward to the left, diagonally upward to the right, and diagonally downward to the right. As the relative distance between the hand and the display screen changes, the diagonally upward to the left movement can also include: moving diagonally upward to the left forward and diagonally upward to the left backward. Here, the direction closer to the display screen indicates "forward," and the direction further away from the display screen indicates "backward." The diagonally downward to the left movement can also include: moving diagonally downward to the left forward and diagonally downward to the left backward. The diagonally upward to the right movement can also include: moving diagonally upward to the right forward and diagonally upward to the right backward. The diagonally downward to the right movement can also include: moving diagonally downward to the right forward and diagonally downward to the right backward. Of course, the three-dimensional trajectory of the hand relative to the display screen is not limited to these. It can also be any other possible form of three-dimensional trajectory, such as an arc trajectory, a circular trajectory, etc. There are no restrictions on this.

[0075] S202: The acquired distance change information, and / or movement speed information, and / or movement trajectory are used as motion information.

[0076] For example, hand motion information relative to the display screen can be: information on the change in distance between the hand and the display screen, or information on the speed at which the hand moves relative to the display screen, or the motion trajectory formed by the hand projected onto a plane parallel to the display screen.

[0077] For example, hand motion information relative to the display screen could be: information on the change in distance between the hand and the display screen and information on the speed at which the hand moves relative to the display screen.

[0078] For example, hand motion information relative to the display screen can be: information on the change in distance between the hand and the display screen, and the motion trajectory formed by the hand projected onto a plane parallel to the display screen.

[0079] For example, hand motion information relative to the display screen can be: the speed at which the hand moves relative to the display screen and the trajectory of the hand projected onto a plane parallel to the display screen.

[0080] For example, hand motion information relative to the display screen can be: information on the change in distance between the hand and the display screen, information on the speed at which the hand moves relative to the display screen, and the motion trajectory formed by the hand projected onto a plane parallel to the display screen.

[0081] The present disclosure does not limit the above-described combination forms.

[0082] In this embodiment of the disclosure, by using the acquired distance change information, and / or movement speed information, and / or movement trajectory as motion information, the flexibility of motion information capture can be effectively improved, supporting the capture of three-dimensional motion information of the hand in space. Thus, the three-dimensional motion information of the hand in space is used to assist in the subsequent recognition of target gestures, which can effectively improve the flexibility of terminal interaction operation, improve the efficiency of target gesture generation, effectively assist in expanding the application scenarios of gesture recognition, and improve the user experience.

[0083] Optionally, in some embodiments, the acquired distance change information and motion trajectory are used as motion information, including:

[0084] If the distance change information indicates a change in the distance between the hand and the display screen, and the movement trajectory exceeds the area covered by a preset size range, then a candidate gesture trajectory that matches the distance change information and the movement trajectory is determined from the candidate gesture trajectories and used as the motion information. The candidate gesture trajectory includes at least one of the following: moving closer to the display screen and towards a first direction relative to the center of the display screen; moving further away from the display screen and towards a first direction relative to the center of the display screen; moving closer to the display screen and towards a direction between the first and second directions relative to the center of the display screen; and moving further away from the display screen and towards a direction between the first and second directions relative to the center of the display screen. The first direction includes at least one direction determined from the center of the display screen to the midpoint of any side of the display screen; the second direction is a direction determined from the center of the display screen to the midpoint of the side of the display screen adjacent to the first direction.

[0085] In other words, in some application scenarios, the terminal display screen may be configured with relatively high recognition accuracy, but the user may not be able to make particularly accurate motion trajectories during actual operation. This embodiment of the disclosure also provides a redundant recognition mechanism for motion trajectories, that is, for some motion trajectories formed by the user's hand, the gesture trajectory that matches the motion trajectory can be determined as motion information, thereby effectively avoiding misrecognition of motion information.

[0086] The area covered by the preset size range can correspond to the size of the display screen. The area covered by the preset size range can be, for example, the area covered by the size range of the display screen. This area can specifically be the area projected onto the reference plane formed based on the spatial rectangular coordinate system in the above example.

[0087] Since the movement trajectory of the hand is also projected onto the reference plane, the timing for triggering the redundant recognition mechanism can be determined based on whether the movement trajectory exceeds the area covered by the preset size.

[0088] The aforementioned alternative gesture trajectories can be pre-defined. For example, in the example above, the hand can move diagonally upward to the left, diagonally downward to the left, diagonally upward to the right, and diagonally downward to the right along the diagonal inside the virtual cube. There are no restrictions on this.

[0089] This allows the system to determine, based on the user's actual movement trajectory during operation, a candidate gesture trajectory that matches both distance change information and movement trajectory from the available gesture trajectories, and use this as motion information.

[0090] For example, if the user's actual movement trajectory is close to moving diagonally upward to the left, then the diagonal upward movement to the left is identified as movement information; if the user's actual movement trajectory is close to moving diagonally downward to the left, then the diagonal downward movement to the left is identified as movement information, and there is no restriction on this.

[0091] Of course, the redundancy identification mechanism can also be flexibly configured according to actual usage needs, and there are no restrictions on this.

[0092] In this embodiment of the disclosure, the alternative gesture trajectory includes at least one of the following: moving closer to the display screen and towards a first direction relative to the center of the display screen; moving further away from the display screen and towards a first direction relative to the center of the display screen; moving closer to the display screen and towards a direction between the first and second directions relative to the center of the display screen; and moving further away from the display screen and towards a direction between the first and second directions relative to the center of the display screen. The first direction includes at least one direction determined by the center of the display screen to the midpoint of any side of the display screen. The second direction is a direction determined by the center of the display screen to the midpoint of the side of the display screen adjacent to the first direction.

[0093] Since the first direction includes at least the direction determined by the center of the display screen to the midpoint of any side of the display screen, the movement towards the display screen and relative to the center of the display screen in the first direction can be understood as: moving to the left front, moving to the right front, moving to the up front, and moving to the down front.

[0094] Since the first direction includes at least the direction determined by the center of the display screen to the midpoint of any side of the display screen, the movement of the distance from the display screen and the movement relative to the center of the display screen in the first direction can be understood as: moving to the left rear, moving to the right rear, moving to the upper rear, and moving to the lower rear.

[0095] Since the first direction includes at least the direction determined by the center of the display screen to the midpoint of any side of the display screen, and the second direction is the direction determined by the center of the display screen to the midpoint of the side of the display screen adjacent to the first direction, the movement closer and closer to the display screen and relative to the center of the display screen in the direction between the first and second directions can be understood as: moving diagonally forward to the left (upper left or lower left) and moving diagonally forward to the right (upper right or lower right).

[0096] Since the first direction includes at least the direction determined by the center of the display screen to the midpoint of any side of the display screen, and the second direction is the direction determined by the center of the display screen to the midpoint of the side of the display screen adjacent to the first direction, the movement away from the display screen and relative to the center of the display screen in the direction between the first and second directions can be understood as: moving diagonally backward to the left (upper left or lower left) and moving diagonally backward to the right (upper right or lower right).

[0097] The direction closer to the display screen indicates "forward," and the direction further away from the display screen indicates "backward."

[0098] Therefore, this embodiment of the present disclosure can provide a flexible configuration method for alternative gesture trajectories. When a candidate gesture trajectory that meets the distance change information and motion trajectory is determined from the alternative gesture trajectories, the alternative gesture trajectory can be selected by referring to the similarity between the actual motion trajectory made by the user's hand and the alternative gesture trajectory. For example, the similarity can be determined based on the distance change information and the actual motion trajectory made by the hand, without any limitation.

[0099] Optionally, in some embodiments, the acquired movement speed information is used as motion information, including: using the speed of hand movement and / or the acceleration of hand movement measured by a distance sensor as motion information; and / or, determining the spatial position of the hand at different times based on distance change information and movement trajectory, determining the speed of hand movement and / or the acceleration of hand movement based on the spatial position of the hand at different times, and using the determined speed of hand movement and / or the acceleration of hand movement as motion information.

[0100] For example, the acquired movement speed information includes the speed of hand movement measured by a distance sensor.

[0101] For example, the acquired movement speed information includes the acceleration of hand movement measured by a distance sensor.

[0102] For example, the acquired movement speed information includes: the speed of hand movement measured by a distance sensor, and the acceleration of hand movement measured by a distance sensor.

[0103] For example, the acquired movement speed information includes: the speed of hand movement and / or the acceleration of hand movement determined by the spatial position of the hand at different times.

[0104] The “and / or” above indicates that one of the three, or any two or three of them, can be selected, without restriction.

[0105] Therefore, in this embodiment of the present disclosure, movement speed information can be flexibly obtained. It can not only assist in determining the target gesture based on the speed of hand movement, but also assist in determining the target gesture based on the acceleration of hand movement, and can also determine the target gesture based on the combination of the speed and acceleration of hand movement. Furthermore, the measurement methods for the speed and acceleration of hand movement are also flexible and diverse, thereby effectively expanding the application scenarios of gesture recognition and improving the flexibility of gesture recognition methods.

[0106] S203: Hand images acquired during the process of acquiring motion information.

[0107] S204: Based on the hand image, identify the shape information of the hand during the process of acquiring motion information.

[0108] In this embodiment of the disclosure, the motion information may correspond to the hand image. The correspondence means that the motion information is the actual movement of the hand included in the hand image that is acquired at the same time as the hand image is acquired. That is, while acquiring multiple hand images, multiple motion information corresponding to each of the multiple hand images is acquired, so that the motion information can be matched with the hand image, which facilitates the subsequent analysis of accurate target gestures and ensures the accuracy of terminal control.

[0109] S205: Determine the identified target gesture based on motion and morphological information.

[0110] The descriptions of S203-S205 can be found in the above embodiments, and will not be repeated here.

[0111] In this embodiment of the disclosure, a method for flexibly analyzing target gestures is also provided, including: determining the identified target gesture based on at least one morphological information identified under the same motion information; and / or determining the identified target gesture based on at least one motion information identified under the same morphological information; and / or determining the identified target gesture based on a combination of multiple motion information and multiple morphological information identified.

[0112] In other words, the recognized motion information can be of various types, and the morphological information can also be of various types. Furthermore, corresponding gestures can be pre-configured for different combinations of motion information and morphological information to obtain multiple candidate combinations and candidate gestures corresponding to each candidate combination. A correspondence table is then formed based on the multiple candidate combinations and the candidate gestures corresponding to each candidate combination.

[0113] Therefore, after obtaining motion and morphological information, matching candidate combinations can be determined from the correspondence table based on the motion and morphological information. Then, the candidate gestures corresponding to the matching candidate combinations are used as the target gestures without any restrictions.

[0114] The above correspondence table can also be configured with gesture numbers corresponding to different candidate gestures, so that the gesture number of the target gesture can be read to call the target operation corresponding to the target gesture.

[0115] The above-mentioned method determines the identified target gesture based on at least one morphological information identified under the same motion information; and / or, determines the identified target gesture based on at least one motion information identified under the same morphological information; and / or, determines the identified target gesture based on a combination of multiple motion information and multiple morphological information identified. This method can assist in accurately identifying the user's interactive operation intention and can generate a rich variety of target gestures, facilitating multi-dimensional control of the terminal display screen based on the target gestures, thereby improving the flexibility, accuracy, and comprehensiveness of interactive control.

[0116] In this embodiment, by acquiring hand motion information relative to the display screen and hand images captured during the motion information acquisition process, and based on the hand images, identifying the hand's shape information during the motion information acquisition process, and determining the identified target gesture based on the motion and shape information, the acquisition of hand image features and hand motion features enables the combined determination of the target gesture using hand motion and shape information. This allows for interactive operations with the terminal based on the target gesture, making the terminal's interactive operation method more convenient, effectively improving the terminal's interactive operation efficiency, and effectively reducing the wear and tear on the terminal's display screen. This expands the application scenarios of the terminal's interactive operation and enhances the terminal's interactive operation effect. By using the acquired distance change information, and / or movement speed information, and / or movement trajectory as motion information, the flexibility of motion information capture can be effectively improved. This supports capturing the three-dimensional motion information of the hand in space, thereby using the three-dimensional motion information of the hand in space to assist in the subsequent recognition of the target gesture. This effectively improves the flexibility of the terminal's interactive operation, increases the efficiency of target gesture generation, effectively expands the application scenarios of gesture recognition, and enhances the user experience. It also provides a redundant recognition mechanism for motion trajectories. For the motion trajectories formed by the user's hands, it can identify the hand gesture trajectory that matches the motion trajectory as motion information, thus effectively avoiding misidentification of motion information. It can flexibly acquire movement speed information, not only based on the speed of hand movement to assist in determining the target gesture, but also based on the acceleration of hand movement, and even based on a combination of hand movement speed and acceleration. Furthermore, the measurement methods for hand movement speed and acceleration are flexible and diverse, effectively expanding the application scenarios of gesture recognition and improving the flexibility of gesture recognition methods. The target gesture is determined based on at least one morphological information identified under the same motion information; and / or, the target gesture is determined based on at least one motion information identified under the same morphological information; and / or, the target gesture is determined based on a combination of multiple motion information and multiple morphological information identified. This helps to accurately identify the user's interactive operation intention and can generate a rich variety of target gestures, facilitating multi-dimensional control of the terminal display screen based on the target gestures, thereby improving the flexibility, accuracy, and comprehensiveness of interactive control.

[0117] Figure 3 This is a schematic flowchart of a terminal control method proposed in another embodiment of this disclosure.

[0118] This embodiment illustrates the example of a terminal control method configured in a terminal control device.

[0119] In this embodiment, the terminal control method can be configured in a terminal control device, which can be located in a terminal. The terminal can be a hardware device such as a mobile phone, tablet computer, personal digital assistant, or wearable device that has various operating systems, imaging devices, voice acquisition devices, voice processing devices, and voice playback devices.

[0120] It should be noted that the execution entity of the embodiments disclosed herein may be, for example, a central processing unit (CPU) in a terminal in terms of hardware, and may be, for example, a related background service in the terminal in terms of software, without limitation.

[0121] The terminal in this embodiment of the disclosure may specifically be a terminal configured with a flexible screen, and there is no limitation thereto.

[0122] like Figure 3 As shown, the terminal control method includes:

[0123] S301: Obtain the recognized target gesture according to the gesture recognition method described above.

[0124] Target gestures, for example:

[0125] The target gesture can be any gesture that moves closer to or further away from the target, or a gesture that accelerates upwards, downwards, leftwards, or rightwards. The target gesture can also be any other possible gesture shape, without any restrictions.

[0126] S302: The terminal executes the instruction corresponding to the target gesture based on the target gesture.

[0127] The instructions corresponding to the target gestures can be pre-configured. For example, target gesture A matches instruction A, target gesture B matches instruction B, and target gesture C matches instruction C. There are no restrictions on this.

[0128] Therefore, after recognizing the target gesture, the terminal can execute the command corresponding to the target gesture.

[0129] For example:

[0130] Example 1: Approaching / Relocating Target Gestures. When the target gesture determination activation condition is met, if the hand in the hand image acquired by the ultrasonic sensor gradually increases in size, and the relative distance value D detected by the ultrasonic sensor correspondingly gradually decreases, the system determines that the target gesture is approaching the target and reports it to the terminal processor. The terminal processor can then execute the control action corresponding to the target gesture, such as enlarging the image displayed on the screen. Similarly, if the hand in the hand image gradually decreases in size, and the detected relative distance value D increases accordingly, it is determined to be a receding target gesture, and the terminal processor executes the corresponding control action, such as shrinking the image displayed on the screen.

[0131] Example 2: Target gesture judgment based on up / down / left / right and instantaneous acceleration. The direction of movement corresponding to the target gesture is determined based on continuously acquired hand images. At the same time, the ultrasonic sensor, combined with the acquired instantaneous acceleration, determines the change speed of the user interface (UI) on the terminal display. For example, if the target gesture is judged to be to the left and the instantaneous acceleration is large (the magnitude of the instantaneous acceleration needs to be set as a comparison benchmark), the terminal can perform a left-side split-screen operation. The transition speed of the UI can be adapted by referring to the different reported acceleration values, making the control processing logic closer to the actual user operation experience.

[0132] Based on the explanations in the above embodiments, the use of ultrasonic sensors and low-power camera devices can effectively reduce the probability of misjudgment when relying solely on camera devices to collect hand images. At the same time, since the target gesture is determined by incorporating the motion and shape information of the hand, it can also provide more options for gesture interaction processing methods, and has good applicability.

[0133] In this embodiment of the disclosure, in order to avoid consuming terminal computing resources, scene images can also be captured; if the scene image includes a local area image of the hand, the target gesture is obtained according to the gesture recognition method.

[0134] In other words, the target gesture is only obtained by the gesture recognition method when the hand can be recognized from the scene image. If the hand is not included in the scene image, the target gesture is not recognized.

[0135] For example, a low-power camera device can monitor and capture scene images in real time. When the scene image features captured in the field of view (FOV) indicate the presence of a hand, the low-power camera device can be triggered to enter Active mode to track gesture images and continuously acquire multiple hand images. While acquiring the current hand image, an ultrasonic sensor is triggered to detect the relative distance and movement speed information (e.g., instantaneous acceleration) between the hand and the terminal's display screen. If the relative distance value D is less than a set threshold L, it is determined that the user is performing a normal interactive operation on the terminal, rather than a false trigger caused by a distant hand wave. This effectively avoids false triggers and improves the accuracy of interactive operation control.

[0136] When the above-mentioned camera device and ultrasonic sensor work simultaneously, the real-time hand images captured can be as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of a hand image in an embodiment of this disclosure.

[0137] In this embodiment of the disclosure, in order to further ensure the accuracy of interactive operations, ensure the effect of terminal control, and avoid accidental triggering by users, the gesture movement feature pattern is also learned. This can be done by obtaining the relative distance value of the hand relative to the display screen and obtaining the movement speed information of the hand relative to the display screen. If the relative distance value is less than or equal to the distance threshold and the movement speed information meets the set conditions, then the recognized target gesture is obtained according to the gesture recognition method.

[0138] Optionally, in some embodiments, the movement speed information of the hand relative to the display screen is obtained. If the movement speed information indicates that the hand changes at a non-uniform or non-linear speed, the target gesture is obtained according to the gesture recognition method. This enables the determination of whether it is a false trigger by referring to the actual situation of the user's hand operating the terminal display screen. This can effectively help determine the appropriate timing for target gesture recognition, effectively avoid false triggers, and reduce the computational resource consumption of gesture generation and recognition.

[0139] In this embodiment, the target gesture is only acquired using the gesture recognition method when a hand can be identified from the scene image. If the scene image does not include a hand, the target gesture is not recognized, thus avoiding the consumption of terminal computing resources. By learning the characteristics of gesture movement, the relative distance between the hand and the display screen can be obtained, along with the hand's movement speed relative to the display screen. If the relative distance is less than or equal to a distance threshold, and the movement speed information meets set conditions, the target gesture is acquired using the gesture recognition method. This further ensures the accuracy of interactive operations, the effectiveness of terminal control, and avoids accidental triggering by the user. By acquiring the hand's movement speed information relative to the display screen, if the movement speed information indicates that the hand is changing at a non-uniform or non-linear speed, the target gesture is acquired using the gesture recognition method. This allows for the determination of whether a false trigger has occurred by referencing the actual situation of the user's hand operating the terminal display screen, effectively assisting in determining the appropriate timing for target gesture recognition. This effectively avoids false triggering while reducing the consumption of computing resources for gesture generation and recognition. By acquiring the identified target gesture according to the gesture recognition method described above, and controlling the terminal to execute the corresponding instruction based on the target gesture, a rich variety of target gestures can be generated. This facilitates multi-dimensional control of the terminal display screen based on the target gesture, improving the flexibility, accuracy, and comprehensiveness of interactive control.

[0140] Figure 5 This is a schematic diagram of the structure of a gesture recognition device according to an embodiment of the present disclosure.

[0141] like Figure 5 As shown, the gesture recognition device 50 includes:

[0142] The first acquisition module 501 is used to acquire motion information of the hand relative to the display screen, as well as hand images captured during the acquisition of motion information.

[0143] The recognition module 502 is used to recognize the shape information of the hand during the acquisition of motion information based on the hand image;

[0144] The determination module 503 is used to determine the identified target gesture based on motion information and morphological information.

[0145] In some embodiments of this disclosure, the first acquisition module 501 is specifically used for:

[0146] Acquire information on the change in distance between the hand and the display screen, and / or the speed of movement of the hand relative to the display screen, and / or the motion trajectory formed by the hand projected onto a plane parallel to the display screen;

[0147] The acquired distance change information, and / or movement speed information, and / or movement trajectory are used as motion information.

[0148] In some embodiments of this disclosure, the first acquisition module 501 is specifically used for:

[0149] If the distance change information indicates a change in the distance between the hand and the display screen, and the movement trajectory exceeds the area covered by the preset size range, then a candidate gesture trajectory that matches the distance change information and the movement trajectory is determined from the candidate gesture trajectories and used as the motion information.

[0150] Among them, the alternative gesture trajectory includes at least one of the following: moving closer and closer to the display screen and moving in a first direction relative to the center of the display screen; moving further and further away from the display screen and moving in a first direction relative to the center of the display screen; moving closer and closer to the display screen and moving in a direction between the first and second directions relative to the center of the display screen; and moving further and further away from the display screen and moving in a direction between the first and second directions relative to the center of the display screen.

[0151] The first direction includes at least the direction determined by the center of the display screen to the midpoint of any side of the display screen; the second direction is the direction determined by the center of the display screen to the midpoint of the side of the display screen adjacent to the first direction.

[0152] In some embodiments of this disclosure, the first acquisition module 501 is specifically used for:

[0153] The speed and / or acceleration of hand movement, measured by a distance sensor, are used as motion information.

[0154] And / or, determine the spatial position of the hand at different times based on distance change information and movement trajectory, determine the speed and / or acceleration of the hand movement based on the spatial position of the hand at different times, and use the determined speed and / or acceleration of the hand movement as motion information.

[0155] In some embodiments of this disclosure, the determining module 503 is specifically used for:

[0156] The identified target gesture is determined based on at least one morphological information identified under the same motion information;

[0157] And / or, based on at least one motion information identified under the same morphological information, determine the identified target gesture;

[0158] And / or, based on the combination of multiple motion information and multiple morphological information identified, the identified target gesture is determined.

[0159] It should be noted that the foregoing explanation of the gesture recognition method embodiment also applies to the gesture recognition device of this embodiment, and will not be repeated here.

[0160] In this embodiment, by acquiring hand motion information relative to the display screen and hand images collected during the acquisition of motion information, and by identifying hand shape information during the acquisition of motion information based on the hand images, and by determining the identified target gesture based on the motion and shape information, the method combines hand image features and hand motion features to assist in determining the target gesture. This allows for interactive operation with the terminal based on the target gesture, making the interactive operation method of the terminal more convenient, effectively improving the efficiency of the terminal's interactive operation, and effectively reducing the wear and tear on the terminal's display screen. This expands the application scenarios of the terminal's interactive operation and improves the interactive operation effect of the terminal.

[0161] Figure 6 This is a schematic diagram of the structure of a gesture recognition device according to an embodiment of the present disclosure.

[0162] like Figure 6 As shown, the terminal control device 60 includes:

[0163] The second acquisition module 601 is used to acquire the recognized target gesture according to the gesture recognition device described above;

[0164] The execution module 602 is used to control the terminal to execute instructions corresponding to the target gesture based on the target gesture.

[0165] In some embodiments of this disclosure, the second acquisition module 601 is specifically used for:

[0166] Capture scene images;

[0167] If the scene image includes a local area image of a hand, the target gesture is obtained based on the gesture recognition device.

[0168] In some embodiments of this disclosure, the second acquisition module 601 is specifically used for:

[0169] Obtain the relative distance between the hand and the display screen, and obtain the hand's movement speed information relative to the display screen;

[0170] If the relative distance value is less than or equal to the distance threshold, and the movement speed information meets the set conditions, then the target gesture is obtained according to the gesture recognition device.

[0171] In some embodiments of this disclosure, the second acquisition module 601 is specifically used for:

[0172] Obtain information on the speed of hand movement relative to the display screen;

[0173] If the movement speed information indicates that the hand is changing at a non-uniform or non-linear speed, then the target gesture is obtained by the gesture recognition device.

[0174] It should be noted that the foregoing explanation of the terminal control method embodiment also applies to the terminal control device of this embodiment, and will not be repeated here.

[0175] In this embodiment, by obtaining the identified target gesture according to the gesture recognition method described above, and controlling the terminal to execute the instruction corresponding to the target gesture, a rich variety of target gestures can be generated. This facilitates multi-dimensional control of the terminal display screen based on the target gesture, improving the flexibility, accuracy, and comprehensiveness of interactive control.

[0176] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present disclosure.

[0177] The terminal includes:

[0178] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0179] When the processor 702 executes the program, it implements the gesture recognition method or the terminal control method provided in the above embodiments.

[0180] In one possible implementation, the terminal also includes:

[0181] Communication interface 703 is used for communication between memory 701 and processor 702.

[0182] The memory 701 is used to store computer programs that can run on the processor 702.

[0183] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0184] The processor 702 is used to implement the gesture recognition method or the terminal control method of the above embodiments when executing a program.

[0185] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0186] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0187] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0188] Figure 8 This is a schematic diagram of another terminal provided in an embodiment of the present disclosure.

[0189] like Figure 8 As shown, the terminal 80 includes:

[0190] Under-display sensor 801, the under-display sensor 801 includes a motion information acquisition unit 8011; wherein,

[0191] The motion information acquisition unit 8011 is used to acquire motion information of the hand relative to the display screen;

[0192] The under-display sensor 801 is used to perform the gesture recognition method and / or the terminal control method described above.

[0193] In some embodiments of this disclosure, the under-display sensor 801 includes an under-display camera 8012, wherein,

[0194] The under-display camera 8012 is used to capture hand images during the process of the motion information acquisition unit 8011 acquiring the motion information.

[0195] The under-display camera 8012 can be a low-power camera device.

[0196] Of course, the under-display camera 8012 can also be configured as any other possible type of camera device, without any restrictions.

[0197] In some embodiments of this disclosure, the motion information acquisition unit 8011 includes an ultrasonic sensor 80111.

[0198] The ultrasonic sensor 80111 can be a narrow-slit ultrasonic sensor, and there are no restrictions on this.

[0199] Of course, the motion information acquisition unit 8011 can also be configured as any other possible type of sensor, without any restrictions.

[0200] In other words, in this embodiment of the present disclosure, it is possible to use an ultrasonic sensor to acquire hand motion information relative to the display screen, and to use a low-power camera device to support image monitoring and processing, so as to acquire hand images during the acquisition of motion information.

[0201] It should be noted that the foregoing explanations of the gesture recognition method and / or terminal control method embodiments also apply to the terminal of this embodiment, and will not be repeated here.

[0202] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the gesture recognition method or terminal control method described above.

[0203] To implement the above embodiments, this disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the above-described gesture recognition method and / or terminal control method.

[0204] It should be noted that in the description of this disclosure, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0205] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0206] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0207] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0208] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0209] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0210] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0211] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A gesture recognition method, characterized in that, The method includes: The system acquires hand motion information relative to the display screen, as well as hand images captured during the acquisition of the motion information. The motion information includes distance change information, movement speed information, and a motion trajectory formed by the hand projected onto a plane parallel to the display screen. If the distance change information indicates a change in distance between the hand and the display screen, and the motion trajectory exceeds the area covered by a preset size range, then a candidate gesture trajectory that matches the distance change information and the motion trajectory is determined from the candidate gesture trajectories and used as the motion information. The alternative gesture trajectory includes at least one of the following: moving closer to the display screen and moving in a first direction relative to the center of the display screen; moving further away from the display screen and moving in a first direction relative to the center of the display screen; moving closer to the display screen and moving in a direction between the first and second directions relative to the center of the display screen; and moving further away from the display screen and moving in a direction between the first and second directions relative to the center of the display screen. The first direction includes at least: a direction determined from the center of the display screen to the midpoint of any side of the display screen; the second direction is: a direction determined from the center of the display screen to the midpoint of the side of the display screen adjacent to the first direction. Based on the hand image, the shape information of the hand during the acquisition of the motion information is identified, wherein the shape information is based on image feature recognition of multiple hand images captured during the acquisition of motion information, in order to analyze the obtained shape change information. The identified target gesture is determined based on the motion information and the morphological information; wherein, it includes: The identified target gesture is determined based on at least one morphological information identified under the same motion information; And / or, based on at least one motion information identified under the same morphological information, determine the identified target gesture; And / or, based on the combination of multiple motion information and multiple morphological information identified, the identified target gesture is determined.

2. The gesture recognition method according to claim 1, characterized in that, The acquired movement speed information is used as the motion information, including: And / or, determine the spatial position of the hand at different times based on the distance change information and the motion trajectory, determine the speed of hand movement and / or the acceleration of hand movement based on the spatial position of the hand at different times, and use the determined speed of hand movement and / or acceleration of hand movement as the motion information.

3. A terminal control method, characterized in that, include: The gesture recognition method according to any one of claims 1-2 obtains the recognized target gesture; The terminal executes instructions corresponding to the target gesture according to the target gesture.

4. The terminal control method according to claim 3, characterized in that, The target gesture is obtained according to the gesture recognition method, including: Capture scene images; If the scene image includes a local area image of a hand, then the target gesture is obtained according to the gesture recognition method.

5. The terminal control method according to claim 3, characterized in that, The target gesture is obtained according to the gesture recognition method, including: Obtain the relative distance value of the hand with respect to the display screen, and obtain the moving speed information of the hand with respect to the display screen; If the relative distance value is less than or equal to the distance threshold, and the movement speed information meets the set conditions, then the recognized target gesture is obtained according to the gesture recognition method.

6. The terminal control method according to claim 3, characterized in that, The target gesture is obtained according to the gesture recognition method, including: Obtain the movement speed information of the hand relative to the display screen; If the movement speed information indicates that the hand changes at a non-uniform, non-linear speed, then the target gesture is obtained according to the gesture recognition method.

7. A gesture recognition device, characterized in that, The device includes: The first acquisition module is used to acquire motion information of the hand relative to the display screen, and hand images captured during the acquisition of the motion information. The motion information includes distance change information, movement speed information, and motion trajectory formed by the hand projected onto a plane parallel to the display screen. If the distance change information indicates a change in distance between the hand and the display screen, and the motion trajectory exceeds the area covered by a preset size range, then a candidate gesture trajectory that matches the distance change information and the motion trajectory is determined from the candidate gesture trajectories and used as the motion information. The alternative gesture trajectory includes at least one of the following: moving closer to the display screen and moving in a first direction relative to the center of the display screen; moving further away from the display screen and moving in a first direction relative to the center of the display screen; moving closer to the display screen and moving in a direction between the first and second directions relative to the center of the display screen; and moving further away from the display screen and moving in a direction between the first and second directions relative to the center of the display screen. The first direction includes at least: a direction determined from the center of the display screen to the midpoint of any side of the display screen; the second direction is: a direction determined from the center of the display screen to the midpoint of the side of the display screen adjacent to the first direction. The recognition module is used to identify the shape information of the hand during the process of acquiring the motion information based on the hand image. The shape information is based on image feature recognition of multiple hand images captured during the process of acquiring motion information, so as to analyze the obtained shape change information. The determination module is used to determine the identified target gesture based on the motion information and the morphological information; The determination module is used to determine the identified target gesture based on at least one type of morphological information identified under the same motion information; And / or, based on at least one motion information identified under the same morphological information, determine the identified target gesture; And / or, based on the combination of multiple motion information and multiple morphological information identified, the identified target gesture is determined.

8. A terminal control device, characterized in that, include: The second acquisition module is used to acquire the recognized target gesture by the gesture recognition device according to claim 7. The execution module is used to execute instructions corresponding to the target gesture on the control terminal according to the target gesture.

9. A terminal, characterized in that, include: The under-display sensor includes: a motion information acquisition unit; wherein, The motion information acquisition unit is used to acquire motion information of the hand relative to the display screen; The under-display sensor is used to perform the gesture recognition method according to any one of claims 1-2, and / or the terminal control method according to any one of claims 3-6.

10. The terminal as described in claim 9, characterized in that, The under-display sensor includes an under-display camera, wherein... The under-display camera is used to capture hand images during the process of the motion information acquisition unit acquiring motion information.

11. The terminal as described in claim 9, characterized in that, The motion information acquisition unit includes an ultrasonic sensor.

12. A terminal, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the gesture recognition method as described in any one of claims 1-2, and / or implements the terminal control method as described in any one of claims 3-6.

13. A storage medium, wherein when instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to perform the gesture recognition method as described in any one of claims 1-2, and / or to perform the terminal control method as described in any one of claims 3-6.

Images