Equipment control method and device, electronic equipment and storage medium

Abstract

The invention relates to an equipment control method and device, electronic equipment and a storage medium, and relates to the technical field of terminal control, and the method comprises the steps: obtaining a multi-frame depth image which is collected by a depth sensor of a terminal and comprises a user gesture action; matching the multiple frames of depth images with a plurality of preset recognition gestures to obtain target gestures corresponding to the multiple frames of depth images; determining a target control instruction corresponding to the target gesture from a plurality of preset control instructions according to the target gesture; and executing the target control instruction. Thus, a user can conveniently control the terminal to trigger the corresponding target control instruction by triggering the air gesture action without directly contacting the terminal, and the use experience of the user is greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal control technology, and in particular to a device control method, apparatus, electronic device and storage medium. Background Art

[0002] In order to improve the user experience and realize air gesture recognition, some manufacturers often collect image data through the front camera of the mobile phone, and then use image recognition algorithms to identify the user's gestures based on the image data, and then match the corresponding operation instructions (such as turning pages, sliding the screen, etc.).

[0003] However, when the camera collects image data, it is easily affected by the environment (such as lighting, color, texture, etc.), which will affect the accuracy of gesture recognition. In addition, cameras often consume a lot of power, and using cameras for data collection will further increase the power consumption of mobile phones. At the same time, complex image recognition algorithms also have a high consumption of mobile phone computing resources. During the gesture recognition process, the image data collected by the camera may contain the user's face or other user activities, which may easily lead to the leakage of the user's personal information. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a control method, an apparatus, an electronic device and a storage medium for a device.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for controlling a device is provided, the method comprising: acquiring a multi-frame depth image containing user gesture actions acquired by a depth sensor of a terminal; matching the multi-frame depth image with a plurality of preset recognition gestures to obtain a target gesture corresponding to the multi-frame depth image; determining, according to the target gesture, a target control instruction corresponding to the target gesture from a plurality of preset control instructions; and executing the target control instruction.

[0006] Optionally, matching the multiple frames of depth images with multiple preset recognition gestures to obtain target gestures corresponding to the multiple frames of depth images includes: performing feature extraction on the depth image for each frame of the depth image to obtain gesture features in the depth image; and performing feature matching on multiple gesture features with multiple preset recognition gestures to obtain the target gesture.

[0007] Optionally, for each frame of the depth image, performing feature extraction on the depth image to obtain the gesture feature in the depth image includes: for each frame of the depth image, determining the hand contour area in the depth image to obtain a target depth map containing the hand contour area; performing feature extraction on the target depth map to obtain the gesture feature.

[0008] Optionally, determining the hand contour area in the depth image for each frame of the depth image includes: for each frame of the depth image, obtaining a depth image corresponding to a previous frame; and determining the hand contour area in the depth image corresponding to the current frame by an inter-frame difference method based on the depth image corresponding to the current frame and the depth image corresponding to the previous frame.

[0009] Optionally, the method further includes: filtering the target depth map according to the hand contour area; extracting features from the target depth map to obtain the gesture features includes: extracting features from the filtered target depth map to obtain the gesture features.

[0010] Optionally, performing feature extraction on the target depth map to obtain the gesture feature includes: determining the center of mass position corresponding to the center of mass of the hand contour area in each of the target depth maps; for each of the target depth maps, obtaining the center of mass position corresponding to the target depth map of the previous frame; and determining the gesture feature corresponding to the target depth map of the current frame based on the center of mass position corresponding to the target depth map of the current frame and the center of mass position corresponding to the target depth map of the previous frame.

[0011] Optionally, determining the center of mass position corresponding to the center of mass of the hand contour area in each of the target depth maps includes: converting the target depth map into a corresponding binary image; and determining the center of mass position corresponding to the center of mass of the hand contour area based on the binary image.

[0012] Optionally, the center of mass position includes the center of mass coordinates of the center of mass of the hand contour area in a plane coordinate system, and the plane coordinate system is pre-established based on the depth image; determining the gesture feature corresponding to the target depth map of the current frame according to the center of mass position corresponding to the target depth map of the current frame and the center of mass position corresponding to the target depth map of the previous frame includes: determining the angle between the straight line where the center of mass corresponding to the target depth map of the current frame and the center of mass corresponding to the target depth map of the previous frame are located and the horizontal axis of the plane coordinate system according to the center of mass coordinates corresponding to the target depth map of the current frame and the center of mass coordinates corresponding to the target depth map of the previous frame, and obtaining a first center of mass angle corresponding to the center of mass in the target depth map of the current frame; determining the gesture feature corresponding to the target depth map of the current frame according to the first center of mass angle.

[0013] Optionally, the gesture feature is used to characterize characteristic values ​​corresponding to different center of mass angles, and determining the gesture feature corresponding to the target depth map of the current frame based on the first center of mass angle includes: performing feature conversion on the first center of mass angle so that the first center of mass angle is converted into a characteristic value within a preset numerical range to obtain the gesture feature.

[0014] Optionally, the feature matching of the multiple gesture features with the multiple preset recognition gestures to obtain the target gesture includes: determining a first depth map from the multiple frames of depth images; the number of the first depth maps is less than the number of depth images; for each frame of the first depth map, determining in sequence the Euclidean distance value between the gesture feature corresponding to the first depth map and the sequence feature corresponding to each preset recognition gesture; for each preset recognition gesture, determining the sum of the Euclidean distance values ​​corresponding to multiple frames of the first depth map to obtain a first distance value; taking the minimum value of the multiple first distance values ​​as the target distance value; and determining the target gesture based on the target distance value.

[0015] Optionally, determining the target gesture according to the target distance value includes: when the target distance value is less than or equal to a preset distance threshold, determining a second depth map from the multiple frames of depth images; the number of the second depth maps is less than the number of the depth images, and the second depth map is a continuous multiple-frame image; determining the gesture phase difference corresponding to each two adjacent frames of the second depth map; taking the sum of the multiple gesture phase differences as the target phase difference; and determining the target gesture according to the target phase difference.

[0016] Optionally, determining the gesture phase difference corresponding to each two adjacent frames of the second depth map includes: obtaining the second center of mass angle corresponding to each frame of the second depth map; for each two adjacent frames of the second depth map, taking the difference between the second center of mass angles corresponding to the current two adjacent frames of the second depth map as the gesture phase difference corresponding to the current two adjacent frames of the second depth map.

[0017] Optionally, determining the target gesture based on the target phase difference includes: taking a preset recognition gesture corresponding to the target distance value as a candidate gesture; obtaining a preset phase difference value range corresponding to the candidate gesture; and determining that the candidate gesture is the target gesture when the target phase difference is within the preset phase difference value range.

[0018] Optionally, the method also includes: when the candidate gesture is a specified gesture, acquiring depth data in each frame of the depth image; when the target phase difference is within the preset phase difference value range, determining that the candidate gesture is the target gesture includes: when a change trend of the depth data corresponding to the multiple frames of depth images is the same as a preset change trend corresponding to the specified gesture and the target phase difference is within the preset phase difference value range, determining that the candidate gesture is the target gesture.

[0019] Optionally, determining the target control instruction corresponding to the target gesture from multiple preset control instructions based on the target gesture includes: when the target gesture includes a first gesture, determining the target control instruction as a right swipe instruction; or, when the target gesture includes a second gesture, determining the target control instruction as a left swipe instruction; or, when the target gesture includes a third gesture, determining the target control instruction as a downward swipe instruction; or, when the target gesture includes a fourth gesture, determining the target control instruction as an upward swipe instruction; or, when the target gesture includes a fifth gesture, determining the target control instruction as a play / pause instruction; or, when the target gesture includes a sixth gesture, determining the target control instruction as an answer call / hang up call instruction.

[0020] According to a second aspect of an embodiment of the present disclosure, a control device of a device is provided, the device comprising: an acquisition module, configured to acquire a multi-frame depth image containing a user gesture action acquired by a depth sensor of a terminal; a matching module, configured to match the multi-frame depth image with a plurality of preset recognition gestures to obtain a target gesture corresponding to the multi-frame depth image; a determination module, configured to determine, according to the target gesture, a target control instruction corresponding to the target gesture from a plurality of preset control instructions; and an execution module, configured to execute the target control instruction.

[0021] Optionally, the matching module includes: an extraction submodule, configured to perform feature extraction on the depth image for each frame of the depth image to obtain gesture features in the depth image; and a matching submodule, configured to perform feature matching on multiple gesture features with multiple preset recognition gestures to obtain the target gesture.

[0022] Optionally, the extraction submodule is configured to determine, for each frame of the depth image, a hand contour area in the depth image to obtain a target depth map including the hand contour area; and perform feature extraction on the target depth map to obtain the gesture feature.

[0023] Optionally, the extraction submodule is configured to obtain, for each frame of the depth image, a depth image corresponding to a previous frame; and determine, based on the depth image corresponding to the current frame and the depth image corresponding to the previous frame, a hand contour area in the depth image corresponding to the current frame by using an inter-frame difference method.

[0024] Optionally, the matching module further includes: a filtering submodule configured to perform filtering processing on the target depth map according to the hand contour area;

[0025] The extraction submodule is configured to perform feature extraction on the filtered target depth map to obtain the gesture feature.

[0026] Optionally, the extraction submodule is configured to determine the center of mass position corresponding to the center of mass of the hand contour area in each of the target depth maps; for each of the target depth maps, obtain the center of mass position corresponding to the target depth map of the previous frame; and determine the gesture feature corresponding to the target depth map of the current frame based on the center of mass position corresponding to the target depth map of the current frame and the center of mass position corresponding to the target depth map of the previous frame.

[0027] Optionally, the extraction submodule is configured to convert the target depth map into a corresponding binary image; and determine a center of mass position corresponding to the center of mass of the hand contour area based on the binary image.

[0028] Optionally, the center of mass position includes the center of mass coordinates of the center of mass of the hand contour area in a plane coordinate system, and the plane coordinate system is pre-established based on the depth image; the extraction submodule is configured to determine the angle between the straight line where the center of mass corresponding to the target depth map of the current frame and the center of mass corresponding to the target depth map of the previous frame are located and the horizontal axis of the plane coordinate system according to the center of mass coordinates corresponding to the target depth map of the current frame and the center of mass coordinates corresponding to the target depth map of the previous frame, and obtain a first center of mass angle corresponding to the center of mass in the target depth map of the current frame; determine the gesture feature corresponding to the target depth map of the current frame based on the first center of mass angle.

[0029] Optionally, the gesture feature is used to characterize eigenvalues ​​corresponding to different center of mass angles, and the extraction submodule is configured to perform feature conversion on the first center of mass angle so that the first center of mass angle is converted into a eigenvalue within a preset numerical range to obtain the gesture feature.

[0030] Optionally, the matching submodule is configured to determine a first depth map from the multiple frames of depth images; the number of the first depth maps is less than the number of the depth images; for each frame of the first depth map, determine in sequence the Euclidean distance value between the gesture feature corresponding to the first depth map and the sequence feature corresponding to each of the preset recognition gestures; for each preset recognition gesture, determine the sum of the Euclidean distance values ​​corresponding to the multiple frames of the first depth maps to obtain a first distance value; take the minimum value of the multiple first distance values ​​as the target distance value; and determine the target gesture based on the target distance value.

[0031] Optionally, the matching submodule is configured to determine a second depth map from the multiple frames of depth images when the target distance value is less than or equal to a preset distance threshold; the number of the second depth maps is less than the number of the depth images, and the second depth maps are continuous multiple frames of images; determine the gesture phase difference corresponding to each two adjacent frames of the second depth maps; take the sum of the multiple gesture phase differences as the target phase difference; and determine the target gesture based on the target phase difference.

[0032] Optionally, the matching submodule is configured to obtain the second center of mass angle corresponding to the second depth map of each frame; for each two adjacent frames of the second depth map, the difference between the second center of mass angles corresponding to the current two adjacent frames of the second depth map is used as the gesture phase difference corresponding to the current two adjacent frames of the second depth map.

[0033] Optionally, the matching submodule is configured to take a preset recognition gesture corresponding to the target distance value as a candidate gesture; obtain a preset phase difference value range corresponding to the candidate gesture; and determine that the candidate gesture is the target gesture when the target phase difference is within the preset phase difference value range.

[0034] Optionally, the matching module further includes:

[0035] an acquisition submodule, configured to acquire depth data in each frame of the depth image when the candidate gesture is a specified gesture;

[0036] The matching submodule is configured to determine that the candidate gesture is the target gesture when a change trend of the depth data corresponding to the multiple frames of depth images is the same as a preset change trend corresponding to the designated gesture and the target phase difference is within the preset phase difference value range.

[0037] Optionally, the determination module is configured to, when the target gesture includes a first gesture, determine that the target control instruction is a right swipe instruction; or, when the target gesture includes a second gesture, determine that the target control instruction is a left swipe instruction; or, when the target gesture includes a third gesture, determine that the target control instruction is a downward swipe instruction; or, when the target gesture includes a fourth gesture, determine that the target control instruction is an upward swipe instruction; or, when the target gesture includes a fifth gesture, determine that the target control instruction is a play / pause instruction; or, when the target gesture includes a sixth gesture, determine that the target control instruction is an answer call / hang up call instruction.

[0038] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the steps of the device control method provided by the first aspect of the present disclosure when calling the executable instructions stored on the memory.

[0039] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the control method of the device provided in the first aspect of the present disclosure are implemented.

[0040] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0041] First, a multi-frame depth image containing user gestures collected by the depth sensor of the terminal is obtained. Secondly, the multi-frame depth image is matched with multiple preset recognition gestures to obtain the target gesture corresponding to the multi-frame depth image. After that, according to the target gesture, the target control instruction corresponding to the target gesture is determined from multiple preset control instructions, and the target control instruction is executed. Using the above method, first, a multi-frame depth image containing user gestures is collected by a depth sensor. Since the depth image only contains the distance between the target object and the depth sensor, it can effectively avoid collecting images containing the user's face or other user activities, ensuring that the user's personal information will not be leaked. Secondly, since the power consumption of the depth sensor itself is low, it will not have a significant impact on the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, which ensures the accuracy of the depth data, thereby improving the accuracy of gesture recognition. By matching the multi-frame depth image with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control instruction is determined according to the target gesture, and the target control instruction is executed. In this way, the user can conveniently control the terminal to trigger corresponding target control instructions by triggering air gestures without directly touching the terminal, which greatly improves the user experience.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0044] Figure 1 The figure is a flow chart of a method for controlling a device according to an exemplary embodiment.

[0045] Figure 2The diagram is a schematic diagram of a scenario according to an exemplary embodiment.

[0046] Figure 3 is another schematic diagram of a scenario according to an exemplary embodiment.

[0047] Figure 4 is another schematic diagram of a scenario according to an exemplary embodiment.

[0048] Figure 5 The figure is a flowchart of another method for controlling a device according to an exemplary embodiment.

[0049] Figure 6 The figure is a flowchart of another method for controlling a device according to an exemplary embodiment.

[0050] Figure 7 The figure is a flowchart of another method for controlling a device according to an exemplary embodiment.

[0051] Figure 8 The figure is a flowchart of another method for controlling a device according to an exemplary embodiment.

[0052] Fig. 9 The invention is a block diagram of a control device of a device according to an exemplary embodiment.

[0053] Fig.10 The invention is a block diagram of a control device of another apparatus according to an exemplary embodiment.

[0054] Fig.11 The invention is a block diagram of a control device of another apparatus according to an exemplary embodiment.

[0055] Fig.12 The invention is a block diagram of a control device of another apparatus according to an exemplary embodiment.

[0056] Fig.13 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0058] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0059] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar objects and do not have to be understood as a specific order or sequence. In addition, in the description with reference to the drawings, the same symbols in different drawings represent the same elements.

[0060] In the description of the present disclosure, unless otherwise specified, "multiple" means two or more than two, and other quantifiers are similar thereto; "at least one item", "one item or multiple items" or similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one item a can represent any number of a; for another example, one item or multiple items among a, b and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple; "and / or" is a kind of description of the association relationship of associated objects, indicating that there can be three kinds of relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " indicates that the associated objects before and after are in an "or" relationship.

[0061] Before introducing a control method, device, electronic device and storage medium of a device provided by the present disclosure, the application scenarios involved in each embodiment of the present disclosure are first introduced. In actual scenarios, when a user answers a call / hangs up a call, turns pages on a mobile phone, etc., it is often necessary to directly touch the screen of the mobile phone to achieve operation control. However, in some scenarios, it may be inconvenient for the user to directly touch the screen with both hands, and the corresponding operation will not be performed at this time. In order to improve the user's experience and realize air gesture recognition, some manufacturers often collect image data through the front camera of the mobile phone, and then identify the user's gesture action through the image recognition algorithm based on the image data, and then match the corresponding operation instructions (such as turning pages, sliding the screen, etc.). Based on the above scenarios, the inventors found that the camera is easily affected by the environment (such as light, color, texture, etc.) when collecting image data, which will affect the accuracy of gesture recognition. In addition, the camera often consumes a lot of power, and using the camera for data collection will further increase the power consumption of the mobile phone. At the same time, complex image recognition algorithms also have a high loss of computing resources for the mobile phone. In the process of gesture recognition, the image data collected by the camera may contain the user's face or other activities of the user, which is easy to cause the user's personal information to be leaked.

[0062] In order to solve the above technical problems, the present invention provides a control method, device, electronic device and storage medium of a device. First, a depth sensor is used to collect multiple frames of depth images containing user gestures. Since the depth image only contains the distance between the target object and the depth sensor, it can effectively avoid collecting images containing the user's face or other activities of the user, thereby ensuring that the user's personal information will not be leaked. Secondly, since the power consumption of the depth sensor itself is low, it will not have a significant impact on the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, thereby ensuring the accuracy of the depth data, thereby improving the accuracy of gesture recognition. By matching multiple frames of depth images with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control instruction can be determined according to the target gesture, and the target control instruction can be executed. In this way, the user can realize that by triggering the air gesture action without directly contacting the terminal, the terminal can be conveniently controlled to trigger the corresponding target control instruction, which greatly improves the user's experience.

[0063] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings.

[0064] Figure 1 The flowchart of a device control method according to an exemplary embodiment is shown. The method can be used in a terminal, wherein the terminal can be a mobile terminal such as a smart phone, a tablet computer, a smart TV, a smart watch, a PDA (English: Personal Digital Assistant; Chinese: Personal Digital Assistant), a portable computer, etc., or a smart home device, such as a sweeping robot, an air purifier, an air conditioner, a lighting lamp, a speaker, a robot, etc. Figure 1 As shown, the method may include the following steps:

[0065] In step S101, a plurality of depth images containing user gestures collected by a depth sensor of a terminal are obtained.

[0066] Among them, a depth sensor is set on the terminal, and the depth sensor can be, for example, a time of flight (TOF) sensor or other sensor capable of detecting depth information of a target object, which is not specifically limited in the present disclosure. In an actual scenario, the terminal can collect a depth image containing a user's gesture action through the depth sensor, and each pixel in the depth image is used to represent the distance between a certain point in the target object and the depth sensor.

[0067] In some embodiments, the triggering condition for the terminal to start the gesture recognition function can also be set. When the depth data collected by the depth sensor meets the triggering condition, the gesture recognition function of the terminal can be turned on. In this way, when the user does not need to use the gesture recognition function, the gesture recognition function can be turned off (for example, if the depth image containing the user's gesture action is not collected within a preset time period, the gesture recognition function can be turned off) to reduce the power consumption of the terminal. For example, the user can trigger the gesture recognition function of the terminal by facing the depth sensor on the terminal and staying for a certain period of time. That is, when the depth sensor continuously collects target objects with the same depth value within a preset range within a preset time period, the gesture recognition function of the terminal can be turned on. In addition, after turning on the gesture recognition function of the terminal, the corresponding prompt information can be displayed on the terminal interface to prompt the user that the gesture recognition function has been turned on in the current terminal, so as to facilitate the user to perform subsequent gesture actions.

[0068] It should be noted that the above triggering conditions for enabling the gesture recognition function are only examples, and the present disclosure is not limited thereto.

[0069] In step S102, multiple frames of depth images are matched with multiple preset recognition gestures to obtain target gestures corresponding to the multiple frames of depth images.

[0070] Among them, the preset recognition gesture may, for example, include pre-set gesture actions, such as drawing a circle gesture, sliding from left to right gesture, sliding from right to left gesture, sliding from top to bottom gesture, sliding from bottom to top gesture, double-click gesture, etc.

[0071] In this step, the gesture features in each frame of the depth image, that is, the features related to the user's gesture action in the depth image, can be identified, and then multiple gesture features can be matched with multiple preset recognition gestures to obtain the target gesture.

[0072] In step S103, according to the target gesture, a target control instruction corresponding to the target gesture is determined from a plurality of preset control instructions.

[0073] In a possible implementation, a target control instruction corresponding to a target gesture in a plurality of preset control instructions may be determined by a preset correspondence relationship. The preset correspondence relationship may include, for example, a correspondence relationship between a preset control instruction and a gesture action. The preset control instruction may include, for example, a left slide instruction, a right slide instruction, an up slide instruction, a down slide instruction, a play / pause instruction, a call answer / hang up instruction, etc.

[0074] For example, when the target gesture includes a first gesture, the target control instruction is determined to be a right slide instruction. The first gesture may be, for example, a slide gesture from left to right. Alternatively, when the target gesture includes a second gesture, the target control instruction is determined to be a left slide instruction. The second gesture may be, for example, a slide gesture from right to left. Alternatively, when the target gesture includes a third gesture, the target control instruction is determined to be a down slide instruction. The third gesture may be, for example, a slide gesture from top to bottom. Alternatively, when the target gesture includes a fourth gesture, the target control instruction is determined to be an up slide instruction. The fourth gesture may be, for example, a slide gesture from bottom to bottom. Alternatively, when the target gesture includes a fifth gesture, the target control instruction is determined to be a play / pause instruction. The fifth gesture may be, for example, a double-click gesture or a single-click gesture (the user may trigger the gesture action by pressing the palm toward the screen of the terminal). Alternatively, when the target gesture includes a sixth gesture, the target control instruction is determined to be a call connect / hang up instruction. The sixth gesture may be, for example, a circle gesture.

[0075] For ease of understanding, Figure 2 As shown in the actual scenario, the user can move along the Figure 2 At this time, the depth sensor of the terminal can collect multiple consecutive frames of depth images, which contain gesture information of the user's gesture action. By extracting gesture features from the depth image and then matching multiple gesture features with multiple preset recognition gestures, it can be confirmed that the user's target gesture is a circle gesture. Similarly, if Figure 3 As shown, users can Figure 3 The horizontal dotted line in the middle performs a sliding action from left to right or from right to left, thereby triggering the target gesture of the sliding gesture from left to right or from right to left. Figure 4 As shown, users can Figure 4 The vertical dotted line in the middle performs a sliding action from top to bottom or a sliding action from bottom to top to trigger the target gesture of the sliding gesture from top to bottom or from bottom to top.

[0076] In step S104, the target control instruction is executed.

[0077] For example, when it is determined that the target control instruction is a call connection / hang up instruction, if there is a call in the current terminal but it is not connected, the call can be controlled to be connected according to the target control instruction. Correspondingly, if there is a call being connected in the current terminal, the call can be controlled to be hung up according to the target control instruction. In this way, the user can control the answering or hanging up of the phone in the terminal by drawing a circle gesture in the air. For another example, when it is determined that the target control instruction is a right slide instruction or a left slide instruction, if the current terminal is in a video playback / music playback interface, the playback progress of the audio in the video / music playback interface in the video playback interface can be adjusted according to the target control instruction (or the switching of the video or audio can be controlled, and the specific control method can be set according to actual needs, and the present disclosure does not make specific limitations on this). Similarly, if the current terminal is in a document display interface (such as PPT playback), the document in the document display interface can be turned right or left according to the target control instruction. In this way, the user can control the adjustment / switching of the video / audio playback progress and the turning of the document by sliding left and right gestures in the air. For another example, when it is determined that the target control instruction includes an up-slide instruction / down-slide instruction, if the current terminal is in a document display interface (such as WORD playback), the document in the document display interface can be browsed by sliding up and down according to the target control instruction. If the current terminal is in a short video playback interface, the video of the short video playback interface can be switched according to the target control instruction. If the current terminal is in the main interface (it can also be a video playback or other interface, which can be set according to demand), the adjustment of the media volume can also be controlled according to the target control instruction. In this way, the user can control the switching of short videos, the adjustment of the media volume, and the browsing of documents by sliding up and down gestures in the air. For another example, when it is determined that the target control instruction includes a play / pause instruction, if the video / audio in the interface where the terminal is located is in a paused playback state, the video / audio playback can be controlled according to the target control instruction. Correspondingly, if the video / audio in the interface where the terminal is located is in a playback state, the video / audio can be controlled to pause playback according to the target control instruction. In this way, the user can control the pause / play media by clicking gestures in the air.

[0078] It should be noted that the above examples are for illustrative purposes only, and the present disclosure is not limited thereto, and other corresponding relationships are also possible.

[0079] In some scenarios, this embodiment can also be used in scenarios where multiple terminals are interconnected, for example, terminal A and terminal B are interconnected through Bluetooth, WIFI, etc., and terminal B supports gesture recognition. Then when terminal A is interconnected with terminal B, the user can control the corresponding operations on terminal A through terminal B. For example, the user can control the answering of the phone call on terminal A by performing corresponding gesture actions through terminal B. In this way, in the interconnection mode, the interconnection control between multiple devices can be realized through air gesture recognition. That is to say, in this embodiment, when the terminal is the first device, the target control instruction can be used to control the first device, and can also be used to control the second device connected to the first device.

[0080] Using the above method, first, a multi-frame depth image containing user gestures is collected through a depth sensor. Since the depth image only contains the distance between the target object and the depth sensor, it can effectively avoid collecting images containing the user's face or other activities of the user, ensuring that the user's information will not be leaked. Secondly, since the power consumption of the depth sensor itself is low, it will not have a significant impact on the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, which ensures the accuracy of the depth data and improves the accuracy of gesture recognition. By matching the multi-frame depth image with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control instruction can be determined according to the target gesture, and the target control instruction can be executed. In this way, the user can easily control the terminal to trigger the corresponding target control instruction by triggering the air gesture without directly contacting the terminal, which greatly improves the user experience.

[0081] The above step S102 is described in detail below. Figure 5 As shown, in step S102, matching multiple frames of depth images with multiple preset recognition gestures to obtain target gestures corresponding to the multiple frames of depth images may include the following steps:

[0082] In step S1021, for each frame of the depth image, feature extraction is performed on the depth image to obtain gesture features in the depth image.

[0083] In this step, for each frame of the depth image, the hand contour area in the depth image may be first identified, and then the depth image may be feature extracted based on the hand contour area to obtain the gesture features in the depth image.

[0084] Specifically, Figure 6 As shown, in step S1021, for each frame of the depth image, extracting features from the depth image to obtain gesture features in the depth image may include the following steps:

[0085] In step S1021a, for each frame of the depth image, a hand contour area in the depth image is determined to obtain a target depth map including the hand contour area.

[0086] For example, for each frame of the depth image, the depth image corresponding to the previous frame can be obtained first. And according to the depth image corresponding to the current frame and the depth image corresponding to the previous frame, the hand contour area in the depth image corresponding to the current frame is determined by the frame difference method. Since the hand area of ​​the user is always moving during the gesture, the moving area can be determined by the depth image between two consecutive frames, and then the hand contour area in the depth image can be determined according to the moving area. In this way, by continuously analyzing the hand contour area in each frame of the depth image, the user's gesture contour area can be tracked. After determining the hand contour area in each frame of the depth image, in order to facilitate subsequent further feature extraction, the hand and background areas in the depth image can be separated according to the hand contour area to obtain a target depth map containing the hand contour area.

[0087] In step S1021b, feature extraction is performed on the target depth map to obtain the gesture feature.

[0088] In this step, the following steps may be included:

[0089] Step A, determining the center of mass position corresponding to the center of mass of the hand contour area in each target depth map.

[0090] In this embodiment, in order to more accurately track the user's gestures, it is necessary to determine the movement trajectory of the gestures in multiple depth images. In order to improve the calculation efficiency, the centroid of the hand contour area in each frame of the target depth image can be used as the tracking point of the target depth image of that frame, and then the movement trajectory of the user's gestures can be tracked according to the tracking points corresponding to the multiple frame target depth images.

[0091] In order to determine the centroid of the hand contour area in each frame of the depth map, the target depth map can be converted into a corresponding binary image. In the binary image, for example, the hand contour area can be set to 1 and the rest can be set to 0. And according to the binary image, the centroid position corresponding to the centroid of the hand contour area is determined.

[0092] For example, a plane coordinate system can be pre-established, and the plane coordinate system can be pre-established based on the depth image. For example, the plane coordinate system can be established with the center line of the depth image as the coordinate axis and the center as the origin. At this time, the center of mass position includes the center of mass coordinates of the center of mass of the hand contour area in the plane coordinate system. It can be understood that after the target depth map is converted into a binary image, the mass distribution of the hand contour area is uniform, so the center of mass coordinates can be set to (cxt ,cy t ), then the centroid coordinates satisfy the following formula:

[0093]

[0094]

[0095] Among them, t represents the number of different frames, L represents the length of the depth image, and W represents the width of the depth image.

[0096] Step B, for each target depth map, obtain the center of mass position corresponding to the target depth map of the previous frame; and determine the gesture feature corresponding to the target depth map of the current frame according to the center of mass position corresponding to the target depth map of the current frame and the center of mass position corresponding to the target depth map of the previous frame.

[0097] Through the above step A, the center of mass position corresponding to each target depth map can be obtained. In order to further determine the moving trajectory of the user's gesture action, in this step, the center of mass angle α between the straight line where the center of mass of the target depth map of two adjacent frames is located and the horizontal axis in the plane coordinate system can be determined, and then the gesture feature corresponding to the target depth map can be determined according to the center of mass angle. Among them, the center of mass angle can be determined in the following way: set the center of mass coordinates in the target depth map at time t and t+1 respectively to (x t ,y t ) and (x t+1 ,y t+1 ), then the centroid angle α can be expressed by the following formula:

[0098]

[0099] Specifically, the angle between the straight line where the center of mass corresponding to the target depth map of the current frame and the center of mass corresponding to the target depth map of the previous frame are located and the horizontal axis of the plane coordinate system can be determined based on the center of mass coordinates corresponding to the target depth map of the current frame and the center of mass corresponding to the target depth map of the previous frame, and the first center of mass angle corresponding to the center of mass in the target depth map of the current frame can be obtained. The first center of mass angle corresponding to each target depth map can be determined in turn, and multiple first center of mass angles corresponding to multiple frames of target depth maps can be obtained. Then, the gesture feature corresponding to the target depth map of the current frame can be determined based on the first center of mass angle.

[0100] Considering that for some gestures, such as the circle drawing gesture, due to the different gesture habits of each user, where to start drawing the circle, and whether to draw the circle counterclockwise or clockwise, the first centroid angle corresponding to the same frame will be different. In order to eliminate the influence of different user gesture habits and reduce the difficulty of matching. The first centroid angle can be quantized (that is, feature conversion) so that the first centroid angle is converted into a feature value within a preset numerical range, and then the gesture feature is obtained. Among them, the gesture feature is used to characterize the feature values ​​corresponding to different centroid angles. For example, the plane coordinate system can be divided into 12 equal parts, each corresponding to an angle of 30 degrees, and the area where the positive direction of the horizontal axis is located is marked as 1 to 12 clockwise. In this step, the first centroid angle can be converted to any value in (1, 2, 3, ..., 12), and the feature conversion can be completed to obtain the corresponding gesture feature.

[0101] In addition, considering that in some scenarios, there may be some interference factors at the edge of the hand contour area in the depth image (for example, the depth value of the object adjacent to the user's hand is close to the depth value of the user's hand), in order to eliminate the influence of interference factors on subsequent gesture recognition, such as Figure 7 As shown, the method may also include the following steps:

[0102] In step S1021c, the target depth map is filtered according to the hand contour area.

[0103] For example, the hand contour area in the target depth map may be filtered by a mean filtering method.

[0104] Accordingly, performing feature extraction on the target depth map to obtain the gesture feature in step S1021b may include: performing feature extraction on the filtered target depth map to obtain the gesture feature.

[0105] In step S1022, feature matching is performed between the plurality of gesture features and a plurality of preset recognition gestures to obtain a target gesture.

[0106] Among them, each preset recognition gesture corresponds to a preset sequence feature, and the sequence feature can reflect the action feature of each gesture action. In this step, by matching multiple gesture features with multiple preset recognition gestures, the target gesture corresponding to the gesture feature can be determined from multiple preset recognition gestures. At the same time, considering that the time taken by different users to make different gestures is not fixed, even if the same user makes the same gesture, it cannot be guaranteed that the time taken is the same. Therefore, multiple gesture templates can be set for each preset recognition gesture (the completion time corresponding to each gesture template is different), and the corresponding sequence features can be determined. In this way, for the same gesture action, multiple templates can be provided for matching to improve the accuracy of recognition.

[0107] In one possible implementation, Figure 8 As shown, in step S1022, feature matching is performed on the plurality of gesture features with a plurality of preset recognition gestures to obtain a target gesture, which may include the following steps:

[0108] In step S1022a, a first depth map is determined from the multiple depth image frames.

[0109] The number of the first depth maps is smaller than the number of the depth images.

[0110] For example, starting from the most recently acquired depth image, the depth images corresponding to the multiple preset interval frames before the frame where the depth image is located can be obtained as the first depth map. The preset interval frames can be, for example but not limited to, frames of different lengths such as 14 frames, 18 frames, 20 frames, 22 frames and 24 frames. In other words, the depth image 14 frames away from the most recently acquired depth image, the depth image 18 frames away from the most recently acquired depth image, the depth image 20 frames away from the most recently acquired depth image, the depth image 22 frames away from the most recently acquired depth image and the depth image 24 frames away from the most recently acquired depth image can be used as the first depth map. It should be noted that the preset interval frames can be set according to the number of frames corresponding to the sequence features included in the gesture template corresponding to the preset recognition gesture.

[0111] In step S1022b, for each frame of the first depth map, the Euclidean distance values ​​between the gesture feature corresponding to the first depth map and the sequence features corresponding to each of the preset recognition gestures are determined in sequence.

[0112] Since the time taken by different users to make different gestures is not fixed, even if the same user makes the same gesture, it cannot be guaranteed that the time taken is the same. Therefore, in this step, gesture matching can be performed by calculating the similarity between two non-equal time series. Specifically, the similarity can be measured by comparing the Euclidean distance values ​​between the two sequences. The smaller the distance value, the more similar they are. For the gesture features of the first depth map of each frame, the Euclidean distance values ​​between the sequence features corresponding to each preset recognition gesture can be determined in turn. As can be seen from the foregoing, each preset recognition gesture may contain multiple gesture templates. At this time, for each preset recognition gesture, the Euclidean distance value between the gesture feature and the sequence feature corresponding to each gesture template can be determined in turn, and the minimum value of the Euclidean distance values ​​corresponding to the multiple gesture templates is used as the Euclidean distance value corresponding to the preset recognition gesture.

[0113] In step S1022c, for each preset recognition gesture, the sum of the Euclidean distance values ​​corresponding to multiple frames of the first depth map is determined to obtain a first distance value.

[0114] In this step, by determining the sum of the Euclidean distance values ​​corresponding to multiple frames of the first depth image, it is equivalent to determining the similarity between the current user's gesture action (ie, multiple frames of depth image) and each preset recognition gesture, that is, the first distance value.

[0115] In step S1022d, the minimum value among the multiple first distance values ​​is used as the target distance value.

[0116] As can be seen from the foregoing, the smaller the distance value, the higher the similarity. At this time, the minimum value in the first distance value can be used as the target distance value. In other words, the preset recognition gesture corresponding to the target distance value is the most similar to the gesture action of the current user, and the preset recognition gesture corresponding to the target distance value can be used as a candidate gesture.

[0117] In step S1022e, the target gesture is determined according to the target distance value.

[0118] In one possible implementation, if the target distance value is less than or equal to a preset distance threshold, it may indicate that the candidate gesture matches the multi-frame depth image successfully, that is, the target gesture is the preset recognition gesture (ie, the candidate gesture) corresponding to the target distance value.

[0119] In another possible implementation, in order to recognize the user's gesture and take into account the different times that different users take to complete gestures, generally speaking, in the first matching process (i.e., step S1022a to step S1022d), the first depth map is often selected from a time interval with a longer time span. However, it is precisely because the set time span is long that it may affect the recognition rate. In order to further accurately determine the user's target gesture, the gesture phase difference may also be judged. Specifically, the following steps may be included:

[0120] S1, when the target distance value is less than or equal to a preset distance threshold, determining a second depth map from the multiple frames of depth images.

[0121] The number of the second depth maps is less than the number of the depth images, and the second depth maps are continuous multi-frame images.

[0122] In this step, the preset recognition gesture corresponding to the target distance value can first be used as a candidate gesture, and then the first depth map corresponding to the minimum value of the multiple Euclidean distance values ​​(obtained in step S1022b) corresponding to the candidate gesture can be used as the third depth map. Then, according to the frame number corresponding to the third depth map, the second depth map is determined from the multi-frame depth image. For example, the depth image between the most recently acquired depth image in the multi-frame depth image and the third depth map can be used as the second depth map. For example, the frame number corresponding to the third depth map is the 14th frame, and the frame number of the most recently acquired depth image in the multi-frame depth image is the 30th frame, and the second depth map is the depth image between the 14th frame and the 30th frame.

[0123] S2, determining a gesture phase difference corresponding to each two adjacent frames of the second depth map.

[0124] Specifically, first, the second center of mass angle corresponding to the second depth map of each frame can be obtained (which can be obtained through the above step S1022b). Then, for each two adjacent frames of the second depth map, the difference between the second center of mass angles corresponding to the current two adjacent frames of the second depth map is used as the gesture phase difference corresponding to the current two adjacent frames of the second depth map. For example, the second center of mass angle corresponding to the second depth map of the tth frame is 45 degrees, and the second center of mass angle corresponding to the second depth map of the t+1th frame is 50 degrees, then the gesture phase difference is 5 degrees. By analogy, the gesture phase difference corresponding to each two adjacent frames of the second depth map can be obtained.

[0125] S3, taking the sum of the phase differences of multiple gestures as the target phase difference.

[0126] The target phase difference is obtained by adding the multiple gesture phase differences obtained in the above step S2.

[0127] S4: Determine the target gesture according to the target phase difference.

[0128] Specifically, first, the preset recognition gesture corresponding to the target distance value can be used as a candidate gesture. And obtain the preset phase difference range corresponding to the candidate gesture. Among them, the preset phase difference range is the preset phase difference range corresponding to each candidate gesture. It can be understood that the phase change of different gestures during the movement process may be different, so different phase difference ranges can be set for different gesture actions. Then, when the target phase difference is within the preset phase difference range, the candidate gesture is determined to be the target gesture. In this way, by judging according to the phase, it is possible to more accurately determine whether the gesture action triggered by the current user matches the candidate gesture. If the target phase difference is within the preset phase difference range, it can be determined that the gesture action triggered by the current user matches the candidate gesture, that is, the target gesture is the candidate gesture.

[0129] Whether it is the first match based on the centroid angle or the second match based on the phase difference, they are all based on the same plane, that is, the change in depth is not determined. However, for some gestures, such as a double-click gesture or a single-click gesture, the user's gesture may not move much in the same plane, but there is a large change in depth. Therefore, in order to identify such gestures, the depth data in each frame of the depth image can also be obtained when the candidate gesture is a specified gesture. Among them, the specified gesture may be, for example, but not limited to, a double-click gesture or a single-click gesture. Then, when the change trend of the depth data corresponding to the multi-frame depth image is the same as the preset change trend corresponding to the specified gesture and the target phase difference is within the preset phase difference value range, the candidate gesture is determined to be the target gesture. Among them, the preset change trend can be pre-set according to actual needs, for example, it can be reduced first and then increased (corresponding to a single-click gesture), or reduced first, then increased, then reduced, and then increased (corresponding to a double-click gesture).

[0130] For example, after determining a candidate gesture with a high similarity through the centroid angle, if the target phase difference is within the preset phase difference value range corresponding to the candidate gesture, the depth data corresponding to the multiple-frame depth image can also be obtained. It can be understood that, in general, each pixel in the depth image corresponds to the depth information of the point. In order to facilitate the determination of the depth change of the user's gesture, in this embodiment, the mean of the depth information corresponding to the hand contour area in the depth image can be used as the depth data, or the depth information corresponding to the centroid of the hand contour area in the depth image can be used as the depth data. Afterwards, the change trend of the depth data in multiple consecutive frames of depth images can be determined, and it can be determined whether the change trend of the depth data is the same as the preset change trend corresponding to the specified gesture. If they are the same, it can be determined that the candidate gesture is the target gesture.

[0131] It should be noted that, when the candidate gesture does not belong to the designated gesture, the comparison of the change trend of the depth data may not be performed, and the candidate gesture may be determined as the target gesture as long as the target phase difference is within the preset phase difference value range.

[0132] Using the above method, first, a multi-frame depth image containing user gestures is collected through a depth sensor. Since the depth image only contains the distance between the target object and the depth sensor, it can effectively avoid collecting images containing the user's face or other activities of the user, ensuring that the user's information will not be leaked. Secondly, since the power consumption of the depth sensor itself is low, it will not have a significant impact on the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, which ensures the accuracy of the depth data and improves the accuracy of gesture recognition. By matching the multi-frame depth image with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control instruction can be determined according to the target gesture, and the target control instruction can be executed. In this way, the user can easily control the terminal to trigger the corresponding target control instruction by triggering the air gesture without directly contacting the terminal, which greatly improves the user experience.

[0133] Fig. 9 is a block diagram of a control device of a device according to an exemplary embodiment. Fig. 9 As shown, the device 200 includes:

[0134] The acquisition module 201 is configured to acquire multiple frames of depth images containing user gestures collected by a depth sensor of the terminal;

[0135] A matching module 202 is configured to match the multiple depth images with multiple preset recognition gestures to obtain target gestures corresponding to the multiple depth images;

[0136] The determination module 203 is configured to determine, according to the target gesture, a target control instruction corresponding to the target gesture from a plurality of preset control instructions;

[0137] The execution module 204 is configured to execute the target control instruction.

[0138] Alternatively, if Fig.10 As shown, the matching module 202 includes:

[0139] The extraction submodule 2021 is configured to perform feature extraction on the depth image for each frame of the depth image to obtain gesture features in the depth image;

[0140] The matching submodule 2022 is configured to perform feature matching on the plurality of gesture features and the plurality of preset recognition gestures to obtain the target gesture.

[0141] Optionally, the extraction submodule 2021 is configured to determine, for each frame of the depth image, a hand contour area in the depth image to obtain a target depth map including the hand contour area; and perform feature extraction on the target depth map to obtain the gesture feature.

[0142] Optionally, the extraction submodule 2021 is configured to obtain, for each frame of the depth image, a depth image corresponding to the previous frame; and determine, based on the depth image corresponding to the current frame and the depth image corresponding to the previous frame, a hand contour area in the depth image corresponding to the current frame by using an inter-frame difference method.

[0143] Alternatively, if Fig.11 As shown, the matching module further includes: a filtering submodule 2023, configured to perform filtering processing on the target depth map according to the hand contour area;

[0144] The extraction submodule 2021 is configured to perform feature extraction on the filtered target depth map to obtain the gesture feature.

[0145] Optionally, the extraction submodule 2021 is configured to determine the center of mass position corresponding to the center of mass of the hand contour area in each target depth map; for each target depth map, obtain the center of mass position corresponding to the target depth map of the previous frame; and determine the gesture feature corresponding to the target depth map of the current frame based on the center of mass position corresponding to the target depth map of the current frame and the center of mass position corresponding to the target depth map of the previous frame.

[0146] Optionally, the extraction submodule 2021 is configured to convert the target depth map into a corresponding binary image; and determine the center of mass position corresponding to the center of mass of the hand contour area according to the binary image.

[0147] Optionally, the center of mass position includes the center of mass coordinates of the center of mass of the hand contour area in a plane coordinate system, which is pre-established based on the depth image; the extraction submodule 2021 is configured to determine the angle between the straight line where the center of mass corresponding to the target depth map of the current frame and the center of mass corresponding to the target depth map of the previous frame are located and the horizontal axis of the plane coordinate system according to the center of mass coordinates corresponding to the target depth map of the current frame and the center of mass coordinates corresponding to the target depth map of the previous frame, and obtain a first center of mass angle corresponding to the center of mass in the target depth map of the current frame; based on the first center of mass angle, determine the gesture feature corresponding to the target depth map of the current frame.

[0148] Optionally, the gesture feature is used to characterize eigenvalues ​​corresponding to different center of mass angles, and the extraction submodule 2021 is configured to perform feature conversion on the first center of mass angle so that the first center of mass angle is converted into a eigenvalue within a preset numerical range to obtain the gesture feature.

[0149] Optionally, the matching submodule 2022 is configured to determine a first depth map from the multiple frames of depth images; the number of the first depth maps is less than the number of the depth images; for each frame of the first depth map, determine in sequence the Euclidean distance value between the gesture feature corresponding to the first depth map and the sequence feature corresponding to each of the preset recognition gestures; for each preset recognition gesture, determine the sum of the Euclidean distance values ​​corresponding to the multiple frames of the first depth maps to obtain a first distance value; take the minimum value of the multiple first distance values ​​as the target distance value; and determine the target gesture based on the target distance value.

[0150] Optionally, the matching submodule 2022 is configured to determine a second depth map from the multiple frames of depth images when the target distance value is less than or equal to a preset distance threshold; the number of the second depth maps is less than the number of the depth images, and the second depth map is a continuous multiple-frame image; determine the gesture phase difference corresponding to each two adjacent frames of the second depth map; take the sum of the multiple gesture phase differences as the target phase difference; and determine the target gesture based on the target phase difference.

[0151] Optionally, the matching submodule 2022 is configured to obtain the second center of mass angle corresponding to the second depth map of each frame; for each two adjacent frames of the second depth map, the difference between the second center of mass angles corresponding to the current two adjacent frames of the second depth map is used as the gesture phase difference corresponding to the current two adjacent frames of the second depth map.

[0152] Optionally, the matching submodule 2022 is configured to take a preset recognition gesture corresponding to the target distance value as a candidate gesture; obtain a preset phase difference value range corresponding to the candidate gesture; and determine that the candidate gesture is the target gesture when the target phase difference is within the preset phase difference value range.

[0153] Alternatively, if Fig.12 As shown, the matching module 202 also includes:

[0154] The acquisition submodule 2024 is configured to acquire depth data in each frame of the depth image when the candidate gesture is a designated gesture;

[0155] The matching submodule 2022 is configured to determine that the candidate gesture is the target gesture when a change trend of the depth data corresponding to the multiple frames of depth images is the same as a preset change trend corresponding to the specified gesture and the target phase difference is within the preset phase difference value range.

[0156] Optionally, the determination module 203 is configured to determine that the target control instruction is a right slide instruction when the target gesture includes the first gesture; or,

[0157] In the case where the target gesture includes the second gesture, determining that the target control instruction is a left slide instruction; or,

[0158] In the case where the target gesture includes the third gesture, determining that the target control instruction is a sliding down instruction; or,

[0159] When the target gesture includes the fourth gesture, determining that the target control instruction is a slide-up instruction; or,

[0160] When the target gesture includes the fifth gesture, determining that the target control instruction is a play / pause instruction; or,

[0161] When the target gesture includes the sixth gesture, it is determined that the target control instruction is a call answering / call ending instruction.

[0162] Using the above method, first, a multi-frame depth image containing user gestures is collected through a depth sensor. Since the depth image only contains the distance between the target object and the depth sensor, it can effectively avoid collecting images containing the user's face or other activities of the user, ensuring that the user's information will not be leaked. Secondly, since the power consumption of the depth sensor itself is low, it will not have a significant impact on the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, which ensures the accuracy of the depth data and improves the accuracy of gesture recognition. By matching the multi-frame depth image with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control instruction can be determined according to the target gesture, and the target control instruction can be executed. In this way, the user can easily control the terminal to trigger the corresponding target control instruction by triggering the air gesture without directly contacting the terminal, which greatly improves the user experience.

[0163] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0164] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the control method of the device provided by the present disclosure are implemented.

[0165] Fig.13 is a block diagram of an electronic device 300 according to an exemplary embodiment. For example, the electronic device 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0166] Reference Fig.13 , the electronic device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output interface 312 , a sensor component 314 , and a communication component 316 .

[0167] The processing component 302 generally controls the overall operation of the electronic device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the control method of the device described above. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.

[0168] The memory 304 is configured to store various types of data to support operations on the electronic device 300. Examples of such data include instructions for any application or method operating on the electronic device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0169] The power supply component 306 provides power to the various components of the electronic device 300. The power supply component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 300.

[0170] The multimedia component 308 includes a screen that provides an output interface between the electronic device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the electronic device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0171] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the electronic device 300 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.

[0172] The input / output interface 312 provides an interface between the processing component 302 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0173] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the electronic device 300. For example, the sensor assembly 314 can detect the open / closed state of the electronic device 300, the relative positioning of components, such as the display and keypad of the electronic device 300, and the sensor assembly 314 can also detect the position change of the electronic device 300 or a component of the electronic device 300, the presence or absence of user contact with the electronic device 300, the orientation or acceleration / deceleration of the electronic device 300, and the temperature change of the electronic device 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0174] The communication component 316 is configured to facilitate wired or wireless communication between the electronic device 300 and other devices. The electronic device 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0175] In an exemplary embodiment, the electronic device 300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the control methods of the above-mentioned devices.

[0176] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, and the instructions can be executed by a processor 320 of an electronic device 300 to complete the control method of the device. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0177] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the control method of the above-mentioned device when executed by the programmable device.

[0178] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for controlling a device, characterized in that: The method comprises: Acquire multiple frames of depth images containing user gestures collected by a depth sensor of the terminal; Matching the multiple depth images with multiple preset recognition gestures to obtain target gestures corresponding to the multiple depth images; According to the target gesture, determining a target control instruction corresponding to the target gesture from a plurality of preset control instructions; Execute the target control instruction.

2. The method according to claim 1, characterized in that The step of matching the multiple depth images with a plurality of preset recognition gestures to obtain target gestures corresponding to the multiple depth images comprises: For each frame of the depth image, extracting features of the depth image to obtain gesture features in the depth image; Feature matching is performed between the plurality of gesture features and the plurality of preset recognition gestures to obtain the target gesture.

3. The method according to claim 2, characterized in that The extracting features of the depth image for each frame of the depth image to obtain gesture features in the depth image includes: For each frame of the depth image, determine a hand contour area in the depth image to obtain a target depth map including the hand contour area; Feature extraction is performed on the target depth map to obtain the gesture feature.

4. The method according to claim 3, characterized in that The step of determining the hand contour area in the depth image for each frame of the depth image includes: For each frame of the depth image, a depth image corresponding to the previous frame is obtained; and based on the depth image corresponding to the current frame and the depth image corresponding to the previous frame, a hand contour area in the depth image corresponding to the current frame is determined by an inter-frame difference method.

5. The method according to claim 3, characterized in that: The method further comprises: According to the hand contour area, filtering the target depth map; The extracting features of the target depth map to obtain the gesture features comprises: Feature extraction is performed on the filtered target depth map to obtain the gesture feature.

6. The method according to claim 3, characterized in that: The extracting features of the target depth map to obtain the gesture features comprises: Determine the center of mass position corresponding to the center of mass of the hand contour area in each of the target depth maps; For each of the target depth maps, obtain the centroid position corresponding to the target depth map of the previous frame; and determine the gesture feature corresponding to the target depth map of the current frame based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame.

7. The method according to claim 6, characterized in that Determining the center of mass position corresponding to the center of mass of the hand contour area in each target depth map comprises: Converting the target depth map into a corresponding binary image; The centroid position corresponding to the centroid of the hand contour area is determined according to the binary image.

8. The method according to claim 6, characterized in that The centroid position includes the centroid coordinates of the centroid of the hand contour area in a plane coordinate system, and the plane coordinate system is pre-established based on the depth image; determining the gesture feature corresponding to the target depth map of the current frame according to the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame includes: According to the centroid coordinates corresponding to the target depth map of the current frame and the centroid coordinates corresponding to the target depth map of the previous frame, determine the angle between the straight line where the centroid corresponding to the target depth map of the current frame and the centroid corresponding to the target depth map of the previous frame are located and the horizontal axis of the plane coordinate system, and obtain a first centroid angle corresponding to the centroid in the target depth map of the current frame; The gesture feature corresponding to the target depth image of the current frame is determined according to the first centroid angle.

9. The method according to claim 8, characterized in that The gesture feature is used to characterize the feature values ​​corresponding to different centroid angles, and determining the gesture feature corresponding to the target depth map of the current frame according to the first centroid angle includes: The first centroid angle is subjected to feature conversion so that the first centroid angle is converted into a feature value within a preset numerical range to obtain the gesture feature.

10. The method according to claim 2, characterized in that The performing feature matching of the plurality of gesture features with the plurality of preset recognition gestures to obtain the target gesture comprises: Determine a first depth map from the multiple depth image frames; the number of the first depth maps is less than the number of the depth images; For each frame of the first depth map, sequentially determining the Euclidean distance value between the gesture feature corresponding to the first depth map and the sequence feature corresponding to each of the preset recognition gestures; For each preset recognition gesture, determine the sum of the Euclidean distance values ​​corresponding to the first depth images of multiple frames to obtain a first distance value; Taking the minimum value among the multiple first distance values ​​as the target distance value; The target gesture is determined according to the target distance value.

11. The method according to claim 10, characterized in that Determining the target gesture according to the target distance value includes: When the target distance value is less than or equal to a preset distance threshold, determining a second depth map from the multiple depth image frames; the number of the second depth maps is less than the number of the depth images, and the second depth map is a continuous multiple-frame image; Determine the gesture phase difference between each two adjacent frames of the second depth map; taking the sum of the phase differences of the plurality of gestures as the target phase difference; The target gesture is determined according to the target phase difference.

12. The method according to claim 11, characterized in that The step of determining the gesture phase difference between each two adjacent frames of the second depth map comprises: Obtain a second centroid angle corresponding to the second depth map of each frame; For each two adjacent frames of the second depth map, the difference between the second centroid angles corresponding to the current two adjacent frames of the second depth map is used as the gesture phase difference corresponding to the current two adjacent frames of the second depth map.

13. The method according to claim 11, characterized in that The determining the target gesture according to the target phase difference comprises: Taking a preset recognition gesture corresponding to the target distance value as a candidate gesture; Obtaining a preset phase difference range corresponding to the candidate gesture; When the target phase difference is within the preset phase difference value range, the candidate gesture is determined to be the target gesture.

14. The method according to claim 13, characterized in that The method further comprises: When the candidate gesture is a designated gesture, obtaining depth data in each frame of the depth image; When the target phase difference is within the preset phase difference value range, determining the candidate gesture as the target gesture includes: When a change trend of the depth data corresponding to the multiple frames of depth images is the same as a preset change trend corresponding to the designated gesture and the target phase difference is within the preset phase difference value range, the candidate gesture is determined to be the target gesture.

15. The method according to any one of claims 1 to 14, characterized in that Determining, according to the target gesture, a target control instruction corresponding to the target gesture from a plurality of preset control instructions comprises: In the case where the target gesture includes the first gesture, determining that the target control instruction is a right slide instruction; or, In the case where the target gesture includes the second gesture, determining that the target control instruction is a left slide instruction; or, In the case where the target gesture includes a third gesture, determining that the target control instruction is a sliding down instruction; or, In a case where the target gesture includes the fourth gesture, determining that the target control instruction is a slide-up instruction; or, In a case where the target gesture includes the fifth gesture, determining that the target control instruction is a play / pause instruction; or, When the target gesture includes the sixth gesture, the target control instruction is determined to be a call answering / call ending instruction.

16. A control device for an equipment, characterized in that: The device comprises: An acquisition module is configured to acquire multiple frames of depth images containing user gestures collected by a depth sensor of the terminal; A matching module, configured to match the multiple frames of depth images with multiple preset recognition gestures to obtain target gestures corresponding to the multiple frames of depth images; a determination module, configured to determine, according to the target gesture, a target control instruction corresponding to the target gesture from a plurality of preset control instructions; The execution module is configured to execute the target control instruction.

17. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the steps of the method described in any one of claims 1 to 15 when calling the executable instructions stored in the memory.

18. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 15 are implemented.

Images