Control method and device of equipment, electronic equipment and storage medium

Abstract

The present disclosure relates to a device control method and device, an electronic device and a storage medium, and relates to the technical field of terminal control. The method comprises: acquiring a plurality of depth images containing user gesture actions collected by a depth sensor of a terminal; matching the plurality of depth images with a plurality of preset recognition gestures to obtain target gestures corresponding to the plurality of depth images; determining a target control instruction corresponding to the target gestures from a plurality of preset control instructions according to the target gestures; and executing the target control instruction. In this way, the user can conveniently control the terminal to trigger the corresponding target control instruction by triggering the gesture action in the air without directly contacting the terminal, greatly improving the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of terminal control technology, and in particular to a control method, apparatus, electronic device, and storage medium for a device. Background Technology

[0002] To enhance the user experience and enable air gesture recognition, some manufacturers often use the phone's front-facing camera to collect image data, and then use image recognition algorithms to identify the user's hand gestures based on the image data, thereby matching the corresponding operation commands (such as turning pages, swiping the screen, etc.).

[0003] However, camera image data is easily affected by environmental factors (such as lighting, color, and texture), which can impact the accuracy of gesture recognition. Furthermore, cameras often consume significant power, and using a camera for data acquisition will further increase the phone's power consumption. Simultaneously, complex image recognition algorithms also place a high demand on the phone's computing resources. During gesture recognition, the image data captured by the camera may contain the user's face or other user activities, potentially leading to the leakage of the user's personal information. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, this disclosure provides a control method, apparatus, electronic device, and storage medium for a device.

[0005] According to a first aspect of the present disclosure, a method for controlling a device is provided, the method comprising: acquiring a multi-frame depth image containing user gestures collected by a depth sensor of a terminal; matching the multi-frame depth image with a plurality of preset recognition gestures to obtain a target gesture corresponding to the multi-frame depth image; determining a target control instruction corresponding to the target gesture from a plurality of preset control instructions based on the target gesture; and executing the target control instruction.

[0006] Optionally, matching the multi-frame depth images with multiple preset recognition gestures to obtain the target gesture corresponding to the multi-frame depth images includes: extracting features from the depth images for each frame of the depth image to obtain gesture features in the depth images; and matching the multiple gesture features with the multiple preset recognition gestures to obtain the target gesture.

[0007] Optionally, the step of extracting features from the depth image for each frame of the depth image to obtain gesture features in the depth image includes: for each frame of the depth image, determining the hand contour region in the depth image to obtain a target depth map containing the hand contour region; and extracting features from the target depth map to obtain the gesture features.

[0008] Optionally, determining the hand contour region in the depth image for each frame of the depth image includes: obtaining the depth image corresponding to the previous frame for each frame of the depth image; and determining the hand contour region in the depth image corresponding to the current frame by means of the inter-frame difference method based on the depth image corresponding to the current frame and the depth image corresponding to the previous frame.

[0009] Optionally, the method further includes: filtering the target depth map based on the hand contour region; and extracting features from the target depth map to obtain the gesture features, which includes: extracting features from the filtered target depth map to obtain the gesture features.

[0010] Optionally, the step of extracting features from the target depth map to obtain the gesture features includes: determining the centroid position corresponding to the centroid of the hand contour region in each target depth map; obtaining the centroid position corresponding to the target depth map of the previous frame for each target depth map; and determining the gesture features corresponding to the target depth map of the current frame based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame.

[0011] Optionally, determining the centroid position corresponding to the centroid of the hand contour region in each of the target depth maps includes: converting the target depth map into a corresponding binary image; and determining the centroid position corresponding to the centroid of the hand contour region based on the binary image.

[0012] Optionally, the centroid position includes the centroid coordinates of the hand contour region in a planar coordinate system, the planar coordinate system being pre-established based on the depth image; determining the gesture feature corresponding to the target depth map of the current frame based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame includes: determining the angle between the straight line containing the centroid of the target depth map of the current frame and the centroid of the target depth map of the previous frame and the horizontal axis of the planar coordinate system, based on the centroid coordinates corresponding to the target depth map of the current frame and the centroid of the target depth map of the previous frame, to obtain a first centroid angle corresponding to the centroid in the target depth map of the current frame; determining the gesture feature corresponding to the target depth map of the current frame based on the first centroid angle.

[0013] Optionally, the gesture feature is used to characterize the feature values ​​corresponding to different centroid angles. Determining the gesture feature corresponding to the target depth map of the current frame based on the first centroid angle includes: performing feature transformation on the first centroid angle so that the first centroid angle is converted into a feature value within a preset numerical range to obtain the gesture feature.

[0014] Optionally, the step of performing feature matching between the multiple gesture features and the multiple preset recognition gestures to obtain the target gesture includes: determining a first depth map from the multiple frames of depth images; the number of first depth maps is less than the number of depth images; for each frame of the first depth map, sequentially determining the Euclidean distance value between the gesture feature corresponding to the first depth map and the sequence feature corresponding to each preset recognition gesture; for each preset recognition gesture, determining the sum of the Euclidean distance values ​​corresponding to the multiple frames of the first depth map to obtain a first distance value; taking the minimum value among the multiple first distance values ​​as the target distance value; and determining the target gesture based on the target distance value.

[0015] Optionally, determining the target gesture based on the target distance value includes: if the target distance value is less than or equal to a preset distance threshold, determining a second depth map from the multi-frame depth images; the number of second depth maps is less than the number of depth images, and the second depth maps are consecutive multi-frame images; determining the gesture phase difference between each pair of adjacent second depth maps; taking the sum of the multiple gesture phase differences as the target phase difference; and determining the target gesture based on the target phase difference.

[0016] Optionally, determining the gesture phase difference between each pair of adjacent second depth maps includes: obtaining the second centroid angle corresponding to each frame of the second depth map; and for each pair of adjacent second depth maps, taking the difference between the second centroid angles corresponding to the current pair of adjacent second depth maps as the gesture phase difference corresponding to the current pair of adjacent second depth maps.

[0017] Optionally, determining the target gesture based on the target phase difference includes: using a preset recognition gesture corresponding to the target distance value as a candidate gesture; obtaining a preset phase difference value range corresponding to the candidate gesture; and determining the candidate gesture as the target gesture when the target phase difference is within the preset phase difference value range.

[0018] Optionally, the method further includes: when the candidate gesture is a specified gesture, acquiring depth data in each frame of the depth image; and determining the candidate gesture as the target gesture when the target phase difference is within the range of the preset phase difference value includes: determining the candidate gesture as the target gesture when the change trend of the depth data corresponding to the multi-frame depth images is the same as the preset change trend corresponding to the specified gesture and the target phase difference is within the range of the preset phase difference value.

[0019] Optionally, determining the target control command corresponding to the target gesture from a plurality of preset control commands based on the target gesture includes: determining the target control command as a right swipe command when the target gesture includes a first gesture; or determining the target control command as a left swipe command when the target gesture includes a second gesture; or determining the target control command as a down swipe command when the target gesture includes a third gesture; or determining the target control command as an up swipe command when the target gesture includes a fourth gesture; or determining the target control command as a play / pause command when the target gesture includes a fifth gesture; or determining the target control command as an answer / hang up command when the target gesture includes a sixth gesture.

[0020] According to a second aspect of the present disclosure, a control apparatus for a device is provided, the apparatus comprising: an acquisition module configured to acquire multi-frame depth images including user gestures collected by a depth sensor of a terminal; a matching module configured to match the multi-frame depth images with a plurality of preset recognition gestures to obtain a target gesture corresponding to the multi-frame depth images; a determination module configured to determine a target control instruction corresponding to the target gesture from a plurality of preset control instructions based on the target gesture; and an execution module configured to execute the target control instruction.

[0021] Optionally, the matching module includes: an extraction submodule configured to extract features from the depth image for each frame of the depth image to obtain gesture features in the depth image; and a matching submodule configured to perform feature matching between multiple gesture features and multiple preset recognition gestures to obtain the target gesture.

[0022] Optionally, the extraction submodule is configured to, for each frame of the depth image, determine the hand contour region in the depth image to obtain a target depth map containing the hand contour region; and perform feature extraction on the target depth map to obtain the gesture features.

[0023] Optionally, the extraction submodule is configured to obtain the depth image corresponding to the previous frame for each frame of the depth image; and determine the hand contour region in the depth image corresponding to the current frame by means of the inter-frame difference method based on the depth image corresponding to the current frame and the depth image corresponding to the previous frame.

[0024] Optionally, the matching module further includes a filtering submodule, configured to filter the target depth map based on the hand contour region;

[0025] The extraction submodule is configured to extract features from the filtered target depth map to obtain the gesture features.

[0026] Optionally, the extraction submodule is configured to determine the centroid position corresponding to the centroid of the hand contour region in each target depth map; for each target depth map, obtain the centroid position corresponding to the target depth map of the previous frame; and determine the gesture feature corresponding to the target depth map of the current frame based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame.

[0027] Optionally, the extraction submodule is configured to convert the target depth map into a corresponding binary image; and determine the centroid position corresponding to the centroid of the hand contour region based on the binary image.

[0028] Optionally, the centroid position includes the centroid coordinates of the hand contour region in a planar coordinate system, which is pre-established based on the depth image; the extraction submodule is configured to determine the angle between the line containing the centroid of the target depth map of the current frame and the centroid of the target depth map of the previous frame and the horizontal axis of the planar coordinate system, based on the centroid coordinates corresponding to the target depth map of the current frame and the centroid coordinates corresponding to the target depth map of the previous frame, to obtain a first centroid angle corresponding to the centroid in the target depth map of the current frame; and determine the gesture feature corresponding to the target depth map of the current frame based on the first centroid angle.

[0029] Optionally, the gesture feature is used to characterize the feature values ​​corresponding to different centroid angles. The extraction submodule is configured to perform feature transformation on the first centroid angle so that the first centroid angle is converted into feature values ​​within a preset numerical range to obtain the gesture feature.

[0030] Optionally, the matching submodule is configured to determine a first depth map from the multi-frame depth images; the number of first depth maps is less than the number of depth images; for each frame of the first depth map, sequentially determine the Euclidean distance value between the gesture feature corresponding to the first depth map and the sequence feature corresponding to each preset recognition gesture; for each preset recognition gesture, determine the sum of the Euclidean distance values ​​corresponding to the multi-frame first depth maps to obtain a first distance value; take the minimum value among the multiple first distance values ​​as the target distance value; and determine the target gesture based on the target distance value.

[0031] Optionally, the matching submodule is configured to, when the target distance value is less than or equal to a preset distance threshold, determine a second depth map from the multi-frame depth images; the number of second depth maps is less than the number of depth images, and the second depth map is a series of consecutive multi-frame images; determine the gesture phase difference between each pair of adjacent second depth maps; take the sum of the multiple gesture phase differences as the target phase difference; and determine the target gesture based on the target phase difference.

[0032] Optionally, the matching submodule is configured to obtain the second centroid angle corresponding to each frame of the second depth map; and for each pair of adjacent frames of the second depth map, the difference between the second centroid angles corresponding to the current pair of adjacent frames of the second depth map is used as the gesture phase difference corresponding to the current pair of adjacent frames of the second depth map.

[0033] Optionally, the matching submodule is configured to take a preset recognition gesture corresponding to the target distance value as a candidate gesture; obtain a preset phase difference value range corresponding to the candidate gesture; and determine the candidate gesture as the target gesture when the target phase difference is within the preset phase difference value range.

[0034] Optionally, the matching module further includes:

[0035] The acquisition submodule is configured to acquire depth data in each frame of the depth image when the candidate gesture is a specified gesture;

[0036] The matching submodule is configured to determine the candidate gesture as the target gesture when the change trend of the depth data corresponding to the multi-frame depth images is the same as the preset change trend corresponding to the specified gesture and the target phase difference is within the preset phase difference value range.

[0037] Optionally, the determining module is configured to determine the target control command as a right swipe command when the target gesture includes a first gesture; or, determine the target control command as a left swipe command when the target gesture includes a second gesture; or, determine the target control command as a down swipe command when the target gesture includes a third gesture; or, determine the target control command as an up swipe command when the target gesture includes a fourth gesture; or, determine the target control command as a play / pause command when the target gesture includes a fifth gesture; or, determine the target control command as an answer / hang up command when the target gesture includes a sixth gesture.

[0038] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the control method for the device provided in the first aspect of the present disclosure when the executable instructions stored in the memory are invoked.

[0039] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the control method for the device provided in the first aspect of the present disclosure.

[0040] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0041] First, the system acquires multiple frames of depth images containing user gestures, captured by the terminal's depth sensor. Second, these multiple frames are matched with multiple preset recognition gestures to obtain the target gesture corresponding to each frame. Then, based on the target gesture, the system determines the corresponding target control command from multiple preset control commands and executes it. This method, by first acquiring multiple frames of depth images containing user gestures using a depth sensor (which only includes the distance between the target object and the depth sensor), effectively avoids capturing images of the user's face or other activities, ensuring the user's personal information is not leaked. Secondly, the depth sensor itself has low power consumption, minimizing its impact on the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is less susceptible to environmental influences, ensuring the accuracy of the depth data and thus improving gesture recognition accuracy. Finally, by matching multiple frames of depth images with multiple preset recognition gestures, the system accurately identifies the target gesture corresponding to the user's gesture, determines the corresponding target control command based on the target gesture, and executes it. In this way, users can conveniently control the terminal to trigger corresponding target control commands without directly touching the terminal, by triggering air gestures, which greatly improves the user experience.

[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0044] Figure 1 This is a flowchart illustrating a control method for a device according to an exemplary embodiment.

[0045] Figure 2This is a schematic diagram of a scenario according to an exemplary embodiment.

[0046] Figure 3 This is a schematic diagram of another scenario according to an exemplary embodiment.

[0047] Figure 4 This is a schematic diagram of another scenario according to an exemplary embodiment.

[0048] Figure 5 This is a flowchart illustrating another method for controlling a device according to an exemplary embodiment.

[0049] Figure 6 This is a flowchart illustrating another method for controlling a device according to an exemplary embodiment.

[0050] Figure 7 This is a flowchart illustrating another method for controlling a device according to an exemplary embodiment.

[0051] Figure 8 This is a flowchart illustrating another method for controlling a device according to an exemplary embodiment.

[0052] Figure 9 This is a block diagram illustrating a control device for an apparatus according to an exemplary embodiment.

[0053] Figure 10 This is a block diagram of a control device for another device according to an exemplary embodiment.

[0054] Figure 11 This is a block diagram of a control device for another device according to an exemplary embodiment.

[0055] Figure 12 This is a block diagram of a control device for another device according to an exemplary embodiment.

[0056] Figure 13 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0058] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0059] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily construed as referring to a specific order or sequence. Furthermore, in the description with reference to the accompanying drawings, the same reference numerals in different drawings denote the same elements.

[0060] In the description of this disclosure, unless otherwise stated, "multiple" means two or more, and other quantifiers are similar; "at least one," "one or more," or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one 'a' can represent any number of 'a's; as another example, one or more of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple; "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " indicates that the preceding and following related objects are in an "or" relationship.

[0061] Before introducing the control method, apparatus, electronic device, and storage medium of the device provided in this disclosure, the application scenarios involved in the various embodiments of this disclosure are first introduced. In practical scenarios, when users answer / hang up calls or turn pages on their phones, they often need to directly touch the phone screen to control the operation. However, in some scenarios, users may not be able to directly touch the screen with both hands, making it impossible to perform the corresponding operations. To improve the user experience and achieve air gesture recognition, some manufacturers often use the front-facing camera of the phone to collect image data, and then use image recognition algorithms to recognize the user's gestures based on the image data, thereby matching the corresponding operation commands (such as turning pages, swiping the screen, etc.). Based on the above scenarios, the inventors found that the image data collected by the camera is easily affected by the environment (such as lighting, color, texture, etc.), which will affect the accuracy of gesture recognition. Furthermore, cameras often consume a lot of power, and using a camera for data collection will further increase the power consumption of the phone. At the same time, complex image recognition algorithms also consume a lot of the phone's computing resources. During the gesture recognition process, the image data collected by the camera may contain the user's face or other user activities, which may easily lead to the leakage of the user's personal information.

[0062] To address the aforementioned technical problems, this invention provides a control method, apparatus, electronic device, and storage medium for a device. First, a depth sensor acquires multiple frames of depth images containing user gestures. Since the depth images only contain the distance between the target object and the depth sensor, it effectively avoids capturing images of the user's face or other user activities, ensuring that the user's personal information is not leaked. Second, because the depth sensor itself has low power consumption, it does not significantly impact the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, ensuring the accuracy of the depth data and thus improving the accuracy of gesture recognition. By matching multiple frames of depth images with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control command can be determined and executed based on the target gesture. In this way, the user can conveniently control the terminal to trigger corresponding target control commands without directly touching the terminal, greatly improving the user experience.

[0063] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0064] Figure 1 This is a flowchart illustrating a device control method according to an exemplary embodiment. The method can be used in a terminal, which can be a smartphone, tablet, smart TV, smartwatch, PDA (Personal Digital Assistant), portable computer, or other mobile terminal, or a smart home device such as a robot vacuum cleaner, air purifier, air conditioner, light bulb, speaker, or robot. Figure 1 As shown, the method may include the following steps:

[0065] In step S101, a multi-frame depth image containing user gestures is acquired by the terminal's depth sensor.

[0066] The terminal is equipped with a depth sensor, which may be, for example, a Time-of-Flight (TOF) sensor or other sensors capable of detecting depth information of a target object; this disclosure does not specifically limit the type of sensor. In practical scenarios, the terminal can use this depth sensor to acquire depth images that include user gestures. Each pixel in the depth image represents the distance between a point on the target object and the depth sensor.

[0067] In some embodiments, trigger conditions for enabling the gesture recognition function on the terminal can be set. When the depth data collected by the depth sensor meets the trigger conditions, the gesture recognition function of the terminal can be enabled. This allows the gesture recognition function to be disabled when the user does not need it (for example, if no depth image containing the user's gesture is collected within a preset time period), thereby reducing the terminal's power consumption. For example, the user can trigger the terminal's gesture recognition function by placing their palm directly on the depth sensor and holding it there for a certain period. That is, the terminal's gesture recognition function can be enabled when the depth sensor continuously collects target objects with the same depth value within a preset time period. Furthermore, after enabling the terminal's gesture recognition function, corresponding prompts can be displayed on the terminal interface to notify the user that the gesture recognition function is currently enabled, facilitating subsequent gesture actions.

[0068] It should be noted that the above-mentioned triggering conditions for enabling gesture recognition are merely illustrative examples, and this disclosure is not limited to them.

[0069] In step S102, the multi-frame depth images are matched with multiple preset recognition gestures to obtain the target gestures corresponding to the multi-frame depth images.

[0070] The preset recognition gesture may include, for example, a pre-set gesture action, such as a circle drawing gesture, a swipe gesture from left to right, a swipe gesture from right to left, a swipe gesture from top to bottom, a swipe gesture from bottom to top, a double-tap gesture, etc.

[0071] In this step, gesture features in each frame of the depth image can be identified, that is, features in the depth image related to the user's gesture actions. Then, multiple gesture features can be matched with multiple preset recognition gestures to obtain the target gesture.

[0072] In step S103, the target control command corresponding to the target gesture is determined from multiple preset control commands based on the target gesture.

[0073] In one possible implementation, a preset correspondence can be used to determine the target control instruction corresponding to the target gesture among multiple preset control instructions. This preset correspondence can, for example, include a correspondence between preset control instructions and gesture actions. The preset control instructions can, for example, include left swipe, right swipe, up swipe, down swipe, play / pause, answer / hang up calls, etc.

[0074] For example, if the target gesture includes a first gesture, the target control command is determined to be a right swipe command. The first gesture could be, for example, a swipe gesture from left to right. Alternatively, if the target gesture includes a second gesture, the target control command is determined to be a left swipe command. The second gesture could be, for example, a swipe gesture from right to left. Alternatively, if the target gesture includes a third gesture, the target control command is determined to be a swipe down command. The third gesture could be, for example, a swipe down from top to bottom. Alternatively, if the target gesture includes a fourth gesture, the target control command is determined to be an up swipe command. The fourth gesture could be, for example, a swipe down from bottom to top. Alternatively, if the target gesture includes a fifth gesture, the target control command is determined to be a play / pause command. The fifth gesture could be, for example, a double-tap or single-tap gesture (triggered by the user pressing their palm towards the screen). Alternatively, if the target gesture includes a sixth gesture, the target control command is determined to be a call answer / hang up command. The sixth gesture could be, for example, a circle gesture.

[0075] For ease of understanding, such as Figure 2 As shown, in a real-world scenario, users can move along a path at a certain distance from the terminal. Figure 2 The user draws a circle using a dotted line. At this time, the terminal's depth sensor can capture multiple consecutive frames of depth images. These depth images contain gesture information of the user's hand gesture. By extracting gesture features from the depth images and then matching these features with multiple preset recognition gestures, the user's target gesture can be confirmed as a circle-drawing gesture. Similarly, as... Figure 3 As shown, users can access the system via... Figure 3 A horizontal dotted line is used to perform a swipe motion from left to right or from right to left, thereby triggering the target gesture of a swipe gesture from left to right or from right to left. For example, ... Figure 4 As shown, users can access the system via... Figure 4 The vertical dotted line is used to perform a swipe motion from top to bottom or from bottom to top, thereby triggering the target gesture of the swipe gesture from top to bottom or from bottom to top.

[0076] In step S104, the target control command is executed.

[0077] For example, if the target control command is determined to be a call answering / hanging up command, and a call is currently connected to the terminal but not yet answered, the target control command can be used to connect the call. Conversely, if a call is currently connected to the terminal, the target control command can be used to hang up the call. Thus, the user can control the answering or hanging up of calls on the terminal using a circular gesture in the air. Similarly, if the target control command is determined to be a right swipe or left swipe command, and the terminal is in a video / music playback interface, the target control command can be used to adjust the playback progress of the audio in the video / music playback interface (or, to control the switching of video or audio; the specific control method can be set according to actual needs, and this disclosure does not specifically limit it). Likewise, if the terminal is in a document display interface (such as a PowerPoint presentation), the target control command can be used to turn right or left pages in the document display interface. Thus, the user can control the adjustment / switching of video / audio playback progress and document page turning using a left or right swipe gesture in the air. For example, if the target control command includes an up / down swipe command, and the current terminal is in a document display interface (such as Word playback), the user can swipe up and down to browse the document in that interface according to the target control command. If the current terminal is in a short video playback interface, the user can switch videos in that short video playback interface according to the target control command. If the current terminal is in the main interface (or a video playback interface or other interfaces, which can be set according to needs), the user can also control the media volume adjustment according to the target control command. In this way, users can control short video switching, media volume adjustment, and document browsing through air swipe gestures. For another example, if the target control command includes a play / pause command, and the video / audio in the terminal's interface is paused, the user can control the video / audio to play according to the target control command. Conversely, if the video / audio in the terminal's interface is playing, the user can control the video / audio to pause playback according to the target control command. In this way, users can control the pause / play of media through air tap gestures.

[0078] It should be noted that the above examples are illustrative and this disclosure is not limited to these examples; other correspondences are also possible.

[0079] In some scenarios, this embodiment can also be used in multi-terminal interconnection situations. For example, terminal A and terminal B are interconnected via Bluetooth, Wi-Fi, etc., and terminal B supports gesture recognition. When terminal A and terminal B are interconnected, the user can control corresponding operations on terminal A through terminal B. For example, the user can use gestures on terminal B to answer a phone call on terminal A. Thus, in interconnected mode, interconnected control between multiple devices can be achieved through air gesture recognition. In other words, in this embodiment, when the terminal is the first device, the target control command can be used to control the first device, and also to control a second device connected to the first device.

[0080] The above method firstly acquires multiple frames of depth images containing user gestures using a depth sensor. Since the depth images only contain the distance between the target object and the depth sensor, it effectively avoids capturing images of the user's face or other user activities, ensuring that user information is not leaked. Secondly, because the depth sensor itself has low power consumption, it does not significantly affect the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, ensuring the accuracy of the depth data and thus improving the accuracy of gesture recognition. By matching multiple frames of depth images with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control command can be determined and executed based on the target gesture. In this way, users can conveniently control the terminal to trigger corresponding target control commands without directly touching the terminal, greatly improving the user experience.

[0081] The following is a detailed explanation of step S102 above, such as... Figure 5 As shown, step S102, which involves matching multiple depth images with multiple preset recognition gestures to obtain the target gesture corresponding to the multiple depth images, may include the following steps:

[0082] In step S1021, for each frame of the depth image, feature extraction is performed on the depth image to obtain the gesture features in the depth image.

[0083] In this step, for each frame of depth image, the hand contour region in the depth image can be identified first, and then the hand contour region can be used to extract features from the depth image to obtain the gesture features in the depth image.

[0084] Specifically, such as Figure 6 As shown, step S1021, which involves extracting features from each frame of the depth image to obtain the gesture features in the depth image, may include the following steps:

[0085] In step S1021a, for each frame of the depth image, the hand contour region in the depth image is determined to obtain a target depth map containing the hand contour region.

[0086] For example, for each frame of this depth image, the depth image of the previous frame can be obtained first. Then, based on the depth image of the current frame and the depth image of the previous frame, the hand contour region in the current frame's depth image is determined using the inter-frame difference method. Since the hand region is constantly moving during a user's gesture, the moving area can be determined using depth images between two consecutive frames, and then the hand contour region in the depth image can be determined based on the moving area. In this way, by continuously analyzing the hand contour region in each frame of the depth image, the user's gesture contour region can be tracked. After determining the hand contour region in each frame of the depth image, to facilitate further feature extraction, the hand in the depth image can be separated from the background region to obtain a target depth map containing the hand contour region.

[0087] In step S1021b, feature extraction is performed on the depth map of the target to obtain the gesture feature.

[0088] This step may include the following steps:

[0089] Step A: Determine the centroid position corresponding to the centroid of the hand contour region in the depth map of each target.

[0090] In this embodiment, to more accurately track the user's hand gestures, it is necessary to determine the movement trajectory of the gestures in multiple frames of depth images. To improve computational efficiency, the centroid of the hand contour region in each frame of the target depth image can be used as the tracking point for that frame of the target depth image. Then, based on the tracking points corresponding to the target depth images in multiple frames, the movement trajectory of the user's hand gestures can be tracked.

[0091] To determine the centroid of the hand contour region in each frame's depth map, the target depth map can be converted into a corresponding binary image. In this binary image, for example, the hand contour region can be set to 1, and the rest can be set to 0. Based on this binary image, the centroid position corresponding to the centroid of the hand contour region can be determined.

[0092] For example, a planar coordinate system can be pre-established based on the depth image. For instance, the planar coordinate system can be established with the centerline of the depth image as the coordinate axis and the center as the origin. In this case, the centroid position includes the centroid coordinates of the hand contour region in the planar coordinate system. It is understood that after converting the target depth map into a binary image, the mass distribution of the hand contour region is uniform; therefore, the centroid coordinates can be set as (cx...).t ,cy t If the centroid coordinates satisfy the following formula:

[0093]

[0094]

[0095] Where t represents different frame numbers, L represents the length of the depth image, and W represents the width of the depth image.

[0096] Step B: For each target depth map, obtain the centroid position corresponding to the target depth map of the previous frame; and determine the gesture feature corresponding to the target depth map of the current frame based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame.

[0097] Step A above yields the centroid position for each target depth map. To further determine the user's gesture trajectory, this step determines the angle α between the centroid of the line containing the centroid in two adjacent target depth maps and the horizontal axis in the planar coordinate system. This centroid angle is then used to determine the gesture feature corresponding to the target depth map. The centroid angle can be determined as follows: The centroid coordinates in the target depth maps at times t and t+1 are set to (x...). t ,y t ) and (x t+1 ,y t+1 If the angle α between the centers of mass is α, then it can be expressed by the following formula:

[0098]

[0099] Specifically, based on the centroid coordinates of the target depth map in the current frame and the target depth map in the previous frame, the angle between the line containing the centroid of the target depth map in the current frame and the line containing the centroid of the target depth map in the previous frame and the horizontal axis of the coordinate system can be determined, thus obtaining the first centroid angle corresponding to the centroid in the target depth map of the current frame. By sequentially determining the first centroid angle for each target depth map, multiple first centroid angles corresponding to multiple target depth maps can be obtained. Then, based on these first centroid angles, the gesture feature corresponding to the target depth map of the current frame can be determined.

[0100] Considering that for some gestures, such as drawing a circle, different users have different gesture habits, such as where to start drawing the circle and whether to draw it clockwise or counterclockwise, resulting in different first centroid angles within the same frame, this issue needs to be addressed to eliminate the influence of different user gesture habits and reduce matching difficulty. The first centroid angle can be quantized (i.e., feature transformation) to convert it into a feature value within a preset range, thus obtaining the gesture feature. This gesture feature is used to characterize the feature values ​​corresponding to different centroid angles. For example, the planar coordinate system can be divided into 12 equal parts, each corresponding to an angle of 30 degrees, and labeled clockwise from the positive direction of the horizontal axis as 1 to 12. In this step, the first centroid angle can be converted into any value within (1, 2, 3, ..., 12) to complete the feature transformation and obtain the corresponding gesture feature.

[0101] Additionally, considering that in some scenarios, the edges of the hand contour region in the depth image may contain interfering factors (such as objects adjacent to the user's hand whose depth values ​​are close to the user's hand's depth value), in order to eliminate the influence of these interfering factors on subsequent gesture recognition, such as... Figure 7 As shown, the method may further include the following steps:

[0102] In step S1021c, the target depth map is filtered based on the hand contour region.

[0103] For example, the hand contour region in the depth map of the target can be filtered using the mean filtering method.

[0104] Accordingly, the feature extraction of the target depth map in step S1021b to obtain the gesture feature may include: extracting features from the filtered target depth map to obtain the gesture feature.

[0105] In step S1022, multiple gesture features are matched with multiple preset recognition gestures to obtain the target gesture.

[0106] Each preset recognition gesture corresponds to a preset sequence feature, which reflects the action characteristics of each gesture. In this step, multiple gesture features are matched with multiple preset recognition gestures to determine the target gesture corresponding to that gesture feature. Furthermore, considering that the time taken by different users to perform different gestures is not fixed, and even for the same user performing the same gesture, the duration cannot be guaranteed to be the same, multiple gesture templates (each with a different completion time) can be set for each preset recognition gesture, and the corresponding sequence features can be determined. In this way, for the same gesture action, multiple templates can be provided for matching, improving the accuracy of recognition.

[0107] In one possible implementation, such as Figure 8 As shown, step S1022, which involves matching multiple gesture features with multiple preset recognition gestures to obtain the target gesture, may include the following steps:

[0108] In step S1022a, a first depth map is determined from the multi-frame depth images.

[0109] The number of the first depth maps is less than the number of the depth images.

[0110] For example, starting with the most recently acquired depth image, depth images corresponding to multiple preset interval frames preceding the frame containing that depth image can be used as the first depth map. These preset interval frames can be, for example, but not limited to, 14, 18, 20, 22, and 24 frames of different lengths. That is, depth images 14 frames, 18 frames, 20 frames, 22 frames, and 24 frames from the most recently acquired depth image can be used as the first depth map. It should be noted that these preset interval frames can be set according to the number of frames corresponding to the sequence features included in the gesture template of the preset recognition gesture.

[0111] In step S1022b, for each frame of the first depth map, the Euclidean distance value between the gesture feature corresponding to the first depth map and the sequence feature corresponding to each preset recognition gesture is determined sequentially.

[0112] Since the time taken by different users to perform different gestures is not fixed, even for the same user performing the same gesture, the duration cannot be guaranteed to be the same. Therefore, in this step, gesture matching can be performed by calculating the similarity between two sequences with unequal time durations. Specifically, the similarity can be measured by comparing the Euclidean distance values ​​between two sequences; the smaller the distance value, the more similar they are. For the gesture features of the first depth map in each frame, the Euclidean distance values ​​between the sequence features corresponding to each preset recognition gesture can be determined sequentially. As mentioned above, each preset recognition gesture may contain multiple gesture templates. In this case, for each preset recognition gesture, the Euclidean distance values ​​between the gesture feature and the sequence features corresponding to each gesture template can be determined sequentially, and the minimum value among the Euclidean distance values ​​corresponding to multiple gesture templates is taken as the Euclidean distance value corresponding to the preset recognition gesture.

[0113] In step S1022c, for each preset recognition gesture, the sum of the Euclidean distance values ​​corresponding to the first depth map of multiple frames is determined to obtain the first distance value.

[0114] In this step, by determining the sum of the Euclidean distance values ​​corresponding to the first depth images across multiple frames, it is equivalent to determining the similarity between the current user's gesture action (i.e., the multi-frame depth images) and each preset recognition gesture, i.e., the first distance value.

[0115] In step S1022d, the minimum value among the multiple first distance values ​​is taken as the target distance value.

[0116] As mentioned earlier, the smaller the distance value, the higher the similarity. In this case, the minimum value among the first distance values ​​can be used as the target distance value. That is to say, the preset recognition gesture corresponding to the target distance value is the most similar to the current user's gesture, and the preset recognition gesture corresponding to the target distance value can be used as a candidate gesture.

[0117] In step S1022e, the target gesture is determined based on the target distance value.

[0118] In one possible implementation, if the target distance value is less than or equal to a preset distance threshold, it can be indicated that the candidate gesture has been successfully matched with multiple frames of depth images, that is, the target gesture is the preset recognition gesture (i.e., candidate gesture) corresponding to the target distance value.

[0119] In another possible implementation, to recognize the user's gesture and take into account the different times when different users complete the gesture, generally, during the first matching process (i.e., steps S1022a to S1022d), a first depth map is often selected from an interval with a relatively long time span. However, precisely because the set time span is relatively long, it may affect the recognition rate. To further accurately determine the user's target gesture, the gesture phase difference can also be judged. Specifically, this can include the following steps:

[0120] S1, if the target distance value is less than or equal to a preset distance threshold, determine a second depth map from the multi-frame depth images.

[0121] The number of the second depth maps is less than the number of the depth images, and the second depth maps are consecutive multi-frame images.

[0122] In this step, the preset recognition gesture corresponding to the target distance value is first used as a candidate gesture. Then, the first depth map corresponding to the minimum value among the multiple Euclidean distance values ​​(obtained in step S1022b) corresponding to the candidate gesture is used as the third depth map. Next, the second depth map is determined from the multi-frame depth images based on the frame number corresponding to the third depth map. For example, the depth image between the most recently acquired depth image and the third depth map in the multi-frame depth images can be used as the second depth map. For instance, if the frame number corresponding to the third depth map is frame 14, and the frame number of the most recently acquired depth image in the multi-frame depth images is frame 30, then the second depth map is the depth image between frames 14 and 30.

[0123] S2, determine the gesture phase difference between each two adjacent frames of the second depth map.

[0124] Specifically, firstly, the second centroid angle corresponding to each frame of the second depth map can be obtained (this can be achieved through step S1022b above). Then, for each pair of adjacent second depth maps, the difference between the second centroid angles corresponding to the current two adjacent second depth maps is taken as the gesture phase difference between the current two adjacent second depth maps. For example, if the second centroid angle corresponding to the second depth map in frame t is 45 degrees and the second centroid angle corresponding to the second depth map in frame t+1 is 50 degrees, then the gesture phase difference is 5 degrees. This process continues to obtain the gesture phase difference between each pair of adjacent second depth maps.

[0125] S3, the sum of the phase differences of multiple gestures is taken as the target phase difference.

[0126] The target phase difference is obtained by adding the multiple gesture phase differences obtained in step S2 above.

[0127] S4, determine the target gesture based on the target phase difference.

[0128] Specifically, firstly, a preset recognition gesture corresponding to the target distance value can be used as a candidate gesture. Then, a preset phase difference range corresponding to the candidate gesture is obtained. This preset phase difference range is a pre-set range of phase difference changes for each candidate gesture. It is understood that different gestures may have different phase changes during movement; therefore, different phase difference ranges can be set for different gestures. Then, if the target phase difference is within the preset phase difference range, the candidate gesture is determined to be the target gesture. In this way, by judging based on phase, it is possible to more accurately determine whether the gesture triggered by the current user matches the candidate gesture. If the target phase difference is within the preset phase difference range, it can be determined that the gesture triggered by the current user matches the candidate gesture, that is, the target gesture is the candidate gesture.

[0129] Both the initial matching based on the centroid angle and the secondary matching based on the phase difference are performed on the same plane, meaning that the change in depth is not determined. However, for some gestures, such as double-tap or single-tap gestures, the user's gesture may not move much within the same plane, but a significant change occurs in depth. Therefore, to recognize such gestures, if the candidate gesture is a specified gesture, depth data can be acquired from each frame of the depth image. This specified gesture can be, for example, but not limited to, a double-tap or single-tap gesture. Then, if the trend of change in the depth data corresponding to the multi-frame depth images is the same as the preset trend of change corresponding to the specified gesture, and the target phase difference is within the preset phase difference value range, the candidate gesture is determined to be the target gesture. This preset trend of change can be pre-set according to actual needs; for example, it could be a decrease followed by an increase (corresponding to a single-tap gesture), or a decrease followed by an increase followed by a decrease followed by an increase (corresponding to a double-tap gesture).

[0130] For example, after identifying candidate gestures with high similarity through the centroid angle, if the target phase difference is within the preset phase difference value range corresponding to the candidate gesture, depth data corresponding to multiple frames of depth images can be obtained. It is understood that each pixel in a depth image generally corresponds to the depth information of that point. To facilitate the determination of depth changes in user gestures, in this embodiment, the average value of the depth information corresponding to the hand contour region in the depth image can be used as the depth data. Alternatively, the depth information corresponding to the centroid of the hand contour region in the depth image can also be used as the depth data. Then, the trend of depth data change in consecutive frames of depth images can be determined, and it can be determined whether the trend of depth data change is the same as the preset trend corresponding to the specified gesture. If they are the same, then the candidate gesture can be determined to be the target gesture.

[0131] It should be noted that if the candidate gesture does not belong to the specified gesture, the comparison of the change trend of the depth data can be omitted. If the target phase difference is within the preset phase difference value range, the candidate gesture can be determined as the target gesture.

[0132] The above method firstly acquires multiple frames of depth images containing user gestures using a depth sensor. Since the depth images only contain the distance between the target object and the depth sensor, it effectively avoids capturing images of the user's face or other user activities, ensuring that user information is not leaked. Secondly, because the depth sensor itself has low power consumption, it does not significantly affect the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, ensuring the accuracy of the depth data and thus improving the accuracy of gesture recognition. By matching multiple frames of depth images with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control command can be determined and executed based on the target gesture. In this way, users can conveniently control the terminal to trigger corresponding target control commands without directly touching the terminal, greatly improving the user experience.

[0133] Figure 9 This is a block diagram illustrating a control device for a device according to an exemplary embodiment, such as... Figure 9 As shown, the device 200 includes:

[0134] The acquisition module 201 is configured to acquire multi-frame depth images containing user gestures collected by the terminal's depth sensor;

[0135] The matching module 202 is configured to match the multi-frame depth image with multiple preset recognition gestures to obtain the target gesture corresponding to the multi-frame depth image;

[0136] The determining module 203 is configured to determine the target control command corresponding to the target gesture from a plurality of preset control commands based on the target gesture.

[0137] Execution module 204 is configured to execute the target control instruction.

[0138] Optionally, such as Figure 10 As shown, the matching module 202 includes:

[0139] The extraction submodule 2021 is configured to perform feature extraction on the depth image for each frame of the depth image to obtain the gesture features in the depth image;

[0140] The matching submodule 2022 is configured to perform feature matching between multiple gesture features and multiple preset recognition gestures to obtain the target gesture.

[0141] Optionally, the extraction submodule 2021 is configured to, for each frame of the depth image, determine the hand contour region in the depth image to obtain a target depth map containing the hand contour region; and perform feature extraction on the target depth map to obtain the gesture feature.

[0142] Optionally, the extraction submodule 2021 is configured to obtain the depth image corresponding to the previous frame for each frame of the depth image; and determine the hand contour region in the depth image corresponding to the current frame by means of the inter-frame difference method based on the depth image corresponding to the current frame and the depth image corresponding to the previous frame.

[0143] Optionally, such as Figure 11 As shown, the matching module also includes a filtering submodule 2023, which is configured to filter the target depth map based on the hand contour region;

[0144] The extraction submodule 2021 is configured to extract features from the filtered target depth map to obtain the gesture features.

[0145] Optionally, the extraction submodule 2021 is configured to determine the centroid position corresponding to the centroid of the hand contour region in each target depth map; for each target depth map, obtain the centroid position corresponding to the target depth map of the previous frame; and determine the gesture feature corresponding to the target depth map of the current frame based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame.

[0146] Optionally, the extraction submodule 2021 is configured to convert the target depth map into a corresponding binary image; and determine the centroid position corresponding to the centroid of the hand contour region based on the binary image.

[0147] Optionally, the centroid position includes the centroid coordinates of the hand contour region in a planar coordinate system, which is pre-established based on the depth image; the extraction submodule 2021 is configured to determine the angle between the line containing the centroid of the target depth map of the current frame and the centroid of the target depth map of the previous frame and the horizontal axis of the planar coordinate system, based on the centroid coordinates corresponding to the target depth map of the current frame and the centroid coordinates corresponding to the target depth map of the previous frame, to obtain the first centroid angle corresponding to the centroid in the target depth map of the current frame; and determine the gesture feature corresponding to the target depth map of the current frame based on the first centroid angle.

[0148] Optionally, the gesture feature is used to characterize the feature values ​​corresponding to different centroid angles. The extraction submodule 2021 is configured to perform feature transformation on the first centroid angle so that the first centroid angle is converted into feature values ​​within a preset numerical range to obtain the gesture feature.

[0149] Optionally, the matching submodule 2022 is configured to determine a first depth map from the multi-frame depth images; the number of the first depth maps is less than the number of depth images; for each frame of the first depth map, sequentially determine the Euclidean distance value between the gesture feature corresponding to the first depth map and the sequence feature corresponding to each preset recognition gesture; for each preset recognition gesture, determine the sum of the Euclidean distance values ​​corresponding to the first depth maps in multiple frames to obtain a first distance value; take the minimum value among the multiple first distance values ​​as the target distance value; and determine the target gesture based on the target distance value.

[0150] Optionally, the matching submodule 2022 is configured to determine a second depth map from the multi-frame depth images when the target distance value is less than or equal to a preset distance threshold; the number of second depth maps is less than the number of depth images, and the second depth map is a series of consecutive multi-frame images; determine the gesture phase difference between each pair of adjacent second depth maps; take the sum of the multiple gesture phase differences as the target phase difference; and determine the target gesture based on the target phase difference.

[0151] Optionally, the matching submodule 2022 is configured to obtain the second centroid angle corresponding to the second depth map in each frame; and for each two adjacent frames of the second depth map, take the difference between the second centroid angles corresponding to the current two adjacent frames of the second depth map as the gesture phase difference corresponding to the current two adjacent frames of the second depth map.

[0152] Optionally, the matching submodule 2022 is configured to take the preset recognition gesture corresponding to the target distance value as a candidate gesture; obtain the preset phase difference value range corresponding to the candidate gesture; and determine the candidate gesture as the target gesture if the target phase difference is within the preset phase difference value range.

[0153] Optionally, such as Figure 12 As shown, the matching module 202 also includes:

[0154] The acquisition submodule 2024 is configured to acquire depth data in each frame of the depth image when the candidate gesture is the specified gesture;

[0155] The matching submodule 2022 is configured to determine the candidate gesture as the target gesture when the change trend of the depth data corresponding to the multi-frame depth image is the same as the preset change trend corresponding to the specified gesture and the target phase difference is within the preset phase difference value range.

[0156] Optionally, the determining module 203 is configured to determine that the target control command is a right swipe command if the target gesture includes a first gesture; or,

[0157] If the target gesture includes a second gesture, the target control command is determined to be a left swipe command; or,

[0158] If the target gesture includes a third gesture, the target control command is determined to be a swipe down command; or,

[0159] If the target gesture includes a fourth gesture, the target control command is determined to be an up swipe command; or,

[0160] If the target gesture includes a fifth gesture, the target control command is determined to be a play / pause command; or,

[0161] If the target gesture includes a sixth gesture, the target control command is determined to be a call connection / hang-up command.

[0162] The above method firstly acquires multiple frames of depth images containing user gestures using a depth sensor. Since the depth images only contain the distance between the target object and the depth sensor, it effectively avoids capturing images of the user's face or other user activities, ensuring that user information is not leaked. Secondly, because the depth sensor itself has low power consumption, it does not significantly affect the overall power consumption of the terminal, and the infrared laser emitted by the depth sensor is not easily affected by the environment, ensuring the accuracy of the depth data and thus improving the accuracy of gesture recognition. By matching multiple frames of depth images with multiple preset recognition gestures, the target gesture corresponding to the user's gesture can be accurately identified, and the corresponding target control command can be determined and executed based on the target gesture. In this way, users can conveniently control the terminal to trigger corresponding target control commands without directly touching the terminal, greatly improving the user experience.

[0163] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0164] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the control method for the device provided in this disclosure.

[0165] Figure 13 This is a block diagram illustrating an electronic device 300 according to an exemplary embodiment. For example, the electronic device 300 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0166] Reference Figure 13 The electronic device 300 may include one or more of the following components: processing component 302, memory 304, power supply component 306, multimedia component 308, audio component 310, input / output interface 312, sensor component 314, and communication component 316.

[0167] Processing component 302 typically controls the overall operation of electronic device 300, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the device control method described above. Furthermore, processing component 302 may include one or more modules to facilitate interaction between processing component 302 and other components. For example, processing component 302 may include a multimedia module to facilitate interaction between multimedia component 308 and processing component 302.

[0168] Memory 304 is configured to store various types of data to support the operation of electronic device 300. Examples of such data include instructions for any application or method operating on electronic device 300, contact data, phonebook data, messages, pictures, videos, etc. Memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0169] Power supply component 306 provides power to various components of electronic device 300. Power supply component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 300.

[0170] Multimedia component 308 includes a screen that provides an output interface between the electronic device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 308 includes a front-facing camera and / or a rear-facing camera. When the electronic device 300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0171] Audio component 310 is configured to output and / or input audio signals. For example, audio component 310 includes a microphone (MIC) configured to receive external audio signals when electronic device 300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 304 or transmitted via communication component 316. In some embodiments, audio component 310 also includes a speaker for outputting audio signals.

[0172] Input / output interface 312 provides an interface between processing component 302 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0173] Sensor assembly 314 includes one or more sensors for providing state assessments of various aspects of electronic device 300. For example, sensor assembly 314 can detect the on / off state of electronic device 300, the relative positioning of components such as the display and keypad of electronic device 300, changes in position of electronic device 300 or a component of electronic device 300, the presence or absence of user contact with electronic device 300, orientation or acceleration / deceleration of electronic device 300, and temperature changes of electronic device 300. Sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 314 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0174] Communication component 316 is configured to facilitate wired or wireless communication between electronic device 300 and other devices. Electronic device 300 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0175] In an exemplary embodiment, the electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the control method of the device.

[0176] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, which can be executed by a processor 320 of an electronic device 300 to perform the control method of the device. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0177] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the control method of the device described above when executed by the programmable device.

[0178] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0179] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for controlling a device, characterized in that, The method includes: In response to the depth sensor continuously collecting target objects with the same depth value within a preset range within a preset time period, the terminal's gesture recognition function is enabled. Acquire multi-frame depth images containing user gestures captured by the terminal's depth sensor; For each frame of the depth image, the hand contour region in the depth image is determined to obtain a target depth map containing the hand contour region, and feature extraction is performed on the target depth map to obtain the gesture features in the depth image; The step of extracting features from the target depth map to obtain gesture features in the depth image includes: Determine the centroid position corresponding to the centroid of the hand contour region in each of the target depth maps; For each target depth map, the centroid position corresponding to the target depth map of the previous frame is obtained; and the gesture feature corresponding to the target depth map of the current frame is determined based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame. The multiple gesture features are matched with multiple preset recognition gestures to obtain the target gesture corresponding to the multi-frame depth image; Based on the target gesture, determine the target control command corresponding to the target gesture from multiple preset control commands; Execute the target control command; The centroid position includes the centroid coordinates of the hand contour region in a planar coordinate system, which is pre-established based on the depth image; determining the gesture feature corresponding to the target depth map of the current frame based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame includes: Based on the centroid coordinates corresponding to the target depth map of the current frame and the centroid coordinates corresponding to the target depth map of the previous frame, determine the angle between the straight line containing the centroid of the target depth map of the current frame and the centroid of the target depth map of the previous frame and the horizontal axis of the plane coordinate system, and obtain the first centroid angle corresponding to the centroid in the target depth map of the current frame. Based on the first centroid angle, the gesture feature corresponding to the target depth map of the current frame is determined.

2. The method according to claim 1, characterized in that, Determining the hand contour region in each frame of the depth image includes: For each frame of the depth image, the depth image corresponding to the previous frame is obtained; and based on the depth image corresponding to the current frame and the depth image corresponding to the previous frame, the hand contour region in the depth image corresponding to the current frame is determined by the inter-frame difference method.

3. The method according to claim 1, characterized in that, The method further includes: The target depth map is filtered based on the hand contour region; The step of extracting features from the target depth map to obtain gesture features in the depth image includes: The gesture features are obtained by extracting features from the filtered target depth map.

4. The method according to claim 1, characterized in that, Determining the centroid position corresponding to the centroid of the hand contour region in each target depth map includes: Convert the target depth map into a corresponding binary image; Based on the binary image, determine the centroid position corresponding to the centroid of the hand contour region.

5. The method according to claim 1, characterized in that, The gesture feature is used to characterize the feature values ​​corresponding to different centroid angles. Determining the gesture feature corresponding to the target depth map of the current frame based on the first centroid angle includes: The first centroid angle is transformed into a feature value within a preset range to obtain the gesture feature.

6. The method according to claim 1, characterized in that, The step of performing feature matching between multiple gesture features and multiple preset recognition gestures to obtain the target gesture corresponding to the multi-frame depth image includes: A first depth map is determined from the multiple depth images; the number of first depth maps is less than the number of depth images. For each frame of the first depth map, the Euclidean distance between the gesture features corresponding to the first depth map and the sequence features corresponding to each preset recognition gesture is determined sequentially. For each preset recognition gesture, the sum of the Euclidean distance values ​​corresponding to the first depth map of multiple frames is determined to obtain the first distance value; The minimum value among multiple first distance values ​​is taken as the target distance value; The target gesture is determined based on the target distance value.

7. The method according to claim 6, characterized in that, Determining the target gesture based on the target distance value includes: If the target distance value is less than or equal to a preset distance threshold, a second depth map is determined from the multi-frame depth images; the number of second depth maps is less than the number of depth images, and the second depth map is a series of consecutive multi-frame images; Determine the corresponding gesture phase difference between each two adjacent frames of the second depth map; The sum of the phase differences of the multiple gestures is taken as the target phase difference; The target gesture is determined based on the target phase difference.

8. The method according to claim 7, characterized in that, The determination of the gesture phase difference between each pair of adjacent second depth maps includes: Obtain the second centroid angle corresponding to the second depth map in each frame; For each pair of adjacent second depth maps, the difference between the angles between the second centroids of the two adjacent second depth maps is taken as the gesture phase difference between the two adjacent second depth maps.

9. The method according to claim 7, characterized in that, Determining the target gesture based on the target phase difference includes: The preset recognition gesture corresponding to the target distance value is used as a candidate gesture; Obtain the preset phase difference value range corresponding to the candidate gesture; If the target phase difference is within the preset phase difference value range, the candidate gesture is determined to be the target gesture.

10. The method according to claim 9, characterized in that, The method further includes: If the candidate gesture is a specified gesture, then the depth data in each frame of the depth image is obtained; Determining the candidate gesture as the target gesture when the target phase difference is within the preset phase difference value range includes: If the trend of change of depth data corresponding to the multi-frame depth images is the same as the preset trend of change corresponding to the specified gesture, and the target phase difference is within the preset phase difference value range, the candidate gesture is determined to be the target gesture.

11. The method according to any one of claims 1 to 10, characterized in that, The step of determining the target control command corresponding to the target gesture from multiple preset control commands based on the target gesture includes: If the target gesture includes the first gesture, the target control command is determined to be a right swipe command; or, If the target gesture includes a second gesture, the target control command is determined to be a left swipe command; or, If the target gesture includes a third gesture, the target control command is determined to be a swipe down command; or, If the target gesture includes a fourth gesture, the target control command is determined to be an up swipe command; or, If the target gesture includes a fifth gesture, the target control command is determined to be a play / pause command; or, If the target gesture includes a sixth gesture, the target control command is determined to be a call connection / hang-up command.

12. A control device for an equipment, characterized in that, The device includes: The acquisition module is configured to activate the gesture recognition function of the terminal in response to the depth sensor continuously acquiring target objects with the same depth value within a preset time period; and to acquire multi-frame depth images containing user gesture actions acquired by the depth sensor of the terminal. The matching module is configured to, for each frame of the depth image, determine the hand contour region in the depth image to obtain a target depth map containing the hand contour region; determine the centroid position corresponding to the centroid of the hand contour region in each target depth map; for each target depth map, obtain the centroid position corresponding to the target depth map of the previous frame; and determine the gesture features corresponding to the target depth map of the current frame based on the centroid position corresponding to the target depth map of the current frame and the centroid position corresponding to the target depth map of the previous frame; perform feature matching between multiple gesture features and multiple preset recognition gestures to obtain the target gesture corresponding to the multi-frame depth images; the centroid position includes the centroid of the hand contour region in the plane. The centroid coordinates in the target depth map are defined in a pre-established planar coordinate system based on the depth image. Determining the gesture feature corresponding to the target depth map of the current frame based on the centroid positions of the current frame and the previous frame includes: determining the angle between the line containing the centroid of the target depth map of the current frame and the centroid of the target depth map of the previous frame and the horizontal axis of the planar coordinate system, based on the centroid coordinates of the current frame and the previous frame, to obtain a first centroid angle corresponding to the centroid in the target depth map of the current frame; and determining the gesture feature corresponding to the target depth map of the current frame based on the first centroid angle. The determining module is configured to determine the target control instruction corresponding to the target gesture from a plurality of preset control instructions based on the target gesture; The execution module is configured to execute the target control instructions.

13. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1 to 11 when executable instructions stored in the memory are invoked.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 1 to 11.

Images