A hand gesture recognition method and device

Abstract

Embodiments of the present application provide a hand action recognition method and device, an electronic device acquires point cloud data collected by a laser radar sensor and an infrared image acquired by an infrared sensor; a candidate point cloud point set corresponding to a hand in the point cloud data is acquired; a first mean value of an infrared value corresponding to the candidate point cloud point set in the infrared image is acquired, if the first mean value is greater than a preset infrared value threshold, the candidate point cloud point set is determined as a target point cloud point set corresponding to the hand; and position information of the hand is determined according to position information of each point cloud point in the target point cloud point set. In the embodiments of the present application, the electronic device determines a candidate point cloud point set that may be a hand according to point cloud data, thereby avoiding privacy exposure, and the electronic device further verifies the candidate point cloud point set according to an infrared image, thereby improving the accuracy of hand action recognition, and the point cloud data is three-dimensional data, which can be used to identify radial action, thereby improving the accuracy of action recognition.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and device for hand motion recognition. Background Technology

[0002] Hand gesture recognition can be used in various scenarios, such as smart homes and smart factories. When a user makes a hand gesture, a sensor recognizes the gesture, and the controlled object responds based on the sensor's recognition result. For example, a user can perform the gesture "open palm - clench fist - move forward - open palm" to achieve a certain control purpose.

[0003] Most current hand motion recognition methods are based on red-green-blue (RGB) images. However, RGB images contain rich color information, posing a risk of privacy breaches. Furthermore, hand motion recognition based on RGB images generally cannot recognize radial movements, only in-plane movements (up, down, left, and right), resulting in significant spatial limitations. Summary of the Invention

[0004] This application provides a hand motion recognition method and device to solve the problems of privacy exposure and inability to recognize radial motions when using RGB images for motion recognition in the prior art.

[0005] In a first aspect, embodiments of this application provide a hand gesture recognition method, the method comprising:

[0006] Acquire point cloud data collected by the lidar sensor and infrared images acquired by the infrared sensor;

[0007] Obtain the candidate point cloud set corresponding to the hand in the point cloud data;

[0008] Obtain the first mean value of each infrared value corresponding to the candidate point cloud set in the infrared image. If the first mean value is greater than a preset infrared value threshold, then determine that the candidate point cloud set is the target point cloud set corresponding to the hand.

[0009] The position information of the hand is determined based on the spatial coordinates of each point in the target point cloud set.

[0010] Secondly, embodiments of this application also provide an electronic device, the electronic device including a processor, the processor being configured to execute a computer program stored in a memory to implement the steps of the hand gesture recognition method as described above.

[0011] In this embodiment, point cloud data collected by a lidar sensor and infrared images acquired by an infrared sensor are obtained; a candidate point cloud set corresponding to the hand is obtained from the point cloud data; a first mean of the infrared values ​​corresponding to the candidate point cloud set in the infrared image is obtained; if the first mean is greater than a preset infrared value threshold, the candidate point cloud set is determined to be the target point cloud set corresponding to the hand; the position information of the hand is determined based on the position information of each point cloud point in the target point cloud set. In this embodiment, the electronic device determines the candidate point cloud set that may be the hand based on the point cloud data, avoiding privacy exposure. Furthermore, the electronic device further verifies the candidate point cloud set based on the infrared image, improving the accuracy of hand action recognition. Simultaneously, the point cloud data is three-dimensional data, which can be used to recognize radial movements, further improving the accuracy of action recognition. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of a hand motion recognition process provided in an embodiment of this application;

[0014] Figure 2 A schematic diagram illustrating the process of determining a candidate point cloud set provided in an embodiment of this application;

[0015] Figure 3 This is a schematic diagram illustrating the process of determining target point cloud points provided in an embodiment of this application;

[0016] Figure 4 This is a schematic diagram illustrating the process of determining other point cloud points of a target, provided in an embodiment of this application.

[0017] Figure 5 A schematic diagram of a process for control based on hand movements provided in an embodiment of this application;

[0018] Figure 6 This is a flowchart illustrating the process of determining hand position information provided in an embodiment of this application.

[0019] Figure 7 A schematic diagram of a hand motion recognition device provided in an embodiment of this application;

[0020] Figure 8 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] To avoid privacy exposure and improve the accuracy of hand gesture recognition, this application provides a hand gesture recognition method and device.

[0023] In this embodiment of the application, the electronic device acquires point cloud data collected by the lidar sensor and infrared images acquired by the infrared sensor; acquires a candidate point cloud set corresponding to the hand in the point cloud data; acquires the first mean value of each infrared value corresponding to the candidate point cloud set in the infrared image; if the first mean value is greater than a preset infrared value threshold, the candidate point cloud set is determined to be the target point cloud set corresponding to the hand; and the position information of the hand is determined according to the spatial coordinates of each point cloud point in the target point cloud set.

[0024] Figure 1 A schematic diagram of a hand motion recognition process provided in this application embodiment, the process including:

[0025] S101: Acquire point cloud data collected by the lidar sensor and infrared images acquired by the infrared sensor.

[0026] The hand gesture recognition method provided in this application is applied to an electronic device, which may be a PC or a server.

[0027] To avoid leaking user privacy, in this embodiment of the application, the electronic device performs hand gesture recognition based on point cloud data and infrared images.

[0028] Specifically, in this embodiment, the lidar sensor emits laser pulses, which are reflected back to the lidar sensor after reaching the surface of the hand. The lidar sensor calculates the three-dimensional spatial position of the hand based on the time interval between emission and reception. By emitting a large number of laser pulses in different directions and receiving the echoes, the lidar sensor can form three-dimensional point cloud data, which means that the point cloud data consists of a large number of three-dimensional spatial points.

[0029] Furthermore, in this embodiment, an infrared sensor is used to acquire infrared band information (i.e., temperature) within the sensing scene and generate an infrared image, wherein the infrared image carries the infrared value corresponding to each point. It should be noted that the infrared wavelength range emitted by the infrared sensor is approximately 0.75 to 100 μm, and the temperature of the human body is generally higher than that of environmental objects; therefore, the human body has a clear outline in the infrared image.

[0030] Furthermore, in this embodiment, the image size of the point cloud data is the same as that of the infrared image, and they are strictly aligned when acquiring the point cloud data and the infrared image. That is, for any point cloud point in the point cloud data, the corresponding infrared value of the point cloud point can be found in the infrared image.

[0031] Specifically, in this embodiment of the application, the density requirement for the point cloud data acquired by the electronic device is: there is at least one point cloud point within 1 cubic centimeter at the human limb; the resolution requirement for the infrared image is: in the usage scenario, the actual plane size represented by each pixel at the human body is at most 1cm×1cm.

[0032] Furthermore, in this embodiment, the lidar sensor acquires point cloud data, and the infrared sensor acquires infrared images. The physical distance between the point cloud data and the infrared images is very close. During data acquisition, they are strictly aligned in the plane perpendicular to the line of sight of the lidar sensor and the infrared sensor. That is, the point cloud points in the point cloud data and the corresponding pixels in the infrared image both indicate the same location of the same target object, and the field of view of the lidar sensor and the infrared sensor are the same. The line of sight of the lidar sensor and the infrared sensor is approximately horizontal, i.e., approximately at eye level, not at a downward angle.

[0033] S102: Obtain the candidate point cloud set corresponding to the hand in the point cloud data.

[0034] In this embodiment, the lidar sensor emits a laser pulse, which is reflected back to the lidar sensor after reaching the surface of the hand. The lidar sensor calculates the three-dimensional spatial position of the hand based on the time interval between emission and reception. In other words, in this embodiment, the electronic device can determine point cloud points that may be on the hand based on point cloud data.

[0035] Based on this, in the embodiments of this application, the electronic device determines each point cloud point in the point cloud data that may correspond to the hand according to the spatial coordinates corresponding to each point cloud point, and determines each point cloud point as a candidate point cloud point set that may correspond to the hand.

[0036] S103: Obtain the first mean value of each infrared value corresponding to the candidate point cloud set in the infrared image. If the first mean value is greater than a preset infrared value threshold, then determine that the candidate point cloud set is the target point cloud set corresponding to the hand.

[0037] Since the image area of ​​the point cloud data is the same as that of the infrared image and they are strictly aligned during data acquisition, for any point cloud point p0, it can be mapped to a pixel in the infrared image based on the correspondence between its x and y coordinates and those of the infrared image. Because each point cloud point can be mapped to a pixel in the infrared image, the infrared value of the corresponding pixel in the infrared image can be assigned to the point cloud point. That is, any point cloud point p0 is assigned the infrared value r0, meaning that point P0 has the information (x0, y0, z0, r0), which is (x-coordinate, y-coordinate, z-coordinate, infrared value).

[0038] Based on this, in this embodiment of the application, after the electronic device determines a candidate point cloud set that is suspected to correspond to the hand, the electronic device will further determine whether the candidate point cloud set is the target point cloud set corresponding to the hand based on the infrared image.

[0039] Specifically, in the embodiments of this application, since the infrared sensor can sense and acquire infrared band information in the scene, and since the human body temperature is generally higher than the ambient temperature, the outline of the hand will appear in the infrared image.

[0040] Based on this, in this embodiment of the application, the electronic device pre-stores the infrared value threshold corresponding to the human body, and the electronic device obtains the first mean value of each infrared value corresponding to the candidate point cloud set, and determines whether the candidate point cloud set is the target point cloud set corresponding to the hand based on the first mean value and the infrared value threshold value.

[0041] Specifically, in this embodiment of the application, if the electronic device determines that the first average value is greater than the preset infrared value threshold, then the electronic device determines that the candidate point cloud set is the target point cloud set corresponding to the hand.

[0042] The electronic device can determine whether the candidate point cloud set is the target point cloud set corresponding to the hand using the following formula:

[0043] r avr = (r1+r2+……+r n ) / n;

[0044] If r avr If t1 >, then determine whether the candidate point cloud set is the target point cloud set corresponding to the hand.

[0045] Where, r avr Let t1 be the first mean, t1 be the infrared threshold, and r1, r2, ..., r n is the infrared value of each point in the candidate point cloud set, and n is the number of point clouds in the candidate point cloud set.

[0046] S104: Determine the position information of the hand based on the spatial coordinates of each point in the target point cloud set.

[0047] In this embodiment of the application, after determining the target point cloud set corresponding to the hand, the electronic device can determine the position information of the hand based on the spatial coordinates of each point cloud point in the target point cloud set.

[0048] Specifically, in this embodiment, the electronic device can generate a spatial coordinate matrix based on the spatial coordinates of each point cloud point, and determine the position of the hand using this spatial coordinate matrix; or, the electronic device can select one point cloud point from each point cloud point and determine the spatial coordinates of that point cloud point as the position information of the hand. Furthermore, the electronic device can also determine the position information of the hand in other ways, which are not limited here.

[0049] In this embodiment, the electronic device determines a candidate point cloud set that may be a hand based on point cloud data, thus avoiding privacy exposure. Furthermore, the electronic device further verifies the candidate point cloud set based on infrared images, improving the accuracy of hand action recognition. At the same time, the point cloud data is three-dimensional data, which can be used to recognize radial movements, further improving the accuracy of action recognition.

[0050] To perform hand motion recognition based on point cloud data and avoid privacy exposure, based on the above embodiments, in this application embodiment, obtaining the candidate point cloud set corresponding to the hand in the point cloud data includes:

[0051] Based on the spatial distance between every two point cloud points in the point cloud set and the spatial coordinates of each point cloud point, determine the target point cloud point corresponding to the high point of the hand;

[0052] Based on the target location information of the target point cloud and the other location information of other point cloud points besides the target point cloud, at least one other target point cloud point corresponding to the other position of the hand is determined.

[0053] The target point cloud points and at least one other target point cloud points are determined as the candidate point cloud point set.

[0054] In this embodiment, the electronic device can determine the candidate point cloud set corresponding to the hand in the point cloud data based on the spatial coordinates of each point cloud point. It should be noted that in this embodiment, when constructing the coordinate system, the plane perpendicular to the line of sight of the lidar sensor can be defined as the plane containing the X and Y axes, and the direction parallel to the line of sight of the lidar sensor and pointing towards the sensor can be defined as the positive direction of the Z axis. The coordinate system of the infrared image is set in the same way as the plane containing the X and Y axes, with the origin of the infrared image located at the bottom left pixel, the rightward direction as the positive X-axis direction, and the upward direction as the positive Y-axis direction.

[0055] Based on this, in the embodiments of this application, when the electronic device determines the candidate point cloud point set corresponding to the hand, the electronic device first determines the target point cloud point set corresponding to the high point of the hand, and then determines at least one other target point cloud point corresponding to other positions of the hand according to the positional relationship between the other point cloud point set and the target point cloud point set.

[0056] Specifically, in this embodiment, the electronic device determines the target point cloud point corresponding to the high point of the hand based on the spatial distance between any two point cloud points in the point cloud set and the spatial coordinates of each point cloud point. Then, based on the target position information of the target point cloud point and the position information of other point cloud points in the point cloud data (excluding the target point cloud point), the electronic device determines at least one other target point cloud point corresponding to other positions of the hand.

[0057] Figure 2 This is a schematic diagram illustrating the process of determining candidate point cloud point sets provided in an embodiment of this application. Figure 2 As shown, the process includes:

[0058] S201: Determine the target point cloud point corresponding to the high point of the hand based on the spatial distance between every two point cloud points in the point cloud set and the spatial coordinates of each point cloud point.

[0059] S202: Based on the target position information of the target point cloud and the other position information of other point cloud points besides the target point cloud, determine at least one other target point cloud corresponding to other positions of the hand.

[0060] S203: Determine the target point cloud points and at least one other target point cloud points as a candidate point cloud point set.

[0061] To perform hand motion recognition based on point cloud data and avoid privacy exposure, in this embodiment of the application, based on the above embodiments, the step of determining the target point cloud point corresponding to the high point of the hand according to the spatial distance between every two point cloud points in the point cloud set and the spatial coordinates of each point cloud point includes:

[0062] For each first point cloud point in the point cloud data, determine each second point cloud point whose spatial distance from the first point cloud point is less than a first preset threshold, and determine each second point cloud point as the first point cloud point set corresponding to the first point cloud point;

[0063] For each first point cloud set, determine whether the ordinate of the first point cloud point corresponding to the first point cloud set is greater than the ordinate of each second point cloud point contained in the first point cloud set; if so, determine the first point cloud point corresponding to the first point cloud set as the target point cloud point.

[0064] In this embodiment, for each first point cloud point in the point cloud data, the electronic device iterates through the spatial distances between the first point cloud point and other point cloud points, and determines each second point cloud point whose spatial distance from the first point cloud point is less than a first preset threshold. Each second point cloud point is then defined as the first point cloud point set corresponding to the first point cloud point. In this embodiment, the first preset threshold is a height threshold, typically measured in centimeters.

[0065] In this application example, the electronic device may identify multiple first point cloud points. The electronic device will then further filter the first point cloud points to determine the target point cloud point that may correspond to the high point of the hand.

[0066] Specifically, in this embodiment, for each first point cloud set, the electronic device determines whether the ordinate of the first point cloud point corresponding to the first point cloud set is greater than the ordinate of each second point cloud point included in the first point cloud set. If so, the electronic device determines that the first point cloud point corresponding to the first point cloud set is the target point cloud point corresponding to the high point of the hand.

[0067] It should be noted that, in the embodiments of this application, the ordinates of all point cloud points within a sphere centered on the target point cloud point and with a first preset threshold as the radius are not higher than the target point cloud point.

[0068] Figure 3 This is a schematic diagram illustrating the process of determining target point cloud points provided in an embodiment of this application. Figure 3 As shown, the process includes:

[0069] S301: For each first point cloud point in the point cloud data, determine each second point cloud point whose spatial distance from the first point cloud point is less than a first preset threshold, and determine each second point cloud point as the first point cloud point set corresponding to the first point cloud point.

[0070] S302: For each first point cloud set, determine whether the ordinate of the first point cloud point corresponding to the first point cloud set is greater than the ordinate of each second point cloud point contained in the first point cloud set; if so, determine the first point cloud point corresponding to the first point cloud set as the target point cloud point.

[0071] In order to perform hand motion recognition based on point cloud data and avoid privacy exposure, on the basis of the above embodiments, in the embodiments of the present application, the step of determining at least one target other point cloud point corresponding to other positions of the hand according to the target position information of the target point cloud point and other position information of other point cloud points except the target point cloud point includes:

[0072] determining a first difference value of the abscissa between the target point cloud point and each of the other point cloud points, and determining a second difference value of the vertical coordinate between the target point cloud point and each of the other point cloud points;

[0073] other point cloud points whose first difference value is less than a second preset threshold and whose second difference value is less than a third preset threshold are determined as the target other point cloud points.

[0074] In the embodiments of the present application, after the electronic device determines the target point cloud point corresponding to the high point of the hand, the electronic device will determine the target other point cloud points corresponding to other positions of the hand based on the target point cloud point.

[0075] Specifically, in the embodiments of the present application, the electronic device determines each other point cloud point that is within the second preset threshold range below the target point cloud point and within the third preset threshold range in the front-rear distance, and determines each of the point cloud points as the target other point cloud points corresponding to other positions of the hand.

[0076] Specifically, in the embodiments of the present application, the electronic device determines the first difference value of the abscissa between the target point cloud point and each of the other point cloud points, and determines the second difference value of the vertical coordinate between the target point cloud point and each of the other point cloud points. The electronic device determines other point cloud points whose first difference value is less than the second preset threshold and whose second difference value is less than the third preset threshold as the target other point cloud points.

[0077] Wherein, in the embodiments of the present application, both the first difference value and the second difference value are positive numbers.

[0078] Wherein, the target other point cloud points may be determined by the following formula: if the abscissa of the other point cloud point is x i and the vertical coordinate is z i , then Δx is determined i =|x0-x i |, Δz i =|z0-z i |, if Δx i <t2, and Δz i <t3, this other point cloud point is determined as a target point cloud point. Wherein, Δx i is the first difference value, Δz it1 is the second difference, t2 is the second preset threshold, t3 is the third preset threshold, x0 is the abscissa of the target point cloud, and z0 is the ordinate of the target point cloud.

[0079] Figure 4 This is a schematic diagram illustrating the process of determining other point cloud points of a target, as provided in an embodiment of this application. Figure 4 As shown, the process includes:

[0080] S401: Determine the first difference between the x-coordinate of the target point cloud point and the x-coordinate of each other point cloud point, and determine the second difference between the vertical coordinate of the target point cloud point and the x-coordinate of each other point cloud point.

[0081] S402: Other point cloud points whose first difference is less than the second preset threshold and whose second difference is less than the third preset threshold are identified as target other point cloud points.

[0082] To achieve hand gesture recognition, based on the above embodiments, the method in this application embodiment further includes:

[0083] If the second preset threshold is a preset fist height and the third preset threshold is a preset palm thickness, then the hand posture is determined to be a clenched fist.

[0084] If the second preset threshold is a preset palm length and the third preset threshold is a preset palm thickness, then the hand posture is determined to be a palm-out state.

[0085] In the embodiments of this application, the electronic device can recognize hands in different postures, such as clenched fists, open palms, etc.

[0086] Specifically, in this embodiment of the application, if the electronic device uses a preset threshold of a preset fist height and a preset threshold of a preset palm thickness when determining other point cloud points of the target, the electronic device determines that the identified hand posture is a clenched fist state; if the electronic device uses a preset threshold of a preset palm length and a preset threshold of a preset palm thickness when determining other point cloud points of the target, the electronic device determines that the identified hand posture is a palm-out state.

[0087] In order to recognize changes in hand movements, based on the above embodiments, the method in this application embodiment further includes:

[0088] Obtain other positional information of the hand that was previously determined;

[0089] The target movement path of the hand is determined based on the other location information and the location information.

[0090] In order to recognize changes in hand movements, based on the above embodiments, the method in this application embodiment further includes:

[0091] Obtain other hand postures previously determined;

[0092] The target posture change of the hand is determined based on the other postures and the posture itself.

[0093] In order to recognize changes in hand movements, based on the above embodiments, the method in this application embodiment further includes:

[0094] Based on the target movement path, the target attitude change, and the pre-saved correspondence between movement path, attitude change, and control command, determine the target control command corresponding to the target movement path and the target attitude change;

[0095] Execute the target control command.

[0096] In this embodiment of the application, after determining the position information and posture of the hand, the electronic device will also acquire other position information and other postures of the hand that were previously determined, and analyze the movement path of the hand based on the position information and other position information, and analyze the changes in the posture of the hand based on the posture and other postures.

[0097] Specifically, in this embodiment, the electronic device determines the target movement path of the hand based on the other location information and the location information; determines the target posture change of the hand based on the other posture and the posture; determines the target control command corresponding to the target movement path and the target posture change based on the target movement path, the target posture change and the pre-saved correspondence between movement path, posture change and control command; and executes the target control command.

[0098] In this embodiment of the application, the target movement path includes the movement of the hand in the forward and backward, left and right, and up and down directions, as well as the movement distance.

[0099] Furthermore, based on different hand postures and target movement paths of different directions and distances, there are several possible combinations. Depending on the specific scenario, different actions and durations can be set to achieve intelligent gesture control. For example: 1. Extend the palm flat for n seconds, then clench the fist for m seconds; 2. Move the palm flat to the upper right a distance L; 3. Extend the palm flat, then clench the fist and move it downwards a distance L, then extend the palm again. All these different combinations can correspond to different gesture control commands.

[0100] Figure 5 This is a flowchart illustrating control based on hand movements, as provided in an embodiment of this application. Figure 5As shown, the process includes:

[0101] S501: Obtain other positional information and other postures of the previously determined hand.

[0102] S502: Determine the target movement path of the hand based on other location information and location information.

[0103] S503: Determine the target posture change of the hand based on other postures and poses.

[0104] S504: Based on the target movement path, target attitude changes, and the pre-saved correspondence between movement paths, attitude changes, and control commands, determine the target control commands corresponding to the target movement path and target attitude changes.

[0105] S505: Execute target control instructions.

[0106] To perform hand motion recognition based on point cloud data and avoid privacy exposure, in this embodiment of the application, based on the above embodiments, the step of determining the position information of the hand according to the spatial coordinates of each point cloud point in the target point cloud set includes:

[0107] Obtain the x-coordinate of each point in the target point cloud set, determine the second mean of each x-coordinate, and use the second mean as the target x-coordinate of the hand;

[0108] Obtain each ordinate of each point in the target point cloud set, determine the third mean of each ordinate, and use the third mean as the target ordinate of the hand;

[0109] Obtain each vertical coordinate of each point in the target point cloud set, determine the fourth mean of each vertical coordinate, and determine the fourth mean as the target vertical coordinate of the hand;

[0110] The target's horizontal coordinate, vertical coordinate, and vertical coordinate are determined as the position information of the hand.

[0111] In this embodiment of the application, when the electronic device determines the position information of the hand based on the spatial coordinates of each point in the target point cloud set, in order to facilitate the subsequent analysis of the hand's movement path, the electronic device will use a spatial coordinate to represent the position information of the hand.

[0112] Specifically, in this embodiment, the electronic device acquires each abscissa of each point in the target point cloud set, determines a second mean of each abscissa, and uses the second mean as the target abscissa of the hand; the electronic device acquires each ordinate of each point in the target point cloud set, determines a third mean of each ordinate, and uses the third mean as the target ordinate of the hand; the electronic device acquires each perpendicular coordinate of each point in the target point cloud set, determines a fourth mean of each perpendicular coordinate, and uses the fourth mean as the target perpendicular coordinate of the hand.

[0113] The electronic device determines the target's horizontal coordinate, vertical coordinate, and vertical coordinate as the hand's position information.

[0114] In this embodiment of the application, the electronic device can determine the position information of the hand using the following formula:

[0115] x avr = (x1+x2+……+x) n ) / n;

[0116] y avr = (y1+y2+……+y n ) / n;

[0117] z avr = (z1+z2+……+z n ) / n;

[0118] Where x1, x2, ..., x n Let y1, y2, ..., y3 be the x-coordinates of each point in the target point cloud set. n Let z1, z2, ..., z be the ordinates of each point in the target point cloud set. n For each vertical coordinate of each point in the target point cloud set, x avr Let y be the x-coordinate of the target. avr Let z be the target ordinate. avr If the vertical coordinate of the target is given, then the position information of the hand is determined as (x...). avr y avr , z avr ).

[0119] Figure 6 This is a flowchart illustrating the process of determining hand position information provided in an embodiment of this application. Figure 6 As shown, the process includes:

[0120] S601: Obtain the x-coordinate of each point in the target point cloud set, determine the second mean of each x-coordinate, and set the second mean as the target x-coordinate of the hand.

[0121] S602: Obtain the ordinate of each point in the target point cloud set, determine the third mean of each ordinate, and set the third mean as the target ordinate of the hand.

[0122] S603: Obtain each vertical coordinate of each point in the target point cloud set, determine the fourth mean of each vertical coordinate, and determine the fourth mean as the target vertical coordinate of the hand.

[0123] S604: Determine the target's horizontal coordinate, vertical coordinate, and vertical coordinate as the hand's position information.

[0124] Figure 7 This application provides a schematic diagram of a hand motion recognition device, which includes:

[0125] The acquisition module 701 is used to acquire point cloud data collected by the lidar sensor and infrared images acquired by the infrared sensor.

[0126] The processing module 702 is used to obtain a candidate point cloud set corresponding to the hand in the point cloud data; obtain the first mean value of each infrared value corresponding to the candidate point cloud set in the infrared image; if the first mean value is greater than a preset infrared value threshold, then determine that the candidate point cloud set is the target point cloud set corresponding to the hand.

[0127] The determination module 703 is used to determine the position information of the hand based on the spatial coordinates of each point in the target point cloud set.

[0128] In one possible implementation, the processing module 702 is specifically configured to: determine the target point cloud point corresponding to the high point of the hand based on the spatial distance between every two point cloud points in the point cloud point set and the spatial coordinates of each point cloud point; determine at least one other target point cloud point corresponding to other positions of the hand based on the target position information of the target point cloud point and other position information of other point cloud points besides the target point cloud point; and determine the target point cloud point and the at least one other target point cloud point as the candidate point cloud point set.

[0129] In one possible implementation, the processing module 702 is specifically configured to, for each first point cloud point in the point cloud data, determine each second point cloud point whose spatial distance from the first point cloud point is less than a first preset threshold, and determine each second point cloud point as a first point cloud point set corresponding to the first point cloud point; for each first point cloud point set, determine whether the ordinate of the first point cloud point corresponding to the first point cloud point set is greater than the ordinate of each second point cloud point included in the first point cloud point set; if so, determine the first point cloud point corresponding to the first point cloud point set as the target point cloud point.

[0130] In one possible implementation, the processing module 702 is specifically used to determine a first difference in the horizontal coordinate between the target point cloud point and each other point cloud point, and to determine a second difference in the vertical coordinate between the target point cloud point and each other point cloud point; and to determine other point cloud points whose first difference is less than a second preset threshold and whose second difference is less than a third preset threshold as the target other point cloud points.

[0131] In one possible implementation, the determining module 703 is further configured to determine the hand posture as a clenched fist if the second preset threshold is a preset fist height and the third preset threshold is a preset palm thickness; and to determine the hand posture as a flat palm if the second preset threshold is a preset palm length and the third preset threshold is a preset palm thickness.

[0132] In one possible implementation, the acquisition module 701 is further configured to acquire other positional information of the hand that was previously determined;

[0133] The determining module 703 is further configured to determine the target movement path of the hand based on the other location information and the location information.

[0134] In one possible implementation, the acquisition module 701 is further configured to acquire other previously determined hand postures;

[0135] The determining module 703 is further configured to determine the target posture change of the hand based on the other postures and the posture.

[0136] In one possible implementation, the determining module 703 is further configured to determine the target control command corresponding to the target movement path and the target attitude change based on the target movement path, the target attitude change, and the pre-saved correspondence between movement path, attitude change, and control command; and execute the target control command.

[0137] In one possible implementation, the determining module 703 is specifically configured to: acquire each abscissa of each point in the target point cloud set; determine a second mean of each abscissa; and determine the second mean as the target abscissa of the hand; acquire each ordinate of each point in the target point cloud set; determine a third mean of each ordinate; and determine the third mean as the target ordinate of the hand; acquire each perpendicular coordinate of each point in the target point cloud set; determine a fourth mean of each perpendicular coordinate; and determine the fourth mean as the target perpendicular coordinate of the hand; and determine the target abscissa, the target ordinate, and the target perpendicular coordinate as the position information of the hand.

[0138] Based on the above embodiments, this application also provides an electronic device. Figure 8 This application provides a schematic diagram of an electronic device structure, such as... Figure 8 As shown, it includes: processor 801, communication interface 802, memory 803 and communication bus 804, wherein processor 801, communication interface 802 and memory 803 communicate with each other through communication bus 804.

[0139] The memory 803 stores a computer program. When the program is executed by the processor 801, the processor 801 performs the following steps:

[0140] Acquire point cloud data collected by the lidar sensor and infrared images acquired by the infrared sensor;

[0141] Obtain the candidate point cloud set corresponding to the hand in the point cloud data;

[0142] Obtain the first mean value of each infrared value corresponding to the candidate point cloud set in the infrared image. If the first mean value is greater than a preset infrared value threshold, then determine that the candidate point cloud set is the target point cloud set corresponding to the hand.

[0143] The position information of the hand is determined based on the spatial coordinates of each point in the target point cloud set.

[0144] In one possible implementation, the processor is further configured to:

[0145] Based on the spatial distance between every two point cloud points in the point cloud set and the spatial coordinates of each point cloud point, determine the target point cloud point corresponding to the high point of the hand;

[0146] Based on the target location information of the target point cloud and the other location information of other point cloud points besides the target point cloud, at least one other target point cloud point corresponding to the other position of the hand is determined.

[0147] The target point cloud points and at least one other target point cloud points are determined as the candidate point cloud point set.

[0148] In one possible implementation, the processor is further configured to:

[0149] For each first point cloud point in the point cloud data, determine each second point cloud point whose spatial distance from the first point cloud point is less than a first preset threshold, and determine each second point cloud point as the first point cloud point set corresponding to the first point cloud point;

[0150] For each first point cloud set, determine whether the ordinate of the first point cloud point corresponding to the first point cloud set is greater than the ordinate of each second point cloud point contained in the first point cloud set; if so, determine the first point cloud point corresponding to the first point cloud set as the target point cloud point.

[0151] In one possible implementation, the processor is further configured to:

[0152] Determine a first difference between the x-coordinate of the target point cloud point and each other point cloud point, and determine a second difference between the vertical coordinate of the target point cloud point and each other point cloud point;

[0153] Other point cloud points whose first difference is less than a second preset threshold and whose second difference is less than a third preset threshold are identified as the target other point cloud points.

[0154] In one possible implementation, the processor is further configured to:

[0155] If the second preset threshold is a preset fist height and the third preset threshold is a preset palm thickness, then the hand posture is determined to be a clenched fist.

[0156] If the second preset threshold is a preset palm length and the third preset threshold is a preset palm thickness, then the hand posture is determined to be a palm-out state.

[0157] In one possible implementation, the processor is further configured to:

[0158] Obtain other positional information of the hand that was previously determined;

[0159] The target movement path of the hand is determined based on the other location information and the location information.

[0160] In one possible implementation, the processor is further configured to:

[0161] Obtain other hand postures previously determined;

[0162] The target posture change of the hand is determined based on the other postures and the posture itself.

[0163] In one possible implementation, the processor is further configured to:

[0164] Based on the target movement path, the target attitude change, and the pre-saved correspondence between movement path, attitude change, and control command, determine the target control command corresponding to the target movement path and the target attitude change;

[0165] Execute the target control command.

[0166] In one possible implementation, the processor is further configured to:

[0167] Obtain the x-coordinate of each point in the target point cloud set, determine the second mean of each x-coordinate, and use the second mean as the target x-coordinate of the hand;

[0168] Obtain each ordinate of each point in the target point cloud set, determine the third mean of each ordinate, and use the third mean as the target ordinate of the hand;

[0169] Obtain each vertical coordinate of each point in the target point cloud set, determine the fourth mean of each vertical coordinate, and determine the fourth mean as the target vertical coordinate of the hand;

[0170] The target's horizontal coordinate, vertical coordinate, and vertical coordinate are determined as the position information of the hand.

[0171] Since the principle of the above-mentioned electronic device in solving the problem is similar to that of the hand motion recognition method, the implementation of the above-mentioned electronic device can be found in the embodiments of the method, and repeated details will not be repeated.

[0172] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 802 is used for communication between the aforementioned electronic device and other devices. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0173] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0174] Based on the above embodiments, this invention also provides a computer-readable storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to perform the following steps:

[0175] Acquire point cloud data collected by the lidar sensor and infrared images acquired by the infrared sensor;

[0176] Obtain the candidate point cloud set corresponding to the hand in the point cloud data;

[0177] Obtain the first mean value of each infrared value corresponding to the candidate point cloud set in the infrared image. If the first mean value is greater than a preset infrared value threshold, then determine that the candidate point cloud set is the target point cloud set corresponding to the hand.

[0178] The position information of the hand is determined based on the spatial coordinates of each point in the target point cloud set.

[0179] In one possible implementation, obtaining the candidate point cloud set corresponding to the hand in the point cloud data includes:

[0180] Based on the spatial distance between every two point cloud points in the point cloud set and the spatial coordinates of each point cloud point, determine the target point cloud point corresponding to the high point of the hand;

[0181] Based on the target location information of the target point cloud and the other location information of other point cloud points besides the target point cloud, at least one other target point cloud point corresponding to the other position of the hand is determined.

[0182] The target point cloud points and at least one other target point cloud points are determined as the candidate point cloud point set.

[0183] In one possible implementation, determining the target point cloud corresponding to the high point of the hand based on the spatial distance between every two point cloud points in the point cloud set and the spatial coordinates of each point cloud point includes:

[0184] For each first point cloud point in the point cloud data, determine each second point cloud point whose spatial distance from the first point cloud point is less than a first preset threshold, and determine each second point cloud point as the first point cloud point set corresponding to the first point cloud point;

[0185] For each first point cloud set, determine whether the ordinate of the first point cloud point corresponding to the first point cloud set is greater than the ordinate of each second point cloud point contained in the first point cloud set; if so, determine the first point cloud point corresponding to the first point cloud set as the target point cloud point.

[0186] In one possible implementation, determining at least one other target point cloud point corresponding to another position of the hand based on the target position information of the target point cloud point and other position information of other point cloud points besides the target point cloud point includes:

[0187] Determine a first difference between the x-coordinate of the target point cloud point and each other point cloud point, and determine a second difference between the vertical coordinate of the target point cloud point and each other point cloud point;

[0188] Other point cloud points whose first difference is less than a second preset threshold and whose second difference is less than a third preset threshold are identified as the target other point cloud points.

[0189] In one possible implementation, the method further includes:

[0190] If the second preset threshold is a preset fist height and the third preset threshold is a preset palm thickness, then the hand posture is determined to be a clenched fist.

[0191] If the second preset threshold is a preset palm length and the third preset threshold is a preset palm thickness, then the hand posture is determined to be a palm-out state.

[0192] In one possible implementation, the method further includes:

[0193] Obtain other positional information of the hand that was previously determined;

[0194] The target movement path of the hand is determined based on the other location information and the location information.

[0195] In one possible implementation, the method further includes:

[0196] Obtain other hand postures previously determined;

[0197] The target posture change of the hand is determined based on the other postures and the posture itself.

[0198] In one possible implementation, the method further includes:

[0199] Based on the target movement path, the target attitude change, and the pre-saved correspondence between movement path, attitude change, and control command, determine the target control command corresponding to the target movement path and the target attitude change;

[0200] Execute the target control command.

[0201] In one possible implementation, determining the position information of the hand based on the spatial coordinates of each point in the target point cloud set includes:

[0202] Obtain the x-coordinate of each point in the target point cloud set, determine the second mean of each x-coordinate, and use the second mean as the target x-coordinate of the hand;

[0203] Obtain each ordinate of each point in the target point cloud set, determine the third mean of each ordinate, and use the third mean as the target ordinate of the hand;

[0204] Obtain each vertical coordinate of each point in the target point cloud set, determine the fourth mean of each vertical coordinate, and determine the fourth mean as the target vertical coordinate of the hand;

[0205] The target's horizontal coordinate, vertical coordinate, and vertical coordinate are determined as the position information of the hand.

[0206] Since the principle of the computer-readable storage medium in solving the problem is similar to that of the hand gesture recognition method, the implementation of the computer-readable storage medium can be found in the embodiments of the method, and repeated details will not be repeated.

[0207] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0209] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0210] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0211] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A hand gesture recognition method, characterized in that, The method includes: Acquire point cloud data collected by the lidar sensor and infrared images acquired by the infrared sensor; Obtain the candidate point cloud set corresponding to the hand in the point cloud data; Obtain the first mean value of each infrared value corresponding to the candidate point cloud set in the infrared image. If the first mean value is greater than a preset infrared value threshold, then determine that the candidate point cloud set is the target point cloud set corresponding to the hand. The position information of the hand is determined based on the spatial coordinates of each point in the target point cloud set; The step of obtaining the candidate point cloud set corresponding to the hand in the point cloud data includes: Based on the spatial distance between every two point cloud points and the spatial coordinates of each point cloud point in the point cloud data, the target point cloud point corresponding to the high point of the hand is determined; Based on the target location information of the target point cloud and the other location information of other point cloud points besides the target point cloud, at least one other target point cloud point corresponding to the other position of the hand is determined. The target point cloud points and at least one other target point cloud points are determined as the candidate point cloud point set; The step of determining the target point corresponding to the high point of the hand based on the spatial distance between every two point cloud points and the spatial coordinates of each point cloud point includes: For each first point cloud point in the point cloud data, determine each second point cloud point whose spatial distance from the first point cloud point is less than a first preset threshold, and determine each second point cloud point as the first point cloud point set corresponding to the first point cloud point; For each first point cloud set, determine whether the ordinate of the first point cloud point corresponding to the first point cloud set is greater than the ordinate of each second point cloud point contained in the first point cloud set; if so, determine the first point cloud point corresponding to the first point cloud set as the target point cloud point. The step of determining at least one other target point cloud point corresponding to other positions of the hand based on the target position information of the target point cloud point and the other position information of other point cloud points besides the target point cloud point includes: Determine a first difference between the x-coordinate of the target point cloud point and each other point cloud point, and determine a second difference between the vertical coordinate of the target point cloud point and each other point cloud point; Other point cloud points whose first difference is less than the second preset threshold and whose second difference is less than the third preset threshold are determined as the target other point cloud points; If the second preset threshold is a preset fist height and the third preset threshold is a preset palm thickness, then the hand posture is determined to be a clenched fist. If the second preset threshold is a preset palm length and the third preset threshold is a preset palm thickness, then the hand posture is determined to be a palm-out state.

2. The method according to claim 1, characterized in that, Determining the position information of the hand based on the spatial coordinates of each point in the target point cloud set includes: Obtain the x-coordinate of each point in the target point cloud set, determine the second mean of each x-coordinate, and use the second mean as the target x-coordinate of the hand; Obtain each ordinate of each point in the target point cloud set, determine the third mean of each ordinate, and use the third mean as the target ordinate of the hand; Obtain each vertical coordinate of each point in the target point cloud set, determine the fourth mean of each vertical coordinate, and determine the fourth mean as the target vertical coordinate of the hand; The target's horizontal coordinate, vertical coordinate, and vertical coordinate are determined as the position information of the hand.

3. An electronic device, characterized in that, The electronic device includes a processor that executes a computer program stored in a memory to implement the steps of the hand movement recognition method as described in any one of claims 1-2.

Images