Gesture recognition method, electronic device, gesture recognition system, and storage medium

Abstract

The application relates to a gesture recognition method, an electronic device, a gesture recognition system and a storage medium. According to three-dimensional posture information of a target object, a projection plane of at least one side of the trunk on the left and right sides of the target object is constructed, and a pointing vector of at least one hand of the left and right hands is determined. A spatial position relationship between the pointing vector of a first hand and the projection plane of a first trunk is determined, the first hand being the hand closest to the first trunk among the left and right hands. Whether the first hand is in a pointing state is determined according to the spatial position relationship, the problem that left and right hand pointing gestures cannot be accurately distinguished in the related art is solved, and left and right hand pointing gestures are independently and in parallel recognized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human three-dimensional pose detection, and in particular, to a gesture recognition method, an electronic device, a gesture recognition system, and a storage medium. BACKGROUND

[0002] Pointing gesture, as a high-frequency action language in people's communication and interaction, provides an efficient expression method for the orientation, place, and object designation in daily life. Existing robots need engineers to pre-plan the designated task target through programs when operating some objects, navigating the path of a place, and avoiding some directions. With the rapid development of service robots, the interaction mode between people and robots presents a multi-modal trend. People can quickly, conveniently, and naturally tell robots the designation target of the current task through pointing gestures. In recent years, with the rapid development of computer vision, deep learning, and RGB-D stereo camera technology, it has become possible for robots to completely capture the three-dimensional pose of the human body from the first perspective. However, human pointing gestures have problems such as left and right hands in parallel, a large range of pointing angles, and complex action backgrounds. Traditional gesture recognition methods cannot stably recognize left and right hand pointing and are prone to recognition errors.

[0003] To address the issue of being unable to accurately distinguish between left and right hand pointing gestures in related technologies, no effective solutions have been proposed. SUMMARY

[0004] A gesture recognition method, an electronic device, a gesture recognition system, and a storage medium are provided in the present embodiment to address the issue of being unable to accurately distinguish between left and right hand pointing gestures in related technologies.

[0005] In a first aspect, a gesture recognition method is provided in the present embodiment, comprising:

[0006] constructing a projection plane of at least one of the left and right sides of the torso of the target object according to the three-dimensional pose information of the target object, and determining a pointing vector of at least one of the left and right hands;

[0007] determining a spatial position relationship between the pointing vector of the first hand and the projection plane of the first torso, the first hand being the hand closest to the first torso among the left and right hands;

[0008] determining whether the first hand is in a pointing state according to the spatial position relationship.

[0009] In some embodiments, determining whether the first hand is in a pointing state according to the spatial position relationship comprises:

[0010] if the pointing vector of the first hand is outside the projection plane of the first torso, determining that the first hand is in the pointing state; or

[0011] if the pointing vector of the first hand is inside the projection plane of the first torso, and an included angle between the pointing vector of the first hand and a base vector inside the projection plane of the first torso satisfies a first constraint rule, determining that the first hand is in the pointing state.

[0012] In some embodiments, the method further comprises: if an included angle between the pointing vector of the first hand and a normal vector of the projection plane of the first torso satisfies a second constraint rule, determining that the pointing vector of the first hand is outside the projection plane of the first torso; or

[0013] if the included angle between the pointing vector of the first hand and the normal vector of the projection plane of the first torso satisfies a third constraint rule, determining that the pointing vector of the first hand is inside the projection plane of the first torso.

[0014] In some embodiments, before determining the spatial positional relationship between the pointing vector of the first hand and the projection plane of the first torso, and determining whether the first hand is in the pointing state according to the spatial positional relationship, the method further comprises:

[0015] determining an included angle between a large arm vector and a small arm vector close to the first torso according to the three-dimensional posture information of the target object;

[0016] if the included angle between the large arm vector and the small arm vector satisfies a fourth constraint rule, determining the spatial positional relationship between the pointing vector of the first hand and the projection plane of the first torso, and determining whether the first hand is in the pointing state according to the spatial positional relationship.

[0017] In some embodiments, the method further comprises: if the included angle between the large arm vector and the small arm vector does not satisfy the fourth constraint rule, determining that the first hand is not in the pointing state.

[0018] In some embodiments, the three-dimensional posture information of the target object comprises three-dimensional coordinates of a key point close to a shoulder joint of the first torso, three-dimensional coordinates of a key point close to an elbow joint, and three-dimensional coordinates of a key point close to a hand joint, the large arm vector of the first hand is determined according to the three-dimensional coordinates of the key point close to the shoulder joint and the three-dimensional coordinates of the key point close to the elbow joint, and the small arm vector of the first hand is determined according to the three-dimensional coordinates of the key point close to the shoulder joint and the three-dimensional coordinates of the key point close to the hand joint.

[0019] In some embodiments, the method further comprises: obtaining at least one angle threshold, wherein the angle threshold is obtained by collecting human pointing action statistics;

[0020] determining a corresponding constraint rule according to the angle threshold, wherein the constraint rule comprises: determining a first constraint rule according to a vertical angle threshold; or determining a second constraint rule according to a plane angle threshold; or determining a third constraint rule according to a plane angle threshold; or determining a fourth constraint rule according to a straight line angle threshold.

[0021] In some embodiments, the three-dimensional pose information of the target object comprises a key point three-dimensional coordinate of a left shoulder joint, a key point three-dimensional coordinate of a right shoulder joint, and a key point three-dimensional coordinate of a hip joint close to the first hand, and a projection plane of the first torso is constructed according to the key point three-dimensional coordinate of the left shoulder joint, the key point three-dimensional coordinate of the right shoulder joint, and the key point three-dimensional coordinate of the hip joint; or,

[0022] The three-dimensional pose information of the target object comprises a key point three-dimensional coordinate of a left shoulder joint, a key point three-dimensional coordinate of a right shoulder joint, a key point three-dimensional coordinate of a left hip joint, and a key point three-dimensional coordinate of a right hip joint, and a total projection plane is constructed according to the key point three-dimensional coordinate of the left shoulder joint, the key point three-dimensional coordinate of the right shoulder joint, the key point three-dimensional coordinate of the left hip joint, and the key point three-dimensional coordinate of the right hip joint, and one half of the total projection plane is divided into a projection plane of the first torso.

[0023] In some embodiments, the three-dimensional pose information of the target object comprises a key point three-dimensional coordinate of a shoulder joint close to the first hand, a key point three-dimensional coordinate of an elbow joint, and a key point three-dimensional coordinate of a hand joint, and a pointing vector of the first hand is determined according to the key point three-dimensional coordinate of the shoulder joint, the key point three-dimensional coordinate of the elbow joint, and the key point three-dimensional coordinate of the hand joint; or,

[0024] The three-dimensional pose information of the target object comprises a key point three-dimensional coordinate of a shoulder joint close to the first hand and a key point three-dimensional coordinate of a hand joint, and a pointing vector of the first hand is determined according to the key point three-dimensional coordinate of the shoulder joint and the key point three-dimensional coordinate of the hand joint; or,

[0025] The three-dimensional pose information of the target object comprises a key point three-dimensional coordinate of an elbow joint close to the first hand and a key point three-dimensional coordinate of a hand joint, and a pointing vector of the first hand is determined according to the key point three-dimensional coordinate of the elbow joint and the key point three-dimensional coordinate of the hand joint.

[0026] In some embodiments, before constructing the projection plane of at least one of the left and right sides of the target object according to the three-dimensional posture information of the target object, the method further comprises:

[0027] obtaining a visible light image and a depth image containing an image of the target object;

[0028] obtaining two-dimensional posture information of the target object according to the visible light image;

[0029] determining a depth value corresponding to the two-dimensional posture of each object according to the depth image, and obtaining three-dimensional posture information of the target object according to the two-dimensional posture information of the target object and the depth value.

[0030] In a second aspect, an electronic device is provided in the present embodiment, comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to execute the gesture recognition method of the first aspect.

[0031] In a third aspect, a gesture recognition system is provided in the present embodiment, comprising an image acquisition device and the electronic device of the second aspect, the image acquisition device being connected to the electronic device, and the image acquisition device being capable of acquiring a visible light image and a depth image.

[0032] In a fourth aspect, a computer readable storage medium is provided in the present embodiment, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the gesture recognition method of the first aspect.

[0033] Compared with the related art, the gesture recognition method, the electronic device, the gesture recognition system and the storage medium provided in the present embodiment solve the problem that the left and right hand pointing gestures cannot be accurately distinguished in the related art by constructing the projection plane of at least one of the left and right sides of the target object according to the three-dimensional posture information of the target object, and determining the pointing vector of at least one of the left and right hands; determining the spatial position relationship between the pointing vector of the first hand and the projection plane of the first trunk, the first hand being the hand closest to the first trunk among the left and right hands; and determining whether the first hand is in a pointing state according to the spatial position relationship, thereby realizing independent and parallel recognition of the left and right hand pointing gestures.

[0034] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0036] Figure 1 is a hardware structure block diagram of a terminal of a gesture recognition method in an embodiment of the application;

[0037] Figure 2 is a flow chart of a gesture recognition method in an embodiment of the application;

[0038] Figure 3 is a schematic diagram of a three-dimensional posture detection result of a target object in a first-person perspective in an embodiment of the application;

[0039] Figure 4 is a flow chart of another gesture recognition method in an embodiment of the application;

[0040] Figure 5 is a schematic diagram of multi-person human body color image and depth image acquisition in a first-person perspective in an embodiment of the application;

[0041] Figure 6 is a schematic diagram of a human body posture topology and each key joint thereof in an embodiment of the application;

[0042] Figure 7 is a schematic diagram of a recognition result of a gesture recognition method in an embodiment of the application. DETAILED DESCRIPTION

[0043] In order to more clearly understand the purpose, technical solutions and advantages of the application, the application is described and explained in detail below with reference to the drawings and embodiments.

[0044] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the same meaning as those commonly understood by a person of ordinary skill in the art to which the present application belongs. The terms "one", "a", "an", "the", "these", and similar terms in the present application do not mean "only one" or "exactly one", but can mean "one or more". The terms "include", "contain", "have", and any variant thereof in the present application are intended to cover the non-exclusive inclusion; for example, a process, method, system, product or device containing a series of steps or modules (units) is not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. The terms "connect", "connect", "couple" and the like in the present application are not limited to physical or mechanical connection, but can include electrical connection, whether direct or indirect. The term "multiple" in the present application means two or more. The term "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. Generally, the character " / " represents an "or" relationship between the associated objects. The terms "first", "second", "third" and the like in the present application are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0045] The method embodiments provided in the present embodiment can be executed in a terminal, a computer or a similar computing device. For example, the method embodiments are executed on a terminal, Figure 1 is a hardware structure diagram of a terminal of the gesture recognition method of an embodiment of the present application. As shown in Figure 1 , the terminal can include one or more (only one is shown in Figure 1 ) processor 102 and memory 104 for storing data, wherein the processor 102 can include but not limited to processing devices such as microprocessor MCU or programmable logic device FPGA. The above terminal can also include transmission device 106 for communication function and input / output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above terminal. For example, the terminal can include more or less components than those shown in Figure 1 , or have a different configuration from that shown in Figure 1 .

[0046] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the gesture recognition method in the embodiment. The processor 102 can execute various functional applications and data processing, i.e., implement the method described above, by running the computer program stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include memories remotely arranged with respect to the processor 102, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0047] The transmission device 106 is configured to receive or send data via a network. The network includes a wireless network provided by a communication provider of the terminal. In an example, the transmission device 106 includes a network interface controller (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In an example, the transmission device 106 can be a radio frequency (RF) module configured to communicate with the Internet in a wireless manner.

[0048] In an embodiment, as shown in Figure 2 , a flowchart of a gesture recognition method is provided, which is applied to the terminal in Figure 1 and includes the following steps:

[0049] In step S101, a projection plane of at least one of left and right sides of a torso of a target object is constructed according to three-dimensional posture information of the target object, and a pointing vector of at least one of left and right hands is determined.

[0050] The target object can be a real person, a mannequin, or a robot. Figure 3 A schematic diagram of a three-dimensional posture detection result of the target object in a first-person perspective is given, as shown in Figure 3 , in which black circles represent joint key points. The three-dimensional posture information of the target object refers to three-dimensional coordinates of the joint key points, including: three-dimensional coordinates of a left shoulder joint key point three-dimensional coordinates of a right shoulder joint key point three-dimensional coordinates of a left elbow joint key point three-dimensional coordinates of a right elbow joint key point three-dimensional coordinates of a left hand joint key point three-dimensional coordinates of a right hand joint key point three-dimensional coordinates of a left hip joint key point Key point three-dimensional coordinates of right hip joint

[0051] There are several ways to construct the projection plane of the left and right torso, and only way (1) is shown in the figure.

[0052] (1) The triangular region surrounded by the left shoulder joint key point, the right shoulder joint key point, and the left hip joint key point is taken as the projection plane of the left torso; the triangular region surrounded by the left shoulder joint key point, the right shoulder joint key point, and the right hip joint key point is taken as the projection plane of the right torso.

[0053] (2) According to the left shoulder joint key point, the right shoulder joint key point, the left hip joint key point, and the right hip joint key point, a total projection plane is constructed, and a symmetry axis is made between the left shoulder joint key point and the right shoulder joint key point to divide the total projection plane into left and right parts, i.e., the projection plane of the left torso and the projection plane of the right torso.

[0054] There are several ways to determine the pointing vector of the left and right hands, among which way (1) is the most accurate for recognizing the pointing gesture.

[0055] (1) The mean of the left upper arm vector and the left lower arm vector is taken as the pointing vector of the left hand; the mean of the right upper arm vector and the right lower arm vector is taken as the pointing vector of the right hand.

[0056] (2) According to the left shoulder joint key point and the left hand joint key point, the pointing vector of the left hand is determined; according to the right shoulder joint key point and the right hand joint key point, the pointing vector of the right hand is determined.

[0057] (3) According to the left elbow joint key point and the left hand joint key point, the pointing vector of the left hand is determined; according to the right elbow joint key point and the right hand joint key point, the pointing vector of the right hand is determined.

[0058] Step S102, determine the spatial position relationship between the pointing vector of the first hand and the projection plane of the first torso, the first hand being the hand closest to the first torso among the left and right hands.

[0059] The spatial position relationship between the pointing vector of the first hand and the projection plane of the first torso can be whether the pointing vector of the first hand is within the projection plane of the first torso, or the included angle between the pointing vector of the first hand and the projection plane of the first torso. If the first torso is the left torso, the first hand is the left hand; if the first torso is the right torso, the first hand is the right hand.

[0060] Step S103, determine whether the first hand is in a pointing state according to the spatial position relationship.

[0061] The association between the spatial position relationship and the pointing state can be established in advance, and whether the first hand is in the pointing state is determined according to the actually detected spatial position relationship.

[0062] In the above steps S101 to S103, the projection of the human torso is divided into two projection planes, the left and right, the left hand gesture is determined by the spatial position relationship between the pointing vector of the left hand and the projection plane of the left torso, and the right hand gesture is determined by the spatial position relationship between the pointing vector of the left hand and the projection plane of the right torso, so as to avoid interference between the left hand gesture recognition and the right hand gesture recognition, solve the problem that the left and right hand pointing gestures cannot be accurately distinguished in the related art, and realize independent and parallel recognition of the left and right hand pointing gestures.

[0063] In one embodiment, the above step S103 further includes: if the pointing vector of the first hand is outside the projection plane of the first torso, determining that the first hand is in the pointing state. If the pointing vector of the first hand is inside the projection plane of the first torso, and the included angle between the pointing vector of the first hand and the base vector inside the projection plane of the first torso satisfies the first constraint rule, determining that the first hand is in the pointing state.

[0064] Taking the projection plane of the right torso as an example, the base vector includes two direction vectors x and y, and the expression of the base vector is as follows:

[0065]

[0066]

[0067] wherein, represents the x-axis direction vector of the projection plane of the right torso, represents the three-dimensional coordinates of the right hip joint key point, represents the three-dimensional coordinates of the right shoulder joint key point, represents the y-axis direction vector of the projection plane of the right torso, represents the three-dimensional coordinates of the left shoulder joint key point, represents the three-dimensional coordinates of the right shoulder joint key point. If the pointing vector P R of the right hand is inside the projection plane of the right torso, the gesture of the right hand needs to form a certain included angle with the vertical direction of the torso of the target object, so as to be determined as being in the pointing state, that is, the included angle between the pointing vector P R of the right hand and the base vector inside the projection plane of the right torso satisfies the first constraint rule. The first constraint rule can be determined by a vertical angle threshold . If the pointing vector P R of the right hand is inside the projection plane of the right torso, and the included angle between the pointing vector P R of the right hand and the base vector satisfies the first constraint rule. If the angle between the pointing vector of the right hand and the normal vector of the projection plane of the right side torso satisfies the second constraint rule, it is determined that the pointing vector of the right hand is outside the projection plane of the right side torso. If the angle between the pointing vector of the right hand and the normal vector of the projection plane of the right side torso satisfies the third constraint rule, it is determined that the pointing vector of the right hand is inside the projection plane of the right side torso. In the embodiment, the two cases of the pointing vector being inside / outside the projection plane are classified and discussed by analyzing the two cases of the pointing vector being inside and outside the projection plane, and if the pointing vector is outside the projection plane, it is directly determined that the pointing gesture is in the pointing state; if the pointing vector is inside the projection plane, further determination is made in combination with the first constraint rule to avoid missing the case of the pointing vector being inside the projection plane and also being in the pointing state, thereby improving the detection rate of the pointing gesture recognition.

[0068] In one embodiment, a method for determining the spatial positional relationship between the pointing vector of the first hand and the projection plane of the first torso is given. If the angle between the pointing vector of the first hand and the normal vector of the projection plane of the first torso satisfies the second constraint rule, it is determined that the pointing vector of the first hand is outside the projection plane of the first torso. If the angle between the pointing vector of the first hand and the normal vector of the projection plane of the first torso satisfies the third constraint rule, it is determined that the pointing vector of the first hand is inside the projection plane of the first torso.

[0069] Continuing to refer to Figure 3 , taking the projection plane of the right side torso as an example, the normal vector N R perpendicular to the projection plane of the right side torso can be obtained from the basis vector of the projection plane of the right side torso, and the solving expression is as follows:

[0070]

[0071] The second constraint rule can be determined by a plane angle threshold . If the angle θ R between the pointing vector P R of the right hand and the normal vector N PR satisfies or , it is determined that the pointing vector of the right hand is outside the projection plane of the right side torso. The third constraint rule can also be determined by a plane angle threshold . If the angle θ PR between the pointing vector P R of the right hand and the normal vector N R satisfies , it is determined that the pointing vector of the right hand is inside the projection plane of the right side torso. In the embodiment, the second constraint rule and the third constraint rule are designed by using the plane angle threshold, making the determination of the two cases of the pointing vector being inside / outside the projection plane more flexible.

[0072] In one embodiment, before determining the spatial positional relationship between the pointing vector of the first hand and the projection plane of the first torso, and determining whether the first hand is in a pointing state based on the spatial positional relationship, the method further includes: determining the angle between the upper arm vector and the forearm vector near the first torso based on the three-dimensional pose information of the target object; if the angle between the upper arm vector and the forearm vector satisfies the fourth constraint rule, then the spatial positional relationship between the pointing vector of the first hand and the projection plane of the first torso is determined, and whether the first hand is in a pointing state is determined based on the spatial positional relationship. If the angle between the upper arm vector and the forearm vector does not satisfy the fourth constraint rule, then it is determined that the first hand is not in a pointing state.

[0073] Continue to refer to Figure 3 Taking the projection plane of the right torso as an example, to further improve the accuracy of pointing gesture recognition, the right hand gesture must satisfy the requirement that the upper arm and forearm are straight. Ideally, the right hand gesture should satisfy the requirement that the upper arm vector and forearm vector be straight and maintain a straight line. The fourth constraint rule can be based on the straight line angle threshold. Determined. If the angle θ between the upper arm vector and the lower arm vector... 01 satisfy If the right hand gesture basically satisfies the condition of the upper arm and forearm being fully extended, then the angle between the upper arm vector and the forearm vector satisfies the fourth constraint rule, and the subsequent steps continue. Otherwise, it is directly determined that the right hand is not in a pointing state. The target object's three-dimensional pose information includes the three-dimensional coordinates of key points near the right side of the torso: the shoulder joint, the elbow joint, and the hand joint. Based on the three-dimensional coordinates of the shoulder and elbow joints, the upper arm vector of the right hand is determined. Determine the forearm vector of the right hand based on the three-dimensional coordinates of the key points of the shoulder joint and the key points of the hand joint. Furthermore, the right-hand pointer points to the vector P. R It can be done through formula To obtain.

[0074] In this embodiment, before analyzing the spatial relationship between the pointing vector and the projection plane, a precondition is set: the upper arm vector and the forearm vector must be close to the same straight line. Then, the gesture recognition is performed by combining the spatial relationship between the pointing vector and the projection plane. This makes the gesture recognition method interpretable and improves the accuracy of pointing gesture recognition.

[0075] In one embodiment, a rule for recognizing pointing gestures is provided. First, the pointing gesture must satisfy the condition that the vector of the upper arm and the vector of the forearm are aligned in a straight line, i.e. When the forearm vector and the small arm vector keep a straight line, and the pointing vector is outside the corresponding projection plane, it can be directly determined that the pointing gesture is true, and the pointing mark is returned as 1. When the forearm vector and the small arm vector keep a straight line, and the pointing vector is inside the corresponding projection plane, the pointing gesture needs to form a certain angle with the vertical direction of the body trunk, for example, the pointing vector P of the right hand R and the base vector forms an angle At this time, it can be determined that the pointing gesture of the right hand is true, and the pointing mark is returned as 1. In other cases, it is determined that the pointing gesture is false, and the pointing mark is returned as 0.

[0076] In one embodiment, the method further comprises: obtaining at least one angle threshold, wherein the angle threshold is obtained by collecting human pointing actions; and determining the corresponding constraint rule according to the angle threshold. For example, determining the first constraint rule according to the vertical angle threshold; or determining the second constraint rule according to the plane angle threshold; or determining the third constraint rule according to the plane angle threshold; or determining the fourth constraint rule according to the straight line angle threshold.

[0077] Exemplarily, a pointing gesture library is collected and established, the angle thresholds of various key constraints are counted and summarized, and the recognition accuracy of the pointing gesture is improved. The pointing gesture library, that is, the set of human pointing actions to a long-distance target in a preset scene, records the three-dimensional posture of human pointing actions by a dynamic stereo capture system based on a reflective marker. The counted and summarized angle thresholds of various constraints include: a straight line angle threshold a plane angle threshold and a vertical angle threshold In this embodiment, the angle thresholds of various constraint rules can be adjusted, so that the gesture recognition method has flexibility, and the angle thresholds of the corresponding constraint rules can be obtained by collecting human pointing actions, which can further improve the recognition accuracy of the pointing gesture.

[0078] In one embodiment, before constructing the projection plane of at least one side of the trunk of the target object according to the three-dimensional posture information of the target object, the method further comprises: obtaining a visible light image and a depth image containing an image of the target object; obtaining two-dimensional posture information of the target object according to the visible light image; determining the depth value corresponding to the two-dimensional posture of each object according to the depth image; and obtaining the three-dimensional posture information of the target object according to the two-dimensional posture information and the depth value of the target object.

[0079] The visible light image can be a color image or a grayscale image. The visible light image can only include the target object or can simultaneously include one or more other objects in addition to the target object. By way of example, a camera is used to capture a multi-person color image and a depth image. A target detection network and a two-dimensional human pose estimation network are used to detect a two-dimensional human bounding box and a two-dimensional human pose from the multi-person color image. In a depth image coordinate system, a depth value corresponding to the two-dimensional human pose is read, and a three-dimensional human pose is calculated in combination with a camera intrinsic parameter.

[0080] In one embodiment, Figure 4 A flowchart of another gesture recognition method is provided, as shown in Figure 4 The flowchart includes the following steps:

[0081] In step S201, a multi-person color image and a depth image are acquired. Figure 5 A multi-person color image and a depth image acquisition schematic diagram in a first-person perspective is given, as shown in Figure 5 The color image and the depth image can be acquired by an RGB-D stereo camera hardware device based on infrared laser speckle coding, and the color image and the depth image are synchronized and aligned in time and space. The height from the ground and the shooting perspective of the camera can be adjusted in advance to better capture the complete multi-person pose.

[0082] In step S202, a target detection network and a two-dimensional human pose estimation network are used to detect a two-dimensional human bounding box and a two-dimensional human pose from the multi-person color image. Both the target detection network and the two-dimensional human pose estimation network use a lightweight real-time open source pre-trained model. The target detection network detects a human class confidence c and a two-dimensional human bounding box (u1, v1, u2, u2) from the color image. When the human class confidence c is greater than or equal to 0.7, (u1, v1, u2, v2) is cropped from the image as a local image. The two-dimensional human pose estimation network calculates two-dimensional human key points from the local image. Figure 6 A human pose topology and a schematic diagram of each key joint are given, as shown in Figure 6 The two-dimensional human key points are left-right mirror symmetric and specifically include 17 joints, such as a left shoulder, a right shoulder, a left elbow, a right elbow, a left hip, and a right hip, denoted as

[0083] In step S203, a depth value corresponding to the two-dimensional human pose is read in a depth image coordinate system to acquire three-dimensional pose information of the human. The depth image D(u, v) ∈ R H×W The depth value corresponding to the two-dimensional human pose is read in a depth image coordinate system to acquire three-dimensional pose information of the human. The depth image D(u, v) ∈ R The coordinates of key points on the human body along the x-axis and y-axis can be obtained from the formulas respectively. and The calculation yields f in the formula. x and f y These represent the focal lengths along the camera's x and y axes, respectively. A 3D human pose can be composed of the 3D coordinates of 17 key joints in the human body, including the shoulder, elbow, hip, and hand joints, in the camera coordinate system. Figure 6 The key points of the human body shown correspond one-to-one, and are denoted as follows:

[0084] Step S204: Based on the three-dimensional pose information of the human body, construct the projection planes of the left and right sides of the torso. For example... Figure 3 As shown, the left side projection plane of the human torso is a grid-shaped shaded plane, which is formed by the left shoulder joint. Right shoulder joint and left hip joint The three-dimensional human body is bounded by the coordinates of key points; the right projection plane of the human torso is a slanted shaded plane, formed by the right shoulder joint. left shoulder joint and right hip joint The coordinates of the key points of the three-dimensional human body are enclosed.

[0085] Step S205: Based on the human body's 3D pose information, determine local vectors, including determining the basis vectors and normal vectors of the projection plane, and determining the pointing vectors of the left and right hands. The local vectors on the left and right sides are mirror-symmetric, and the calculation process is similar, so they can be calculated independently. Figure 3 As shown, the basis vectors include two direction vectors, the x-axis and the y-axis. Taking the projection plane of the right side of the human torso as an example, the basis vectors can be expressed by the formula... and The normal vector N is obtained. R The normal vector perpendicular to the projection plane can be derived from the basis vectors of the projection plane. We obtain the pointing vector P. R It is the mean of the upper arm vector and the lower arm vector, where the upper arm vector and the lower arm vector are respectively... and and These are the right elbow and right hand joints, respectively, and their pointing vectors can be derived from the formula... To obtain.

[0086] Step S206 involves analyzing two cases: the pointing vector is within and outside the torso projection plane. Multi-angle constraint rules are designed to determine and recognize pointing gestures and return pointing markers. These include:

[0087] Step S2061, calculate the included angle θ PR θ PB θ01 where θ PR is the angle between the pointing vector P R and the normal vector N R , θ PB is the angle between the pointing vector P R and the base vector , and θ 01 is the angle between the large arm vector and the small arm vector . Specifically, each angle can be solved using the law of cosines, which is expressed as follows:

[0088]

[0089]

[0090]

[0091] Step S2062, it is determined whether the large arm and the small arm are in a straight line (i.e., it is determined whether is true); if yes, step S2063 is performed; otherwise, step S2066 is performed.

[0092] Step S2063, it is determined whether the pointing vector is in the projection plane (i.e., it is determined whether is true); if yes, step S2064 is performed; otherwise, step S2065 is performed.

[0093] Step S2064, it is determined whether the pointing vector and the vertical direction form a certain angle (i.e., it is determined whether is true); if yes, step S2065 is performed; otherwise, step S2066 is performed.

[0094] Step S2065, the pointing gesture is determined to be true, and the pointing flag is 1.

[0095] Step S2066, the pointing gesture is determined to be false, and the pointing flag is 0.

[0096] Step S207, a common pointing gesture library is collected and established, and angle thresholds of each key constraint are counted and summarized to improve the recognition accuracy of the pointing gesture. The pointing gesture library, i.e., a set of common human pointing gestures to a long-distance target in a preset scene, records the three-dimensional posture of the human pointing gesture by the dynamic stereo capture system based on the reflective marker. The counted and summarized angle thresholds of each constraint include: a straight line angle threshold a plane angle threshold and a vertical angle threshold Exemplarily, Figure 7An identification result schematic diagram of the gesture recognition method in the embodiment is given, and a pointing mark is displayed on the two-dimensional human body bounding box, representing whether the same side hand is pointing or not, 1 representing pointing and 0 representing not pointing. In some embodiments, the pointing action can also be recognized by calculating the local vector of the left and right side projection planes of the human body trunk, and the independent recognition of the left and right pointing actions of multiple human bodies can be realized, which can be applied to the analysis and positioning of the pointing target of the user by the service robot.

[0097] The gesture recognition method provided in the embodiment has good flexibility and interpretability, can realize independent and parallel recognition of actions and return marks for the left and right pointing actions of multiple human bodies, and avoids mutual interference of the left and right pointing actions. The pointing action judgment rule is clear and explicit, and is not prone to errors. The angle threshold of each constraint rule can be adjusted, and the pointing gesture recognition accuracy can be further improved by investigating the pointing action set of different application scenarios, while maintaining high efficiency and stability, and taking into account the flexibility and interpretability of pointing recognition. The method is suitable for dynamic communication of the pointing target of people to the robot in an open scene.

[0098] In one embodiment, an electronic device is provided, including a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0099] Optionally, the electronic device can further include a transmission device and an input and output device, wherein the transmission device is connected with the processor, and the input and output device is connected with the processor.

[0100] Optionally, in the embodiment, the processor can be configured to perform the following steps through the computer program:

[0101] S1, constructing a projection plane of at least one of the left and right trunks of a target object according to three-dimensional posture information of the target object, and determining a pointing vector of at least one of the left and right hands;

[0102] S2, determining a spatial position relationship between the pointing vector of the first hand and the projection plane of the first trunk, the first hand being the hand closest to the first trunk among the left and right hands;

[0103] S3, determining whether the first hand is in a pointing state according to the spatial position relationship.

[0104] It should be noted that the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, which will not be described herein again.

[0105] In one embodiment, a gesture recognition system is provided, comprising: an image acquisition device and the electronic device of the above embodiments, the image acquisition device being connected to the electronic device, the image acquisition device being capable of acquiring visible light images and depth images. The electronic device is used to implement the above embodiments and preferred embodiments, which have been described and will not be repeated. The image acquisition device has a built-in function module or program module, which can be implemented by software or hardware.

[0106] In addition, in combination with the gesture recognition method provided in the above embodiments, a storage medium can also be provided in this embodiment to implement. The storage medium has a computer program stored thereon; the computer program is executed by a processor to implement any one of the gesture recognition methods in the above embodiments.

[0107] It should be understood that the specific embodiments described herein are merely intended to explain the application, but not to limit it. According to the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0108] Obviously, the drawings are only some examples or embodiments of the present application, and can be applied to other similar situations without creative labor for those of ordinary skill in the art. In addition, it can be understood that although the work done in the development process may be complex and long, some design, manufacture or production changes according to the technical content disclosed in the present application are only routine technical means for those of ordinary skill in the art, and should not be regarded as insufficient disclosure of the present application.

[0109] The word "embodiment" in the present application means that the specific features, structures or characteristics described in combination with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean independence or alternative to other embodiments. It can be clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0110] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties. The acquisition, storage, use, processing and other aspects of data in the embodiments of the present application comply with the relevant provisions of national laws and regulations.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0112] The above-mentioned embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of patent protection. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A gesture recognition method, characterized by, The method comprises the following steps: According to the three-dimensional posture information of the target object, a projection plane of at least one of the left and right sides of the torso of the target object is constructed, and a pointing vector of at least one of the left and right hands is determined; The spatial position relationship between the pointing vector of the first hand and the projection plane of the first torso is determined, and the first hand is the hand closest to the first torso among the left and right hands; According to the three-dimensional posture information of the target object, the included angle between the large arm vector and the small arm vector close to the first torso is determined; If the included angle between the large arm vector and the small arm vector satisfies the fourth constraint rule, the spatial position relationship between the pointing vector of the first hand and the projection plane of the first torso is determined, and whether the first hand is in a pointing state is determined according to the spatial position relationship, which comprises: if the pointing vector of the first hand is outside the projection plane of the first torso, it is determined that the first hand is in a pointing state; Or, if the pointing vector of the first hand is inside the projection plane of the first torso, and the included angle between the pointing vector of the first hand and the base vector inside the projection plane of the first torso satisfies the first constraint rule, it is determined that the first hand is in a pointing state.

2. The gesture recognition method of claim 1, wherein, The method further comprises: If the included angle between the pointing vector of the first hand and the normal vector of the projection plane of the first torso satisfies the second constraint rule, it is determined that the pointing vector of the first hand is outside the projection plane of the first torso; or, If the included angle between the pointing vector of the first hand and the normal vector of the projection plane of the first torso satisfies the third constraint rule, it is determined that the pointing vector of the first hand is inside the projection plane of the first torso.

3. The gesture recognition method of claim 1, wherein, The method further comprises: If the included angle between the large arm vector and the small arm vector does not satisfy the fourth constraint rule, it is determined that the first hand is not in a pointing state.

4. The gesture recognition method of claim 1, wherein, The three-dimensional posture information of the target object comprises the three-dimensional coordinates of the key points of the shoulder joint, the elbow joint and the hand joint close to the first torso, the large arm vector of the first hand is determined according to the three-dimensional coordinates of the key points of the shoulder joint and the elbow joint, and the small arm vector of the first hand is determined according to the three-dimensional coordinates of the key points of the shoulder joint and the hand joint.

5. The gesture recognition method according to any one of claims 2 to 3, characterized in that, The method further comprises: At least one angle threshold is obtained, wherein the angle threshold is obtained by collecting human pointing action statistics; According to the angle threshold, the corresponding constraint rule is determined, which comprises: the first constraint rule is determined according to the vertical angle threshold; or, the second constraint rule is determined according to the plane angle threshold; or, the third constraint rule is determined according to the plane angle threshold; or, the fourth constraint rule is determined according to the straight line angle threshold.

6. The gesture recognition method of claim 1, wherein, The three-dimensional posture information of the target object comprises the three-dimensional coordinates of the key points of the left shoulder joint, the right shoulder joint and the hip joint close to the first hand, and the projection plane of the first torso is constructed according to the three-dimensional coordinates of the key points of the left shoulder joint, the right shoulder joint and the hip joint; or, The three-dimensional posture information of the target object includes a key point three-dimensional coordinate of a left shoulder joint, a key point three-dimensional coordinate of a right shoulder joint, a key point three-dimensional coordinate of a left hip joint, and a key point three-dimensional coordinate of a right hip joint; a total projection plane is constructed according to the key point three-dimensional coordinate of the left shoulder joint, the key point three-dimensional coordinate of the right shoulder joint, the key point three-dimensional coordinate of the left hip joint, and the key point three-dimensional coordinate of the right hip joint; and one half of the total projection plane is divided into a projection plane of the first trunk.

7. The gesture recognition method of claim 1, wherein, The three-dimensional posture information of the target object includes a key point three-dimensional coordinate of a shoulder joint close to the first hand, a key point three-dimensional coordinate of an elbow joint, and a key point three-dimensional coordinate of a hand joint; a pointing vector of the first hand is determined according to the key point three-dimensional coordinate of the shoulder joint, the key point three-dimensional coordinate of the elbow joint, and the key point three-dimensional coordinate of the hand joint; or The three-dimensional posture information of the target object includes a key point three-dimensional coordinate of a shoulder joint close to the first hand and a key point three-dimensional coordinate of a hand joint; a pointing vector of the first hand is determined according to the key point three-dimensional coordinate of the shoulder joint and the key point three-dimensional coordinate of the hand joint; or The three-dimensional posture information of the target object includes a key point three-dimensional coordinate of an elbow joint close to the first hand and a key point three-dimensional coordinate of a hand joint; a pointing vector of the first hand is determined according to the key point three-dimensional coordinate of the elbow joint and the key point three-dimensional coordinate of the hand joint.

8. The gesture recognition method of claim 1, wherein, Before constructing a projection plane of at least one of the left and right trunks of the target object according to three-dimensional posture information of the target object, the method further comprises: acquiring a visible light image and a depth image containing an image of the target object; obtaining two-dimensional posture information of the target object according to the visible light image; determining a depth value corresponding to the two-dimensional posture of each object according to the depth image; and obtaining three-dimensional posture information of the target object according to the two-dimensional posture information of the target object and the depth value.

9. An electronic device comprising: An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to perform the gesture recognition method of any one of claims 1 to 8.

10. A gesture recognition system, characterized in that An electronic device comprising: an image acquisition device connected to the electronic device, wherein the image acquisition device is capable of acquiring a visible light image and a depth image.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the gesture recognition method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the gesture recognition method of any one of claims 1 to 8.

Images