Gesture recognition and article display method and device, medium, terminal and program product

Abstract

The invention provides a gesture recognition and article display method and device, a medium, a terminal and a program product, and the method comprises the steps: collecting an image in real time, and recognizing a hand region in the collected image; detecting the three-dimensional position of each hand key point from the hand region by using a hand key point detection model obtained by pre-training; according to the detected three-dimensional position of each hand key point, establishing a relative distance relationship between the hand key points to obtain a hand model; constructing a depth coefficient equation according to the hand model and the three-dimensional position of each hand key point, and solving according to the depth coefficient equation to obtain a depth coordinate of each hand key point so as to obtain a three-dimensional coordinate of each hand key point; and according to the obtained three-dimensional coordinates of each hand key point, performing gesture recognition according to a predefined gesture category. According to the scheme provided by the invention, the cost and complexity of the system can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of control, and in particular to a method, device, medium, terminal and program product for gesture recognition and object display. Specifically, it relates to a method, device, computer-readable storage medium, terminal and computer program product for gesture recognition and an object display method, device, computer-readable storage medium, terminal and computer program product. Background Art

[0002] With the rapid development of virtual reality (VR), augmented reality (AR), metaverse and intelligent human-computer interaction technology, digital humans have become the core components in a variety of application scenarios. Digital humans can not only be used as virtual assistants and virtual anchors, but are also gradually being widely used in entertainment, education, medical and other fields. In these applications, achieving natural and smooth human-computer interaction is the key, especially gesture-based interaction, which has become a hot topic of research due to its intuitiveness and convenience.

[0003] Gesture interaction systems in related technologies usually rely on external sensors (such as gloves, depth cameras, etc.) to capture the user's hand movements. However, such systems have problems such as expensive equipment, inconvenient wearing, and poor environmental adaptability, which limit their promotion in large-scale applications. At the same time, gesture recognition methods based on simple rules or template matching often cannot meet actual needs in terms of recognition accuracy and real-time performance when facing complex gestures or fast movements. Summary of the invention

[0004] The main purpose of the present invention is to overcome the defects of the above-mentioned related technologies and to provide a gesture recognition and object display method, device, medium, terminal and program product to solve the problem that gesture interaction in the related technologies depends on external sensors and the gesture recognition method cannot meet the actual needs in terms of recognition accuracy and real-time performance when facing complex gestures or rapid movements.

[0005] On the one hand, the present invention provides a gesture recognition method, comprising: acquiring images in real time, and identifying a hand area in the acquired images; using a pre-trained hand key point detection model to detect the three-dimensional position of each hand key point from the hand area; the three-dimensional position of the hand key point includes the two-dimensional coordinates and relative depth coordinates of the hand key point; according to the detected three-dimensional position of each hand key point, a relative distance relationship between each hand key point is established to obtain a hand model; according to the hand model and the three-dimensional position of each hand key point, a depth coefficient equation is constructed, and the depth coordinate of each hand key point is obtained by solving the depth coefficient equation, thereby obtaining the three-dimensional coordinate of each hand key point; according to the obtained three-dimensional coordinates of each hand key point, gesture recognition is performed according to a pre-defined gesture category.

[0006] Optionally, the hand area in the captured image is identified; for the first N frames of the captured image, a pre-trained hand detection model is used to detect the hand area in the captured image; after capturing N frames of the image, a motion hypothesis is made based on the motion information of the hand in the first N frames of the captured image, and the hand area in the captured image after the Nth frame is predicted.

[0007] Optionally, the hand detection model is trained by the following steps: obtaining a pre-labeled hand image annotation dataset; and performing model training using the hand image annotation dataset to obtain the hand detection model.

[0008] Optionally, the hand key point detection model is trained through the following steps: obtaining the two-dimensional coordinates and corresponding relative depth coordinates of each pre-labeled hand key point; selecting a model, setting initial model parameters, and using the two-dimensional coordinates and corresponding relative depth coordinates of each pre-labeled hand key point to train the model; constructing a loss function, and optimizing the model parameters using gradient descent and back propagation algorithms based on the constructed loss function to obtain the hand key point detection model.

[0009] Optionally, gesture recognition is performed according to a predefined gesture category based on the obtained three-dimensional coordinates of each hand key point, including: identifying the state of each finger according to the three-dimensional coordinates of each hand key point; determining the gesture category of the current gesture based on the identified state of each finger and the predefined gesture category.

[0010] Optionally, the state of each finger is identified based on the three-dimensional coordinates of each hand key point, including: for any finger, constructing a vector between every two adjacent key points of the finger; calculating a curvature index based on the constructed vector between every two adjacent key points and the angle between every two adjacent vectors; and determining whether the finger state is a bent or straight state based on whether the calculated curvature index exceeds a preset threshold.

[0011] On the one hand, the present invention also provides an object display method, comprising: identifying the gesture category of a gesture currently performed by a user through any of the aforementioned gesture recognition methods; displaying a three-dimensional model of the displayed object according to the identified gesture category, wherein different gesture categories can perform three-dimensional transformation on the displayed object.

[0012] Another aspect of the present invention provides a gesture recognition device, comprising: an acquisition unit, used to acquire images in real time; a hand recognition unit, used to recognize a hand region in the image acquired by the acquisition unit; a detection unit, used to detect a three-dimensional position of each hand key point from the hand region using a pre-trained hand key point detection model; the three-dimensional position of the hand key point includes a two-dimensional coordinate and a relative depth coordinate of the hand key point;

[0013] An establishing unit is used to establish a relative distance relationship between each hand key point according to the three-dimensional position of each hand key point detected by the detecting unit, so as to obtain a hand model; a solving unit is used to construct a depth coefficient equation according to the hand model and the three-dimensional position of each hand key point, and solve the depth coefficient equation to obtain the depth coordinate of each hand key point, thereby obtaining the three-dimensional coordinate of each hand key point; a gesture recognition unit is used to perform gesture recognition according to a predefined gesture category based on the obtained three-dimensional coordinates of each hand key point.

[0014] Optionally, it is characterized in that the hand recognition unit recognizes the hand area in the captured image; for the first N frames of the captured image, a pre-trained hand detection model is used to detect the hand area in the captured image; after capturing N frames of the image, a motion hypothesis is made based on the motion information of the hand in the first N frames of the captured image, and the hand area in the captured image after the Nth frame is predicted.

[0015] Optionally, the hand detection model is trained by the following steps: obtaining a pre-labeled hand image annotation dataset; and performing model training using the hand image annotation dataset to obtain the hand detection model.

[0016] Optionally, the hand key point detection model is trained through the following steps: obtaining the two-dimensional coordinates and corresponding relative depth coordinates of each pre-labeled hand key point; selecting a model, setting initial model parameters, and using the two-dimensional coordinates and corresponding relative depth coordinates of each pre-labeled hand key point to train the model; constructing a loss function, and optimizing the model parameters using gradient descent and back propagation algorithms based on the constructed loss function to obtain the hand key point detection model.

[0017] Optionally, the gesture recognition unit performs gesture recognition according to a predefined gesture category based on the obtained three-dimensional coordinates of each hand key point, including: identifying the state of each finger according to the three-dimensional coordinates of each hand key point; determining the gesture category of the current gesture based on the identified state of each finger and the predefined gesture category.

[0018] Optionally, the gesture recognition unit identifies the state of each finger according to the three-dimensional coordinates of each hand key point, including: for any finger, constructing a vector between every two adjacent key points of the finger; calculating a curvature index based on the constructed vector between every two adjacent key points and the angle between every two adjacent vectors; and determining whether the finger state is a bent or straight state based on whether the calculated curvature index exceeds a preset threshold.

[0019] On the other hand, the present invention also provides an object display device, comprising any of the aforementioned gesture recognition devices, and also comprising: a recognition unit and a display unit; the recognition unit is used to call the gesture recognition device to recognize the gesture category of the gesture currently made by the user; the display unit is used to display the three-dimensional model of the displayed object according to the recognized gesture category, wherein different gesture categories can perform three-dimensional transformation on the displayed object.

[0020] Another aspect of the present invention provides a storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the steps of any of the aforementioned gesture recognition methods, or implements the steps of any of the aforementioned object display methods.

[0021] On the other hand, the present invention provides a terminal, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the aforementioned gesture recognition methods or the steps of any of the aforementioned object display methods are implemented.

[0022] In yet another aspect, the present invention provides a terminal, comprising any of the aforementioned gesture recognition devices, or comprising any of the aforementioned object display devices.

[0023] In yet another aspect, the present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of any of the aforementioned gesture recognition methods, or implements the steps of any of the aforementioned object display methods.

[0024] According to the technical solution of the present invention, the coordinates of the key points of the hand are processed in three dimensions, so that the hand has good gesture recognition accuracy at different angles and scales.

[0025] According to the technical solution of the present invention, a hand detection model is used in combination with motion prediction to perform key point detection, which ensures high-precision detection while achieving real-time response of gesture recognition, ensuring the smoothness of interaction and improving user experience.

[0026] According to the technical solution of the present invention, ordinary cameras can be used to replace dedicated gesture interaction devices (such as depth cameras, sensor gloves), which reduces the cost and complexity of the system, makes the equipment easier to deploy and use, and expands the feasibility of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 is a method schematic diagram of an embodiment of a gesture recognition method provided by the present invention;

[0029] Figure 2 is a schematic diagram of key points of a hand according to a specific embodiment of the present invention;

[0030] Figure 3a is a schematic diagram of the coordinates of key points of a hand in a three-dimensional space according to a specific embodiment of the present invention;

[0031] Figure 3b is a schematic diagram of the coordinates of key points of a hand in a three-dimensional space according to another specific embodiment of the present invention;

[0032] Figure 4 A schematic diagram of linkage with all virtual humans is shown;

[0033] Figure 5 is a method schematic diagram of a specific embodiment of the gesture recognition method provided by the present invention;

[0034] Figure 6 It is a structural block diagram of an embodiment of a gesture recognition device provided by the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] The gesture recognition in the related art still has the following problems:

[0038] High cost and equipment complexity: The gesture interaction systems in related technologies often rely on high-cost external devices, such as depth cameras, sensor gloves, etc., which not only increases the complexity and usage threshold of the system, but also limits the feasibility of its widespread application.

[0039] Insufficient recognition accuracy: The gesture recognition methods based on template matching or simple rules in related technologies have limited recognition accuracy when facing fast or complex gestures, and are difficult to adapt to diverse application scenarios, especially when the gestures are changeable, the lighting is unstable or occluded.

[0040] Poor real-time performance: Some gesture recognition systems in related technologies cannot guarantee real-time performance in terms of computational complexity and processing speed, resulting in delayed response of gesture control and affecting user experience.

[0041] Lack of adaptability and versatility: The gesture interaction systems in related technologies have poor adaptability to environmental conditions (such as lighting, background) or individual differences of users (such as hand shape, skin color), and cannot provide wide and robust applications.

[0042] The invention provides a gesture recognition method.

[0043] Figure 1 1 is a method schematic diagram of an embodiment of a gesture recognition method provided by the present invention.

[0044] like Figure 1 As shown, according to an embodiment of the present invention, the gesture recognition method at least includes step S110, step S120, step S130, step S140, step S150 and step S160.

[0045] Step S110, collecting images in real time, and identifying the hand area in the collected images.

[0046] Specifically, images are collected in real time through image acquisition devices. Preferably, the camera can choose a camera that supports high resolution to ensure recognition accuracy, and the camera can be selected according to the ambient light. For example, a camera with low-light performance capabilities can be selected when used for display equipment in a science and technology exhibition hall. Set parameters such as resolution, refresh rate, aperture, white balance, etc. as needed. For example, the camera can use a resolution of 1080p to improve the accuracy of image recognition, 60FPS to ensure a smooth experience of dynamic interaction, increase gain and shorten exposure time to reduce image blur, and set the aperture, white balance, and focus to automatic mode to ensure clear and bright imaging.

[0047] Step S120, identifying the hand area in the acquired image.

[0048] In a specific embodiment, the hand area in the first N frames of images collected is detected using a pre-trained hand detection model; after collecting N frames of images, a motion hypothesis is made based on the motion information of the hand in the first N frames of images collected, and the hand area in the images collected after the Nth frame is predicted.

[0049] Specifically, the captured image is input into the hand detection model, and the hand area detection result is output. The hand area detection result may specifically include a hand boundary box. The hand detection model may specifically be a YOLO model. For example, it may specifically be a YOLOv5 (You Only Look Once v5) hand detection model.

[0050] The hand detection model is trained through the following steps:

[0051] Step S11, obtaining a pre-annotated hand image annotation dataset.

[0052] Specifically, hand images in different postures and environments are acquired in advance for annotation. The annotation file uses the YOLO format and includes the position of the hand in the image (bounding box). Specifically, it may include: the center point coordinates (x, y), width and height (w, h) of the hand bounding box.

[0053] Step S12: Perform model training using the hand image annotation dataset to obtain the hand detection model.

[0054] Specifically, since the only detection target is the hand and real-time detection is required, the YOLOv5 model can be used, and its pre-trained weights on the COCO dataset can be loaded for transfer learning to accelerate the model training of hand detection.

[0055] By inputting the real-time captured image into the trained hand detection model, the hand area detection result can be output. For example, the hand area is selected with a bounding box in the image and the detection confidence is displayed at the same time.

[0056] In order to achieve the goal of real-time and efficient detection, it is not necessary to perform hand detection for each frame of image first, but a hybrid method of hand detection and hand area tracking is used to quickly identify and track the hand position. That is, for the first N frames of images collected, the hand area in the collected images is detected using a pre-trained hand detection model; after collecting N frames of images, a motion hypothesis is made based on the motion information of the hands in the first N frames of images collected, and the hand area in the images collected after the Nth frame is predicted. The motion information may specifically include: speed, direction, acceleration and motion state.

[0057] The present invention uses a tracking strategy based on historical information and motion prediction. The motion assumption is based on the continuity of hand motion and the smoothness of speed and direction. To perform motion prediction, the system needs to obtain the motion information of the object (hand), mainly including speed, direction, acceleration and motion state.

[0058] The prediction process of the motion hypothesis may specifically include: predicting the position of the hand in the current frame based on the speed and direction information of the hand in the previous N frames. Assume that the center point of the bounding box of the hand in the t-1th frame is (x t-1 ,y t-1 ), and calculated the hand velocity v x ,v y , acceleration a x ,a y , △t is the time difference between two frames of images. In the tth frame, the position of the hand center point can be predicted according to the following formula:

[0059]

[0060] Similarly, we can assume that the position of any hand key point in the t-1th frame is (x t-1 ,y t-1 ), and calculated the hand velocity v x ,v y , acceleration a x ,a y , △t is the time difference between two frames of images. In the tth frame, the position of any hand key point can be predicted according to the following formula: predict the position of each hand key point. When all hand key points are predicted, all hand key points are taken as a whole, and its minimum circumscribed rectangle is calculated to obtain the predicted hand bounding box.

[0061] Step S130, using the pre-trained hand key point detection model, detect the three-dimensional position of each hand key point from the hand area.

[0062] Figure 2 is a schematic diagram of the key points of the hand according to a specific embodiment of the present invention. The key points of the hand may specifically include the joints and fingertips of the hand. The hand may specifically include 21 key points of the hand. Figure 2 As shown, they are wrist 0, thumb base joint 1, thumb metacarpophalangeal joint 2, thumb interphalangeal joint 3, thumb tip 4, index finger metacarpophalangeal joint 5, index finger proximal interphalangeal joint 6, index finger distal interphalangeal joint 7, index finger tip 8, middle finger metacarpophalangeal joint 9, middle finger proximal interphalangeal joint 10, middle finger distal interphalangeal joint 11, middle finger tip 12, ring finger metacarpophalangeal joint 13, ring finger proximal interphalangeal joint 14, ring finger distal interphalangeal joint 15, ring finger tip 16, little finger metacarpophalangeal joint 17, little finger proximal interphalangeal joint 18, little finger distal interphalangeal joint 19, little finger tip 20.

[0063] The three-dimensional position of the hand key point includes the two-dimensional coordinates and relative depth coordinates of the hand key point. In a specific implementation, the hand key point detection model is trained by the following steps:

[0064] Step S21, obtaining the two-dimensional coordinates of each pre-marked hand key point and the corresponding relative depth coordinates.

[0065] Specifically, obtain the two-dimensional coordinates and corresponding relative depth coordinates of the hand key points that have been manually annotated in advance. The relative depth coordinate z, that is, add the relative depth information z on the basis of the two-dimensional coordinate (x, y), that is, the front-to-back relationship between the key points. The relative depth information can be limited to -1 to 1. Specifically, the relative depth of the wrist joint point is z=0, the direction close to the camera is the negative direction, and the direction away from the camera is the positive direction, and the relative depth coordinate of any hand key point can be obtained.

[0066] Preferably, data enhancement and data preprocessing are performed on the two-dimensional coordinates of the pre-annotated hand key points and the corresponding relative depth coordinates, for example, by rotating the original image, changing the hue, adding noise, scaling, etc., and then normalizing to (0,1).

[0067] Step S22, selecting a model, setting initial model parameters, and using the two-dimensional coordinates of each pre-annotated hand key point and the corresponding relative depth coordinates to perform model training;

[0068] Step S23, constructing a loss function, and optimizing model parameters using gradient descent and back propagation algorithms based on the constructed loss function to obtain the hand key point detection model.

[0069] Specifically, weights are assigned to the x, y, and z coordinates respectively, and a loss function is constructed. Based on the constructed loss function, the difference between the coordinates output by the model and the coordinates of the original annotations is compared. The gradient descent and back propagation algorithms are used to optimize the model parameters in the direction of decreasing the loss value. Since the scales of the x, y coordinates and the z coordinate are inconsistent, directly using the same weights may cause the model to pay more attention to the coordinate axis with a larger scale, thereby affecting the accuracy of the prediction. To solve this problem, appropriate weights can be assigned to each coordinate axis to balance their impact on the loss. For example, the loss function constructed based on the variance weight assignment is as follows:

[0070]

[0071] loss = w x ·(x pred -x true ) 2 +w y ·(y pred -y true ) 2 +w z ·(z pred -z true ) 2

[0072] Among them, loss represents loss, a, b, c are the range of change of x, y, z coordinates respectively, the subscript pred represents the predicted value, and the subscript true represents the true value.

[0073] Step S140, establishing a relative distance relationship between each hand key point according to the detected three-dimensional position of each hand key point to obtain a hand model.

[0074] First, the three-dimensional coordinates of each detected hand key point are preprocessed, and the coordinates (x, y) of each detected hand key point are converted from (0, 1) to pixel coordinates. Then, the relative distance relationship between each hand key point is established to obtain the hand model.

[0075] Specifically, the hand key point detection model is used to take hundreds of pictures in the position of the hand parallel camera (as the input of the hand key point detection model), and the corresponding key point coordinates are output. The relative distances between directly connected adjacent key points are averaged, and the distance d1 between key points 0 and 5 is taken as 1. The ratio of d1 and the distance dn between other directly connected key points to d1 is stored as the hand model.

[0076] Step S150, constructing a depth coefficient equation according to the hand model and the three-dimensional position of each hand key point, and solving the depth coefficient equation to obtain the depth coordinates of each hand key point, thereby obtaining the three-dimensional coordinates of each hand key point.

[0077] Specifically, assuming that the scaling coefficient of the relative depth coordinate (Z coordinate) of the hand key point is a polynomial, a depth coefficient equation group is constructed according to the hand model and the current three-dimensional coordinates (x, y, z) of the hand key point.

[0078] In practice, the hand pose is often not parallel to the camera, and z is a relative depth coordinate in the range of (-1,1). If the distance between the two key points a and b is calculated according to the following formula:

[0079]

[0080] The distance often does not conform to the hand model stored in the parallel position with the camera. Therefore, the depth coefficient equation is introduced to solve the mapping coefficient w of the relative depth difference between two coordinates in any position through the equation group. z .

[0081] w z =a n (z a -z b ) n +a n-1 (z a -z b ) n-1 +…+a1(z a -z b )+d

[0082]

[0083] is the mapping coefficient of the relative depth difference between z2 and z1, is the mapping coefficient of the relative depth difference between z3 and z2, ..., and so on, P is the mapping coefficient from pixel coordinates to world coordinates, when P is When it has been solved, the actual world depth coordinate Z1 can be set to 0;

[0084]

[0085] In the above formula, only z2 is unknown, and the value of z2 can be solved.

[0086] Similarly,

[0087] Only z3 is unknown, and the value of z3 can be solved. In this way, all z values ​​can be solved. At this time, the z value is the world coordinate, and z / P is the pixel coordinate. From this, the coordinates (X, Y, Z) in three-dimensional space can be obtained. The three-dimensional coordinates can be plotted by Figure 3a , Figure 3b shown.

[0088] Step S160, performing gesture recognition according to the obtained three-dimensional coordinates of each hand key point and the predefined gesture category.

[0089] According to the three-dimensional coordinates of the key points of the hand, the state of each finger (for example, bent or straight) is identified, and the gesture category of the current gesture is determined according to the identified state of each finger. For any finger, a vector between every two adjacent key points of the finger is constructed; a curvature index is calculated according to the constructed vector between every two adjacent key points and the angle between every two adjacent vectors; and whether the calculated curvature index exceeds a preset threshold is determined to determine whether the finger state is bent.

[0090] refer to Figure 2 As shown in the figure, taking key points 0, 5, 6, 7, and 8 as an example, vector v1 is constructed with key points 0 and 5, and the direction is 0 to 5. Vector v2 is constructed with key points 5 and 6, and the direction is 5 to 6. Similarly, v3 and v4 are constructed. The angle between v1 and v2 is θ1, the angle between v2 and v3 is θ2, and the angle between v3 and v4 is θ3. The evaluation index score is a comprehensive measure of the three angles:

[0091] score=‖v2‖·θ1+‖v3‖·θ2+‖v4‖·θ3

[0092] If the score is greater than a preset threshold t, the state of the finger is determined to be bent, otherwise, the state of the finger is determined to be straight.

[0093] For example, in a specific implementation, the gesture category includes one or more of the following gestures: except for the activation gesture, all other gestures are effective in the activation state:

[0094] Activate gesture: When a palm appears within the detection range and is in an open state (all fingers are stretched out), and the coordinates of its 21 key points do not change significantly within a preset time (e.g. 1 second), the command state is entered.

[0095] Waving: When the five fingers are in a straight state, the direction of the entire hand model changes repeatedly within a certain range.

[0096] Like: When the thumb is extended and the other four fingers are contracted (that is, the other four fingers are bent).

[0097] Virtual cursor: Extend the index finger and shrink the four fingers. A virtual cursor will appear on the screen and move along with the position of key point 8.

[0098] Selected: When the spatial distance between the tip of the index finger (key point 8) and the tip of the middle finger (key point 12) is less than the threshold t, the gesture is valid only when the virtual cursor exists.

[0099] Translation: When the index finger is contracted and the four fingers are spread out, the hand moves in the direction of the wave.

[0100] Rotation: When the spatial distance between the tip of the index finger (key point 8) and the tip of the thumb (key point 4) is less than the threshold t, further movement at this time is equivalent to a rotation in the moving direction around the center of the object.

[0101] Scaling: When the left and right hands appear at the same time, with the thumb and index finger in an extended state and the other fingers in a contracted state, the object is scaled proportionally according to the spatial distance d between the key points 0 of the left and right hands.

[0102] Explosion effect: When all fingers are in a contracted state (clenched fist), it enters the explosion mode, and the amplitude of the explosion changes with the amplitude of the fingers opening.

[0103] Exit gesture: The exit gesture for all modes is the same as the initial activated mode.

[0104] The present invention also provides an object display method, which comprises: identifying the gesture category of the gesture currently made by the user through the gesture recognition method described in any of the above embodiments; and displaying the three-dimensional model of the displayed object according to the recognized gesture category.

[0105] Among them, different gesture categories can transform the displayed items in three dimensions. For example, when the item is an electrical appliance, different gestures can be used to display the three-dimensional model of the electrical appliance and its various components. For example, the gesture recognition method of the present invention can be linked with a holographic virtual digital human.

[0106] Figure 4 The figure shows the schematic diagram of linkage with the whole system virtual human. Figure 4 As shown. First, activate the system through the system activation gesture. After activating the system, you can use voice commands to open the 3D model of a product. The virtual cursor gesture (for example, the virtual cursor appears when the index finger and middle finger are pinched) will generate a corresponding cursor on the screen. You can use the cursor to select a part to enter the 3D model of the part. Other gestures such as explosion, translation, rotation and scaling will perform corresponding 3D transformations on the product or part model. When the exit gesture is executed, it will return to the previous layer, for example, the part view will return to the overall product view. If it is already the top product view, executing the exit gesture will exit the 3D model display program.

[0107] A specific scenario example of the present invention is as follows: When the holographic virtual digital human explains the product, the three-dimensional structure diagram of the product will be presented on the screen. The user or the explainer uses an activation gesture to activate the gesture system. First, he waves to say hello to the virtual digital human and issues a waving command, and the digital human responds at the same time. The structural explainer translates and rotates the model of the entire product structure diagram to understand the entire appearance, and explains the product at the same time. If the user is interested, an exploded view can be displayed, and the core components that the user wants to know can be opened by a digital human driven by a large model or by selecting a gesture for separate display. The zoom command is used to zoom the model to observe the details of the components in detail. Finally, the exit command is executed to complete this explanation.

[0108] In addition to product explanations in virtual digital humans, the gesture recognition technology of the present invention can also be applied to multiple fields such as virtual reality or augmented reality, smart home, medical treatment, education, robot control, and artistic creation. It not only improves the convenience and naturalness of interaction, but also provides contactless and remote control solutions for many scenarios, further enhancing the diversity and possibilities of application scenarios.

[0109] In order to clearly illustrate the technical solution of the present invention, the execution process of the gesture recognition method provided by the present invention is described below with reference to a specific embodiment.

[0110] Figure 5 FIG. 1 is a schematic diagram of a specific embodiment of the gesture recognition method provided by the present invention. Figure 5 As shown, the gesture recognition method includes the following steps:

[0111] Image acquisition: Use a camera installed in a fixed position to collect images in real time.

[0112] Hand detection: Use the YOLOv5 model to detect the hand area.

[0113] Key point detection: Use the key point detection model to detect the hand area and obtain the coordinates of the hand key points.

[0114] Key point coordinate preprocessing: Convert the hand key point coordinates (x, y) from (0, 1) to pixel coordinates.

[0115] Constructing a hand model: Using multiple sets of hand key point coordinates to establish a relative distance relationship between hand key points to obtain a hand model.

[0116] Construct the depth coefficient equation: Assuming that the scaling factor of the Z coordinate is a polynomial, construct the equation group based on the hand model and the coordinates (x, y, z) of the current key point.

[0117] Solve for relative depth coordinates: Use the constructed set of equations to solve the coefficients of the polynomial d.

[0118] Gesture recognition: Based on predefined gestures (translation, rotation, scaling) and 3D key points, identify which gesture the current key point represents.

[0119] Instructions are sent to the device based on the recognized gesture, and the device performs holographic display after execution.

[0120] There are two loops in the above process: 1. The image acquisition loop, the camera is always acquiring images in real time. 2. The hand area tracking loop: only the hand areas of the first few frames are detected by the YOLOv5 model, and the subsequent hand areas are predicted based on the hand motion information in the previous images.

[0121] Figure 6 FIG. 1 is a structural block diagram of an embodiment of a gesture recognition device provided by the present invention. Figure 6 As shown, the gesture recognition device 100 includes: a collection unit 110 , a hand recognition unit 120 , a detection unit 130 , a creation unit 140 , a solution unit 150 and a gesture recognition unit 160 .

[0122] The acquisition unit 110 is used to acquire images in real time.

[0123] Specifically, images are collected in real time through image acquisition devices. Preferably, the camera can choose a camera that supports high resolution to ensure recognition accuracy, and the camera can be selected according to the ambient light. For example, a camera with low-light performance capabilities can be selected when used for display equipment in a science and technology exhibition hall. Set parameters such as resolution, refresh rate, aperture, white balance, etc. as needed. For example, the camera can use a resolution of 1080p to improve the accuracy of image recognition, 60FPS to ensure a smooth experience of dynamic interaction, increase gain and shorten exposure time to reduce image blur, and set the aperture, white balance, and focus to automatic mode to ensure clear and bright imaging.

[0124] The hand recognition unit 120 is used to recognize the hand area in the image captured by the capture unit.

[0125] In a specific embodiment, the hand area in the first N frames of images collected is detected using a pre-trained hand detection model; after collecting N frames of images, a motion hypothesis is made based on the motion information of the hand in the first N frames of images collected, and the hand area in the images collected after the Nth frame is predicted.

[0126] Specifically, the captured image is input into the hand detection model, and the hand area detection result is output. The hand area detection result may specifically include a hand bounding box. The hand detection model may specifically be a YOLO model. For example, it may specifically be a YOLOv5 hand detection model. YOLOv5 (You Only Look Once v5) is an object detection framework based on deep learning. Its advantage is that it can efficiently and in real time detect the bounding boxes of multiple objects and classify each object.

[0127] The hand detection model is trained through the following steps:

[0128] Step S11, obtaining a pre-annotated hand image annotation dataset.

[0129] Specifically, hand images in different postures and environments are acquired in advance for annotation. The annotation file uses the YOLO format and includes the position of the hand in the image (bounding box). Specifically, it may include: the center point coordinates (x, y), width and height (w, h) of the hand bounding box.

[0130] Step S12: Perform model training using the hand image annotation dataset to obtain the hand detection model.

[0131] Specifically, since the only detection target is the hand and real-time detection is required, the YOLOv5 model can be used, and its pre-trained weights on the COCO dataset can be loaded for transfer learning to accelerate the model training of hand detection.

[0132] By inputting the real-time captured image into the trained hand detection model, the hand area detection result can be output. For example, the hand area is selected with a bounding box in the image and the detection confidence is displayed at the same time.

[0133] In order to achieve the goal of real-time and efficient detection, it is not necessary to perform hand detection for each frame of image first, but a hybrid method of hand detection and hand area tracking is used to quickly identify and track the hand position. That is, for the first N frames of images collected, the hand area in the collected images is detected using a pre-trained hand detection model; after collecting N frames of images, a motion hypothesis is made based on the motion information of the hands in the first N frames of images collected, and the hand area in the images collected after the Nth frame is predicted. The motion information may specifically include: speed, direction, acceleration and motion state.

[0134] The present invention uses a tracking strategy based on historical information and motion prediction. The motion assumption is based on the continuity of hand motion and the smoothness of speed and direction. To perform motion prediction, the system needs to obtain the motion information of the object (hand), mainly including speed, direction, acceleration and motion state.

[0135] The prediction process of the motion hypothesis may specifically include: predicting the position of the hand in the current frame based on the speed and direction information of the hand in the previous N frames. Assume that the center point of the bounding box of the hand in the t-1th frame is (x t-1 ,y t-1 ), and calculated the hand velocity v x ,v y , acceleration a x ,a y , △t is the time difference between two frames of images. In the tth frame, the position of the hand center point can be predicted according to the following formula:

[0136]

[0137] Similarly, we can assume that the position of any hand key point in the t-1th frame is (x t-1 ,y t-1 ), and calculated the hand velocity v x ,v y , acceleration a x ,a y , △t is the time difference between two frames of images. In the tth frame, the position of any hand key point can be predicted according to the following formula: predict the position of each hand key point. When all hand key points are predicted, all hand key points are taken as a whole, and its minimum circumscribed rectangle is calculated to obtain the size of the predicted hand bounding box.

[0138] The detection unit 130 is used to detect the three-dimensional position of each hand key point from the hand area using a pre-trained hand key point detection model; the three-dimensional position of the hand key point includes the two-dimensional coordinates and relative depth coordinates of the hand key point.

[0139] Figure 2 is a schematic diagram of the key points of the hand according to a specific embodiment of the present invention. The key points of the hand may specifically include the joints and fingertips of the hand. The hand may specifically include 21 key points of the hand. Figure 2 As shown, they are wrist 0, thumb base joint 1, thumb metacarpophalangeal joint 2, thumb interphalangeal joint 3, thumb tip 4, index finger metacarpophalangeal joint 5, index finger proximal interphalangeal joint 6, index finger distal interphalangeal joint 7, index finger tip 8, middle finger metacarpophalangeal joint 9, middle finger proximal interphalangeal joint 10, middle finger distal interphalangeal joint 11, middle finger tip 12, ring finger metacarpophalangeal joint 13, ring finger proximal interphalangeal joint 14, ring finger distal interphalangeal joint 15, ring finger tip 16, little finger metacarpophalangeal joint 17, little finger proximal interphalangeal joint 18, little finger distal interphalangeal joint 19, little finger tip 20.

[0140] The three-dimensional position of the hand key point includes the two-dimensional coordinates and relative depth coordinates of the hand key point. In a specific implementation, the hand key point detection model is trained by the following steps:

[0141] Step S21, obtaining the two-dimensional coordinates of each pre-marked hand key point and the corresponding relative depth coordinates.

[0142] Specifically, obtain the two-dimensional coordinates and corresponding relative depth coordinates of the hand key points that have been manually annotated in advance. The relative depth coordinate z, that is, add the relative depth information z on the basis of the two-dimensional coordinate (x, y), that is, the front-to-back relationship between the key points. The relative depth information can be limited to -1 to 1. Specifically, the relative depth of the wrist joint point is z=0, the direction close to the camera is the negative direction, and the direction away from the camera is the positive direction, and the relative depth coordinate of any hand key point can be obtained.

[0143] Preferably, data enhancement and data preprocessing are performed on the two-dimensional coordinates of the pre-annotated hand key points and the corresponding relative depth coordinates, for example, by rotating the original image, changing the hue, adding noise, scaling, etc., and then normalizing to (0,1).

[0144] Step S22, selecting a model, setting initial model parameters, and using the two-dimensional coordinates of each pre-annotated hand key point and the corresponding relative depth coordinates to perform model training;

[0145] Step S23, constructing a loss function, and optimizing model parameters using gradient descent and back propagation algorithms based on the constructed loss function to obtain the hand key point detection model.

[0146] Specifically, weights are assigned to the x, y, and z coordinates respectively, a loss function is constructed, and the model is trained based on the constructed loss function. Since the scales of the x, y coordinates and the z coordinate are inconsistent, directly using the same weights may cause the model to pay more attention to the coordinate axis with a larger scale, thereby affecting the accuracy of the prediction. To solve this problem, appropriate weights can be assigned to each coordinate axis to balance their impact on the loss. For example, the loss function constructed based on variance weight allocation is as follows:

[0147]

[0148] loss = w x ·(x pred -x true ) 2 +w y ·(y pred -y true ) 2 +w z ·(z pred-z true ) 2

[0149] Among them, loss represents loss, a, b, c are the range of change of x, y, z coordinates respectively, the subscript pred represents the predicted value, and the subscript true represents the true value.

[0150] The establishing unit 140 is used to establish the relative distance relationship between each hand key point according to the three-dimensional position of each hand key point detected by the detecting unit 130 to obtain a hand model.

[0151] First, the three-dimensional coordinates of each detected hand key point are preprocessed, and the coordinates (x, y) of each detected hand key point are converted from (0, 1) to pixel coordinates. Then, the relative distance relationship between each hand key point is established to obtain the hand model.

[0152] Specifically, the hand key point detection model is used to take hundreds of pictures in the position of the hand parallel camera (as the input of the hand key point detection model), and the corresponding key point coordinates are output. The relative distances between directly connected adjacent key points are averaged, and the distance d1 between key points 0 and 5 is taken as 1. The ratio of d1 and the distance dn between other directly connected key points to d1 is stored as the hand model.

[0153] The solving unit 150 is used to construct a depth coefficient equation according to the hand model and the three-dimensional position of each hand key point, and solve the depth coefficient equation to obtain the depth coordinate of each hand key point, thereby obtaining the three-dimensional coordinate of each hand key point.

[0154] Specifically, assuming that the scaling coefficient of the relative depth coordinate (Z coordinate) of the hand key point is a polynomial, a depth coefficient equation group is constructed according to the hand model and the current three-dimensional coordinates (x, y, z) of the hand key point.

[0155] In practice, the hand pose is often not parallel to the camera, and z is a relative depth coordinate in the range of (-1,1). If the distance between the two key points a and b is calculated according to the following formula:

[0156]

[0157] The distance often does not conform to the hand model stored in the parallel position with the camera. Therefore, the depth coefficient equation is introduced to solve the mapping coefficient w of the relative depth difference between two coordinates in any position through the equation group. z .

[0158] w z =a n (za -z b ) n +a n-1 (z a -z b ) n-1 +…+a1(z a -z b )+d

[0159]

[0160] is the mapping coefficient of the relative depth difference between z2 and z1, is the mapping coefficient of the relative depth difference between z3 and z2, ..., and so on, P is the mapping coefficient from pixel coordinates to world coordinates, when P is When it has been solved, the actual world depth coordinate Z1 can be set to 0;

[0161]

[0162] In the above formula, only z2 is unknown, and the value of z2 can be solved.

[0163] Similarly,

[0164] Only z3 is unknown, so we can solve for z3. In this way, we can solve for all z values. At this time, z value is the world coordinate, z / P is the pixel coordinate, and thus, we can get the coordinates (X, Y, Z) in three-dimensional space. For the three-dimensional coordinates, see Figure 3a , Figure 3b shown.

[0165] The gesture recognition unit 160 is used to perform gesture recognition according to the obtained three-dimensional coordinates of each hand key point according to a predefined gesture category.

[0166] According to the three-dimensional coordinates of the key points of the hand, the state of each finger (for example, bent or straight) is identified, and the gesture category of the current gesture is determined according to the identified state of each finger. For any finger, a vector between every two adjacent key points of the finger is constructed; a curvature index is calculated according to the constructed vector between every two adjacent key points and the angle between every two adjacent vectors; and whether the calculated curvature index exceeds a preset threshold is determined to determine whether the finger state is bent.

[0167] Test Figure 2As shown in the figure, taking key points 0, 5, 6, 7, and 8 as an example, vector v1 is constructed with key points 0 and 5, and the direction is 0 to 5. Vector v2 is constructed with key points 5 and 6, and the direction is 5 to 6. Similarly, v3 and v4 are constructed. The angle between v1 and v2 is θ1, the angle between v2 and v3 is θ2, and the angle between v3 and v4 is θ3. The evaluation index score is a comprehensive measure of the three angles:

[0168] score=‖v2‖·θ1+‖v3‖·θ2+‖v4‖·θ3

[0169] If the score is greater than a preset threshold t, the state of the finger is determined to be bent, otherwise, the state of the finger is determined to be straight.

[0170] For example, in a specific implementation, the gesture category includes one or more of the following gestures: except for the activation gesture, all other gestures are effective in the activation state:

[0171] Activate gesture: When a palm appears within the detection range and is in an open state (all fingers are stretched out), and the coordinates of its 21 key points do not change significantly within a preset time (e.g. 1 second), the command state is entered.

[0172] Waving: When the five fingers are in a straight state, the direction of the entire hand model changes repeatedly within a certain range.

[0173] Like: When the thumb is extended and the other four fingers are in a contracted state (that is, the other four fingers are in a bent state).

[0174] Virtual cursor: Extend the index finger and shrink the four fingers. A virtual cursor will appear on the screen and move along with the position of key point 8.

[0175] Selected: When the spatial distance between the tip of the index finger (key point 8) and the tip of the middle finger (key point 12) is less than the threshold t, the gesture is valid only when the virtual cursor exists.

[0176] Translation: When the index finger is contracted and the four fingers are spread out, the hand moves in the direction of the wave.

[0177] Rotation: When the spatial distance between the tip of the index finger (key point 8) and the tip of the thumb (key point 4) is less than the threshold t, further movement at this time is equivalent to a rotation in the moving direction around the center of the object.

[0178] Scaling: When the left and right hands appear at the same time, with the thumb and index finger in an extended state and the other fingers in a contracted state, the object is scaled proportionally according to the spatial distance d between the key points 0 of the left and right hands.

[0179] Explosion effect: When all fingers are in a contracted state (clenched fist), it enters the explosion mode, and the amplitude of the explosion changes with the amplitude of the fingers opening.

[0180] Exit gesture: The exit gesture for all modes is the same as the initial activated mode.

[0181] The present invention also provides an object display device. The device includes any of the above-mentioned gesture recognition devices, and also includes: a recognition unit and a display unit; the recognition unit is used to call the gesture recognition device to recognize the gesture category of the gesture currently made by the user; the display unit is used to display the three-dimensional model of the displayed object according to the recognized gesture category, wherein different gesture categories can perform three-dimensional transformation on the displayed object.

[0182] Among them, different gesture categories can transform the displayed items in three dimensions. For example, when the item is an electrical appliance, different gestures can be used to display the three-dimensional model of the electrical appliance and its various components. For example, the gesture recognition method of the present invention can be linked with a holographic virtual digital human.

[0183] Figure 4 The figure shows the schematic diagram of linkage with the whole system virtual human. Figure 4 As shown. First, activate the system through the system activation gesture. You can use voice commands to open the 3D model of a product. The virtual cursor gesture will generate a corresponding cursor on the screen. You can use the cursor to select a part to enter the 3D model of the part. Other gestures such as explosion, translation, rotation and scaling will perform three-dimensional transformations on the product or part model. When the exit gesture is executed, it will return to the previous layer, for example, the part view will return to the overall product view. If it is already the top product view, executing the exit gesture will exit the 3D model display program.

[0184] A specific scenario example of the present invention is as follows: When the holographic virtual digital human explains the product, the three-dimensional structure diagram of the product will be presented on the screen. The user or the explainer uses an activation gesture to activate the gesture system. First, he waves to say hello to the virtual digital human and issues a waving command, and the digital human responds at the same time. The structural explainer translates and rotates the model of the entire product structure diagram to understand the entire appearance, and explains the product at the same time. If the user is interested, an exploded view can be displayed, and the core components that the user wants to know can be opened by a digital human driven by a large model or by selecting a gesture for separate display. The zoom command is used to zoom the model to observe the details of the components in detail. Finally, the exit command is executed to complete this explanation.

[0185] The present invention also provides a storage medium corresponding to the gesture recognition method, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0186] The present invention also provides a terminal corresponding to the gesture recognition method, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the aforementioned gesture recognition methods are implemented, or the steps of any of the aforementioned object display methods are implemented.

[0187] The present invention also provides a terminal corresponding to the gesture recognition device, including any of the aforementioned gesture recognition devices, or including any of the aforementioned object display devices.

[0188] The present invention also provides a computer program product corresponding to the gesture recognition method, including a computer program, which, when executed by a processor, implements the steps of any of the aforementioned gesture recognition methods, or implements the steps of any of the aforementioned object display methods.

[0189] Based on this, the solution provided by the present invention performs three-dimensional processing on the coordinates of the key points of the hand, so that it has good gesture recognition accuracy under different angles and scales.

[0190] The solution provided by the present invention uses a hand detection model combined with motion prediction to perform key point detection, which achieves real-time response of gesture recognition while ensuring high-precision detection, ensures smooth interaction, and improves user experience.

[0191] The solution provided by the present invention can use ordinary cameras to replace dedicated gesture interaction devices (such as depth cameras and sensor gloves), reducing the cost and complexity of the system, making the equipment easier to deploy and use, and expanding the feasibility of application scenarios.

[0192] The solution provided by the present invention defines a variety of gesture operations, such as virtual cursor control, selection, translation, rotation, zooming, explosion and other interactive gestures, for achieving intuitive and efficient user interaction.

[0193] The solution provided by the present invention utilizes convolutional neural networks (CNN) and other deep learning models to perform hand key point detection and gesture recognition, and the recognition accuracy is still high even in fast motion or occlusion situations.

[0194] The solution provided by the present invention, through an optimized deep learning model and an efficient inference engine, the system can process user gestures in real time and make rapid responses, thereby ensuring the user experience.

[0195] The solution provided by the present invention has strong environmental adaptability through a large amount of diversified data training, and can maintain stable gesture recognition under different lighting conditions and background interference.

[0196] The gesture interaction system using the gesture recognition solution of the present invention is applicable to multiple fields, including virtual reality (VR), augmented reality (AR), smart home control, virtual social interaction, game interaction, virtual shopping, etc., which significantly improves the usability and interactive experience of digital human gesture control and promotes the application of digital human technology in multiple scenarios.

[0197] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0199] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0201] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A gesture recognition method, characterized in that: include: Collecting images in real time and identifying hand areas in the collected images; Using a pre-trained hand key point detection model, the three-dimensional position of each hand key point is detected from the hand area; the three-dimensional position of the hand key point includes the two-dimensional coordinates and relative depth coordinates of the hand key point; According to the detected three-dimensional position of each hand key point, a relative distance relationship between each hand key point is established to obtain a hand model; Constructing a depth coefficient equation according to the hand model and the three-dimensional position of each hand key point, and solving the depth coefficient equation to obtain the depth coordinate of each hand key point, thereby obtaining the three-dimensional coordinate of each hand key point; According to the obtained three-dimensional coordinates of each hand key point, gesture recognition is performed according to a predefined gesture category.

2. The method according to claim 1, characterized in that identifying a hand region in the acquired image; For the first N frames of images collected, use the pre-trained hand detection model to detect the hand area in the collected images; After collecting N frames of images, a motion hypothesis is made based on the motion information of the hand in the first N frames of images collected, and the hand area in the images collected after the Nth frame is predicted.

3. The method according to claim 1, characterized in that The hand detection model is trained through the following steps: Get a pre-annotated hand image annotation dataset; The hand image annotation dataset is used to perform model training to obtain the hand detection model.

4. The method according to claim 1, characterized in that The hand key point detection model is trained through the following steps: Obtain the two-dimensional coordinates of each pre-marked hand key point and the corresponding relative depth coordinates; Selecting a model, setting initial model parameters, and using the pre-annotated two-dimensional coordinates of each hand key point and the corresponding relative depth coordinates to perform model training; A loss function is constructed, and based on the constructed loss function, a gradient descent and back propagation algorithm is used to optimize the model parameters to obtain the hand key point detection model.

5. The method according to any one of claims 1 to 4, characterized in that: According to the obtained three-dimensional coordinates of each hand key point, gesture recognition is performed according to a predefined gesture category, including: According to the three-dimensional coordinates of each key point of the hand, the state of each finger is identified; The gesture category of the current gesture is determined according to the recognized state of each finger and the predefined gesture category.

6. The method according to claim 5, characterized in that According to the three-dimensional coordinates of each key point of the hand, the state of each finger is identified, including: For any finger, construct the vector between every two adjacent key points of the finger; The curvature index is calculated based on the vectors between every two adjacent key points and the angle between every two adjacent vectors; Whether the finger state is bent or straightened is determined based on whether the calculated curvature index exceeds a preset threshold.

7. A method for displaying an item, characterized in that: include: Identifying the gesture category of the gesture currently made by the user by the gesture recognition method as described in any one of claims 1 to 6; The three-dimensional model of the displayed object is displayed according to the recognized gesture category, wherein different gesture categories can perform three-dimensional transformation on the displayed object.

8. A gesture recognition device, characterized in that: include: An acquisition unit, used for acquiring images in real time; A hand recognition unit, used to recognize a hand area in the image collected by the collection unit; A detection unit, configured to detect a three-dimensional position of each hand key point from the hand region using a pre-trained hand key point detection model; the three-dimensional position of the hand key point includes a two-dimensional coordinate and a relative depth coordinate of the hand key point; An establishing unit, used for establishing a relative distance relationship between each hand key point according to the three-dimensional position of each hand key point detected by the detecting unit, so as to obtain a hand model; A solving unit, used for constructing a depth coefficient equation according to the hand model and the three-dimensional position of each hand key point, and solving the depth coefficient equation to obtain the depth coordinate of each hand key point, thereby obtaining the three-dimensional coordinate of each hand key point; The gesture recognition unit is used to perform gesture recognition according to the obtained three-dimensional coordinates of each hand key point according to a predefined gesture category.

9. An article display device, characterized in that: The gesture recognition device according to claim 8 further comprises: a recognition unit and a display unit; a recognition unit, configured to call the gesture recognition device to recognize the gesture category of the gesture currently made by the user; The display unit is used to display the three-dimensional model of the displayed object according to the recognized gesture category, wherein different gesture categories can perform three-dimensional transformation on the displayed object.

10. A storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of the method of any one of claims 1 to 6 or the steps of the method of claim 7 are implemented.

11. A terminal, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method of any one of claims 1 to 6 are implemented, or the steps of the method of claim 7 are implemented.

12. A terminal, characterized in that: Includes the gesture recognition device as claimed in claim 8 or includes the article display device as claimed in claim 9.

13. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 6 or the steps of the method of claim 7.

Images