An image recognition method, apparatus and electronic device
By acquiring hand point clouds from depth images, using a neural network model to extract hand joint positions and grids, and matching target gestures, the problem of low accuracy in hand recognition results is solved, achieving high accuracy and high real-time gesture recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE SHANGHAI ICT CO LTD
- Filing Date
- 2021-08-23
- Publication Date
- 2026-05-19
AI Technical Summary
Existing hand recognition technologies suffer from low accuracy when recognizing 3D hand poses and shapes due to factors such as low hand resolution, cluttered backgrounds, hand interactions with other objects, hand occlusion, similar gestures, and multiple degrees of freedom.
By acquiring hand point clouds from depth images, the positions of hand joints are extracted using a pre-defined neural network model, the hand grid is determined, and the target gesture is matched. The hand recognition result is determined by combining the hand grid and the target gesture. Graph convolutional neural networks and fully connected layers are used for feature fusion, and point cloud registration algorithms are used to improve matching accuracy and real-time performance.
It effectively identifies occluded hand depth images, improves the accuracy and real-time performance of image recognition results, reduces computational load, and enhances the accuracy and real-time performance of gesture template matching.
Smart Images
Figure CN115719507B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image recognition method, apparatus and electronic device. Background Technology
[0002] More and more interactive systems need to accurately identify 3D hand poses and 3D hand shapes, but hand recognition suffers from low hand resolution, cluttered backgrounds, hand interactions with other objects, hand occlusion, similar gestures, and multiple degrees of freedom, resulting in low accuracy of hand recognition results. Summary of the Invention
[0003] This application provides an image recognition method, apparatus, and electronic device to solve the problem of low accuracy in hand recognition results.
[0004] In a first aspect, embodiments of this application provide an image recognition method, including:
[0005] Obtain the hand point cloud from the depth image;
[0006] The hand point cloud is input into a preset neural network model to obtain the hand joint positions output by the preset neural network model;
[0007] The hand mesh of the depth image is determined based on the hand joint position;
[0008] Determine the target gesture that matches the hand joint position;
[0009] Based on the hand grid and the target gesture, the hand recognition result of the depth image is determined.
[0010] Secondly, embodiments of this application also provide an image recognition device, comprising:
[0011] The first acquisition module is used to acquire the hand point cloud of the depth image;
[0012] An input module is used to input the hand point cloud into a preset neural network model to obtain the hand joint positions output by the preset neural network model;
[0013] The first determining module is used to determine the hand grid of the depth image based on the hand joint position;
[0014] The second determining module is used to determine the target gesture that matches the hand joint position;
[0015] The third determining module is used to determine the hand recognition result of the depth image based on the hand grid and the target gesture.
[0016] Thirdly, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and running on the processor, wherein the program or instructions, when executed by the processor, implement the steps in the method described in the first aspect above.
[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium for storing a program, wherein the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps of the method described in the first aspect above.
[0018] In this embodiment, based on the hand joint position output by the preset neural network model, the hand grid of the depth image and the matching target gesture are determined. Based on the hand grid and the target gesture, the hand recognition result of the depth image is determined. By locating the hand joint position, the depth map of the hand with occlusion can be effectively identified, thereby improving the accuracy of the image recognition result. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of this application, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic flowchart of the image recognition method provided in the embodiments of this application;
[0021] Figure 2 This is a schematic diagram of the structure of the gesture recognition network provided in an embodiment of this application;
[0022] Figure 3 This is a schematic diagram of the hand joint extraction network provided in an embodiment of this application;
[0023] Figure 4 This is a schematic diagram of the hand mesh recovery network provided in an embodiment of this application;
[0024] Figure 5 This is one of the schematic diagrams of the gesture recognition network output results provided in the embodiments of this application;
[0025] Figure 6 This is a second schematic diagram of the output results of the gesture recognition network provided in the embodiments of this application;
[0026] Figure 7 This is the third schematic diagram of the gesture recognition network output results provided in the embodiments of this application;
[0027] Figure 8 This is a schematic diagram of the output results of the gesture recognition prototype network provided in the embodiments of this application;
[0028] Figure 9 This is one of the structural schematic diagrams of the image recognition device provided in the embodiments of this application;
[0029] Figure 10 This is a second schematic diagram of the structure of the image recognition device provided in the embodiments of this application;
[0030] Figure 11 This is the third schematic diagram of the structure of the image recognition device provided in the embodiments of this application;
[0031] Figure 12 This is the fourth schematic diagram of the structure of the image recognition device provided in the embodiments of this application;
[0032] Figure 13 This is the fifth schematic diagram of the structure of the image recognition device provided in the embodiments of this application;
[0033] Figure 14 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0035] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.
[0036] Please see Figure 1 , Figure 1 This application provides an image recognition method, such as... Figure 1 As shown, it includes the following steps:
[0037] Step 101: Obtain the hand point cloud from the depth image.
[0038] The aforementioned depth image can be an image acquired by any image acquisition device, with the distance from the image acquisition device to each point as the pixel value. Examples include stereo cameras, TOF (Time of Flight) cameras, and laser scanning cameras. Furthermore, the depth image can be converted into a point cloud in the coordinate system of the image acquisition device using its internal parameters. The depth image can be obtained by rendering a hand model using the image acquisition device, or it can be a depth image directly input from other devices.
[0039] Step 102: Input the hand point cloud into a preset neural network model to obtain the hand joint positions output by the preset neural network model.
[0040] It is understandable that the aforementioned pre-defined neural network model can be used to extract the hand joint positions from a hand point cloud. The output hand joint positions can include the position coordinates of multiple hand joints and the type of each hand joint. For example, hand joints can be divided into two types: palm joints and finger joints. During the training of the aforementioned pre-defined neural network model, the number of hand joints varies across different datasets. Specifically, taking the hand joint model of the NYU dataset as an example, each finger includes 3 hand joints, and the palm includes 1 hand joint, for a total of 16 hand joints. Thus, the hand joint positions output by the aforementioned pre-defined neural network model include the position coordinates of these 16 hand joints and the type of each of these 16 hand joints.
[0041] Step 103: Determine the hand grid of the depth image based on the hand joint position.
[0042] Specifically, the point cloud of the hand joint positions can be input into a graph convolutional neural network to obtain the corresponding hand mesh.
[0043] Step 104: Determine the target gesture that matches the hand joint position.
[0044] In this embodiment, steps 103 and 104 are executed in no particular order, and the aforementioned hand grid and the aforementioned target gesture can be obtained simultaneously.
[0045] The target gesture can be obtained from pre-acquired gestures to be matched. For example, among multiple gestures to be matched, the gesture that matches the hand joint position is determined, and the target gesture is obtained based on that gesture.
[0046] Step 105: Based on the hand grid and the target gesture, determine the hand recognition result of the depth image.
[0047] After obtaining the hand mesh and the target gesture based on the hand joint positions, the recognition result may include at least one of hand posture and shape. For example, when it is only necessary to recognize 3D (3-Dimension) hand posture or 3D hand shape, the recognition result may correspond to a hand posture diagram or a hand mesh diagram; when it is necessary to accurately recognize 3D hand posture and 3D hand shape, the recognition result may include both hand posture diagram and hand mesh diagram. This application does not impose any limitations on this.
[0048] In this embodiment, based on the hand joint position output by the preset neural network model, the hand grid of the depth image and the matching target gesture are determined. Based on the hand grid and the target gesture, the hand recognition result of the depth image is determined. By locating the hand joint position, the depth map of the hand with occlusion can be effectively identified, thereby improving the accuracy of the image recognition result.
[0049] Optionally, the determination of the target gesture matching the hand joint position in step 104 may specifically include:
[0050] Obtain a gesture template dataset, wherein the gesture template dataset includes multiple gesture templates;
[0051] The hand joint positions are registered with the multiple gesture templates respectively to obtain a transformation matrix that corresponds one-to-one with the multiple gesture templates;
[0052] Obtain the matching scores between the hand joint positions and the multiple gesture templates;
[0053] Based on the target gesture template and the transformation matrix corresponding to the target gesture template, a target gesture that matches the hand joint position is determined, wherein the target gesture template is the gesture template with the highest matching score among the multiple gesture templates.
[0054] The aforementioned multiple gesture templates can include multiple gesture templates generated based on the same gesture. For example, when generating the gesture template dataset, multiple gesture models can be acquired, along with multiple depth images of each gesture model from different angles and the corresponding hand point clouds. Specifically, the hand model can be placed at the center of a hemisphere with a radius of 400mm. Virtual cameras are placed on this hemisphere at 10-degree intervals to render the model, generating 648 depth images. These depth images are then converted into point clouds, and 2048 points are sampled and stored in the gesture template dataset. By performing the above steps on different hand models, a complete gesture template dataset can be obtained.
[0055] The aforementioned gesture template dataset and hand joint positions can both be saved and used in the form of hand point cloud data. By comparing the hand joint positions one by one, the target gesture template with the highest matching score can be determined.
[0056] In this embodiment of the application, by obtaining the target gesture template with the highest matching score, and determining the target gesture that matches the hand joint position based on the target gesture template and the transformation matrix corresponding to the target gesture template, the possibility of successfully matching two point clouds with large differences can be minimized, thereby improving the accuracy of gesture template matching.
[0057] Optionally, obtaining the matching scores between the hand joint positions and the plurality of gesture templates may include:
[0058] Obtain the set of corresponding point pairs between the hand joint positions and the multiple gesture templates;
[0059] Obtain the distance of each corresponding point pair in the set of corresponding point pairs;
[0060] Obtain the first corresponding point pair in the set of corresponding point pairs, wherein the distance between the first corresponding point pair is less than a first preset threshold;
[0061] The matching score is determined based on the number of the first corresponding point pairs.
[0062] Each set of corresponding point pairs may include each hand joint and its corresponding point in the gesture template. For example, when the obtained hand joint position includes 16 hand joints, the distance between each hand joint and its corresponding point is obtained one by one, and the number of first corresponding point pairs whose distance is less than the first preset threshold is accumulated. The higher the proportion of the number of first corresponding point pairs to the total number of corresponding point pairs, the higher the probability that the corresponding gesture template can be successfully matched.
[0063] Existing point cloud registration algorithms are based on point-to-point ICP (Iterative Closest Point) registration algorithms, which have slow iteration speeds and are prone to getting trapped in local optima.
[0064] In this embodiment, each set of corresponding point pairs may include a pair of corresponding points between each hand joint in the hand joint position and a corresponding point in a gesture template. The first pair of corresponding point pairs with a distance less than a first preset threshold can be understood as successfully matched pairs. Thus, the more first corresponding points in each set of corresponding point pairs, the higher the probability that the gesture template corresponding to the set of corresponding point pairs matches the hand joint position accurately. Selecting the gesture template with the highest matching score as the target gesture template can minimize the possibility of successfully matching two point clouds with large differences, effectively reduce the amount of computation, and improve the real-time performance and accuracy of gesture template matching.
[0065] Optionally, before determining the hand recognition result of the depth image based on the hand grid and the target gesture in step 105, the method may further include the following steps:
[0066] Obtain the average distance between the target gesture and the hand point cloud;
[0067] Step 105, which involves determining the hand recognition result of the depth image based on the hand grid and the target gesture, may specifically include:
[0068] If the average distance is less than a second preset threshold, the hand recognition result of the depth image is determined based on the hand grid and the target gesture.
[0069] In this process, after determining the target gesture that matches the hand joint position based on the target gesture template and the transformation matrix corresponding to the target gesture template, the target gesture can be understood as a matching gesture that is registered with the hand point cloud of the depth image in the same reference system. By comparing whether the average distance between the target gesture and the hand point cloud is less than the second preset threshold, it can be verified whether the target gesture is a correct hand pose.
[0070] In this embodiment, the average distance between the target gesture and the hand point cloud is obtained, and the hand recognition result of the depth image is determined when the average distance is less than a second preset threshold, which can improve the real-time performance and accuracy of the target gesture matching.
[0071] Optionally, the preset neural network model may include multiple extraction modules and fully connected layers connected in sequence. Each extraction module includes a first convolutional layer and a pooling layer. A second convolutional layer connects the pooling layer of the first extraction module to the first convolutional layer of the second extraction module. The multiple extraction modules include the first extraction module and the second extraction module, and the second extraction module is the next level extraction module of the first extraction module.
[0072] In this embodiment, by connecting the pooling layer of the first extraction module with the first convolutional layer of the second extraction module through the second convolutional layer, the upper-layer features and lower-layer features can be fused to ensure that subtle hand features can be considered when extracting hand joints, thereby improving the accuracy of the hand joint position.
[0073] Optionally, the step 103 of determining the hand mesh of the depth image based on the hand joint position includes:
[0074] Based on the hand joint positions, determine the positional relationships between the hand joint points;
[0075] Based on the positional relationship, the number of hand joints is increased to obtain a preset number of hand grid coordinate points;
[0076] Based on the hand grid coordinate points, the hand grid of the depth image is determined.
[0077] In this embodiment, the number of hand joints is increased based on the positional relationship to obtain a preset number of hand grid coordinate points. This allows for the acquisition of a large number of hand grid coordinate points, thereby restoring the hand grid of the depth image and improving the accuracy of the hand grid.
[0078] The various optional implementation methods described in the embodiments of this application can be combined with each other or implemented individually without conflict, and the embodiments of this application do not limit this.
[0079] For ease of understanding, a specific example is as follows:
[0080] like Figure 2 As shown, this application provides a gesture recognition neural network for recognizing the shape and posture of a 3D hand based on a single depth map point cloud. The structure of the gesture recognition neural network is divided into two parts: the first part is a hand joint extraction network and a gesture template matching network, and the second part is a hand mesh recovery network.
[0081] In the first part, the input of the hand joint extraction network is a hand point cloud converted from a depth map. This depth map can be obtained by rendering a hand model by an image acquisition device. In order to take into account structural depth information, the depth image is converted into a point cloud. The hand joint extraction network outputs the spatial position information of the hand joints based on the hand point cloud.
[0082] It is understandable that hand joints can be divided into two parts: the palm and the fingers. The number of hand joints varies across different datasets. The spatial location information of the hand joints mentioned above includes the three-dimensional coordinates and type of each hand joint. This application uses the NYU dataset hand joint model, with each finger including 3 nodes and the palm using 1 joint point, thus a total of 16 hand joint points. Hand joint extraction involves predicting joint positions and classifying them, which can be considered a multi-object target detection task. Figure 3 As shown, the hand joint extraction network takes the hand point cloud converted from the hand depth map as input and outputs the hand joint positions.
[0083] Specifically, such as Figure 3 As shown, the depth map input to the hand joint extraction network is 128×128. The hand joint extraction part consists of 5 modules. Each module includes two 3×3 convolutional layers and one max pooling layer. The number of channels in the convolutional layer in the first module is 4, and the number of channels in the subsequent modules is doubled accordingly. Therefore, the number of channels in the first fully connected layer is 128, and the number of channels in the second fully connected layer is 37 (including 16 joint parameters and 21 fixed parameters).
[0084] The hand joint extraction network consists of 15 convolutional layers, 1 fully connected layer, and 5 pooling layers. A 1×1 convolutional layer connects the pooling layer to the next set of convolutional and pooling layers. This short connection structure allows for the fusion of upper and lower layer features, ensuring that subtle hand features are considered during joint extraction.
[0085] The hand joint extraction network obtains feature maps through multiple convolutional and pooling operations. These feature maps are then input into a fully connected layer to extract the spatial location information of the hand joints from the input depth map. The loss function uses L1 loss (Least Abosulote Error): the average of the Euclidean distances between the 16 joint positions and the ground truth is used as the total loss for backpropagation.
[0086] Based on the hand joint positions output by the aforementioned hand joint extraction network, gesture templates can be matched to obtain hand poses. First, a gesture template point cloud dataset needs to be generated, which can include the following steps: Place the hand model at the center of a hemisphere with a radius of 400mm. Place virtual cameras on this hemisphere at 10-degree intervals to render the model, generating 648 depth images. Convert the depth maps into point clouds and sample 2048 points, storing them in the hand template point cloud dataset. Repeat the above steps to obtain gesture template point clouds for different hand models.
[0087] Secondly, the hand joint positions obtained from the hand joint extraction network are matched with the gesture template point cloud dataset, and a matching score is output for each gesture template point cloud. The gesture template point cloud with the highest score is multiplied by the transformation matrix of the point cloud registration to obtain the 3D pose of the target gesture. Specifically, obtaining the gesture template point cloud with the highest score can include the following steps: Step a: Perform a registration between the hand joint positions and the gesture template point clouds to obtain the initial registration rotation and translation matrices; Step b: Obtain the set of corresponding point pairs of the closest points between the hand joint positions and the gesture template point clouds, and calculate the distance of each closest point pair. If the distance of the closest point pair is less than a threshold distance, the closest point pair is deleted; otherwise, the iteration terminates; Step c: Repeat step b, and record the number of deleted closest point pairs as N. del The point cloud matching score is:
[0088]
[0089] Where, N del This represents the number of nearest pairs that were deleted, and N represents the total number of nearest pairs.
[0090] The rotation matrix of the initial registration multiplied by the rotation matrix of the gesture template point cloud with the highest matching score is the rotation matrix of the final point cloud registration. The rotation matrix of the initial registration multiplied by the midpoint of the point cloud of the real target hand plus the translation matrix of the initial point cloud registration is the translation matrix of the final point cloud registration. The midpoint of the point cloud of the real target hand is the midpoint of each hand joint point in the above-mentioned hand joint positions.
[0091] By using the percentage of the logarithm of the nearest deleted points relative to the total number of point clouds as the point cloud matching score, the probability of successfully matching two significantly different point clouds can be minimized, effectively reducing computational load and improving the real-time performance and accuracy of gesture template matching. This application uses ADD (Average Distance) scoring to verify whether the template with the highest score is in the correct pose. An ADD score less than 1cm is considered a correct hand pose estimation, where:
[0092]
[0093] Among them, R est The estimated value of the target gesture rotation matrix is obtained by multiplying the rotation matrix obtained from point cloud registration by the rotation matrix of the gesture template point cloud; t est R represents the estimated translation matrix of the target gesture; M represents the coordinates of the hand joints; gt The true value of the rotation matrix representing the target gesture; t gt This represents the truth value of the translation matrix of the target gesture, with the gesture template provided by the NYU dataset.
[0094] like Figure 4As shown, the second part is a hand mesh recovery network with a linear graph convolutional neural network, which can recover the hand mesh from the three-dimensional hand joint positions.
[0095] The input to the hand mesh reconstruction network is the 3D position coordinates of the hand joints. The positional relationship between the joints is obtained through fully connected layers and graph convolution. Then, the number of joints is increased by 4 times through upsampling. After 3 upsampling, 2368 hand mesh coordinate points are finally obtained.
[0096] This application also provides an experimental verification method for a gesture recognition neural network. By comparing with other methods, the effectiveness of the gesture recognition neural network in this application is verified. The experimental verification of the gesture recognition neural network includes two parts: the first part uses a synthetic dataset for verification, and the second part uses a real dataset for verification. During the verification process, three evaluation criteria are used: average 3D joint location error (JLE), 3D vertex location error (VLE), and the percentage of images within a specific threshold.
[0097] Validation using synthetic datasets specifically includes the following process:
[0098] The SynHand5M dataset was chosen to train and test the gesture recognition neural network. To verify the effectiveness of the proposed algorithm for occluded gesture estimation, ground truth grid vertices in the SynHand5M dataset were randomly removed. The original SynHand5M dataset contained 1193 vertices and 22 3D hand joints; after random removal, the number of vertices was reduced to 800 and the number of 3D hand joints to 16. 80% of the synthetic depth images were used for training, and 20% were used for testing. To coarsen the graph, the number of nodes in the graph convolutional neural network layers was set to 98, 392, and 1568. Furthermore, state-of-the-art methods were trained on the SynHand5M dataset for comparison.
[0099] During training of the gesture recognition neural network, the batch size was set to 64, and the Adam optimizer with an initial learning rate of 0.001 was used. The learning rate was halved every 10 generations. Training was performed using a single Nvidia GeForce GTX 1080Ti, and the results showed a generation rate of over 30Hz for 3D hand joints and 3D hand shapes. The gesture recognition neural network recognition results of this application are as follows: Figure 5As shown, the first column is the hand point cloud obtained from the depth image, the second column is the ground truth of the hand 3D mesh, the third column is one of the predicted values of the hand 3D mesh, the fourth column is another of the predicted values of the hand 3D mesh, the fifth column is the ground truth of the hand 3D gesture, and the sixth column is the predicted value of the hand 3D gesture. Figure 6 The diagram illustrates the results of the gesture recognition neural network in this application compared to those of other existing recognition networks. Figure 6 As shown, the hand shape recovered by the gesture recognition neural network of this application is similar to the true hand shape, and the identified joint positions are close to the true joint values. Furthermore, by... Figure 6 It can be seen that the gesture recognition neural network of this application outperforms various existing hand pose and shape recognition methods in terms of 3D vertex position error, and has a smaller average 3D joint position error on the SynHand5M dataset.
[0100] Validation using real datasets specifically includes the following processes:
[0101] The gesture recognition neural network of this application was trained and tested using three real-world datasets: the NYU dataset, the MSRA dataset, and the ICVL dataset.
[0102] The NYU dataset contains 72,757 training data points and 8,252 test data points. Each data point contains a depth image and a color image, along with 16 annotated joint locations. Typically, 14 joints are selected for comparison. To obtain hand segmentation from the depth image, the centroid of the hand is pre-calculated to remove the background.
[0103] MSRA contains 76,375 data points, each with 21 annotated joints. The dataset contains hand segmentation.
[0104] The ICVL dataset contains 330,866 training data points and 1,596 test data points, with background removal using simple depth thresholding segmentation.
[0105] Since all real-world datasets lack ground truth values, three quantitative results from the MSRA, ICVL, and NYU datasets are provided for pose comparison. The 3D joint position error results on these three datasets are as follows: Figure 7 As shown, Figure 7 The diagram shows the results of the gesture recognition network of this application compared with other existing recognition networks. It can be seen that the gesture recognition neural network of this application is superior to other existing gesture recognition networks.
[0106] This application also provides a gesture recognition prototype system, comprising hardware and software components. The hardware includes an Intel RealSense D415 camera and camera stand, as well as a desktop computer. The D415 camera acquires depth images of the hand, the camera stand adjusts the camera pose, and the desktop computer runs the software to recognize hand poses and reconstruct the hand mesh. The software embeds the gesture recognition neural network proposed in this application into a graphical interface, enabling one-click recognition of hand poses and reconstruction of the hand mesh. The graphical interface is written in Python using Tkinter and includes five functions: camera pose adjustment, image zooming, screenshotting, displaying hand joints, and reconstructing the hand mesh. The software acquires gestures and reconstructs hand models as follows: Figure 8 As shown, hand pose and hand shape can be recognized from the input depth image. With an Nvidia Geforce GTX 1080 Ti, the gesture recognition speed can reach 30fps, demonstrating high real-time performance and accuracy. The reconstructed hand model also exhibits high precision.
[0107] To verify the practicality of the gesture recognition prototype system, it was applied to the secondary sorting of industrial parts. For parts that are difficult to identify, secondary sorting is required, which necessitates manual input of the type of object to be sorted to instruct the robotic arm to grasp it. Gesture recognition can reduce the number of times workers switch between the production line and the input screen, effectively improving the efficiency of human-machine interaction.
[0108] Specifically, parts requiring secondary sorting are placed in a designated area with a specified posture, directly below the robotic arm's gripper. The user then places their right hand in front of an Intel RealSense D415 camera and inputs the gesture corresponding to the part type. The software determines the relative position of the gesture's joints and acquires the gesture information. The robotic arm then grasps the corresponding parts based on different gestures. The results show that the gesture recognition software can correctly identify different gestures and has practical value.
[0109] Please see Figure 9 , Figure 9 This is one of the structural schematic diagrams of the image recognition device provided in the embodiments of this application, such as... Figure 9 As shown, the image recognition device 900 includes:
[0110] The first acquisition module 901 is used to acquire the hand point cloud of the depth image;
[0111] The input module 902 is used to input the hand point cloud into a preset neural network model to obtain the hand joint position output by the preset neural network model;
[0112] The first determining module 903 is used to determine the hand grid of the depth image based on the hand joint position;
[0113] The second determining module 904 is used to determine the target gesture that matches the hand joint position;
[0114] The third determining module 905 is used to determine the hand recognition result of the depth image based on the hand grid and the target gesture.
[0115] Optional, such as Figure 10 As shown, the second determining module 904 may specifically include:
[0116] The first acquisition unit 9041 is used to acquire a gesture template dataset, wherein the gesture template dataset includes multiple gesture templates;
[0117] The registration unit 9042 is used to register the hand joint positions with the multiple gesture templates respectively to obtain a transformation matrix corresponding to each of the multiple gesture templates.
[0118] The second acquisition unit 9043 is used to acquire the matching scores of the hand joint positions with the multiple gesture templates respectively;
[0119] The first determining unit 9044 is used to determine a target gesture that matches the hand joint position based on the target gesture template and the transformation matrix corresponding to the target gesture template, wherein the target gesture template is the gesture template with the highest matching score among the plurality of gesture templates.
[0120] Optional, such as Figure 11 As shown, the second acquisition unit 9043 may specifically include:
[0121] The first acquisition subunit 90431 is used to acquire the set of corresponding point pairs between the hand joint positions and the multiple gesture templates;
[0122] The second acquisition subunit 90432 is used to acquire the distance of each corresponding point pair in the corresponding point pair set;
[0123] The third acquisition subunit 90433 is used to acquire the first corresponding point pair in the corresponding point pair set, wherein the distance between the first corresponding point pair is less than a first preset threshold.
[0124] Subunit 90434 is defined for determining the matching score based on the number of the first corresponding point pairs.
[0125] Optional, such as Figure 12 As shown, the image recognition device 900 also includes:
[0126] The second acquisition module 906 is used to acquire the average distance between the target gesture and the hand point cloud;
[0127] The determining module 905 may specifically include:
[0128] The second determining unit 9051 is used to determine the hand recognition result of the depth image based on the hand grid and the target gesture if the average distance is less than a second preset threshold.
[0129] Optionally, the preset neural network model includes multiple extraction modules and fully connected layers connected in sequence. Each extraction module includes a first convolutional layer and a pooling layer. A second convolutional layer connects the pooling layer of the first extraction module to the first convolutional layer of the second extraction module. The multiple extraction modules include the first extraction module and the second extraction module, and the second extraction module is the next level extraction module of the first extraction module.
[0130] Optional, such as Figure 13 As shown, the first determining module 903 may specifically include:
[0131] The third determining unit 9031 is used to determine the positional relationship between hand joint points based on the hand joint position;
[0132] The third acquisition unit 9032 is used to expand the number of hand joints based on the positional relationship to obtain a preset number of hand grid coordinate points;
[0133] The fourth determining unit 9033 is used to determine the hand grid of the depth image based on the hand grid coordinate points.
[0134] The image recognition device 900 can realize the embodiments of this application. Figure 1 The various processes in the method embodiments, and the ways to achieve the same beneficial effects, will not be repeated here to avoid repetition.
[0135] This application also provides an electronic device. Please refer to [link to relevant documentation]. Figure 14 The electronic device 1400 includes a processor 1401, a memory 1402, and a program or instructions stored in the memory 1402 and executable on the processor 1401. When the program or instructions are executed by the processor 1401, they implement the various processes of the above-described image recognition method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0136] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.
[0137] This application also provides a computer-readable storage medium for storing a program. The computer-readable storage medium stores a computer program, which, when executed by a processor, can perform the above-described functions. Figure 1 Any step in the corresponding method embodiment can achieve the same technical effect, and will not be repeated here to avoid repetition.
[0138] The storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0139] The above description represents the preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An image recognition method, characterized in that, include: Obtain the hand point cloud from the depth image; The hand point cloud is input into a preset neural network model to obtain the hand joint positions output by the preset neural network model; The hand grid of the depth image is determined based on the hand joint position; Obtain a gesture template dataset, wherein the gesture template dataset includes multiple gesture templates; The hand joint positions are registered with the multiple gesture templates respectively to obtain a transformation matrix that corresponds one-to-one with the multiple gesture templates; Obtain the set of corresponding point pairs between the hand joint positions and the multiple gesture templates; Obtain the distance of each corresponding point pair in the set of corresponding point pairs; Obtain the first corresponding point pair in the set of corresponding point pairs, wherein the distance between the first corresponding point pair is less than a first preset threshold; The matching scores of the hand joint positions with the multiple gesture templates are determined based on the number of the first corresponding point pairs; Based on the target gesture template and the transformation matrix corresponding to the target gesture template, a target gesture that matches the hand joint position is determined, wherein the target gesture template is the gesture template with the highest matching score among the multiple gesture templates; Based on the hand grid and the target gesture, the hand recognition result of the depth image is determined.
2. The method as described in claim 1, characterized in that, Before determining the hand recognition result of the depth image based on the hand grid and the target gesture, the method further includes: Obtain the average distance between the target gesture and the hand point cloud; The step of determining the hand recognition result of the depth image based on the hand grid and the target gesture includes: If the average distance is less than a second preset threshold, the hand recognition result of the depth image is determined based on the hand grid and the target gesture.
3. The method as described in claim 1, characterized in that, The preset neural network model includes multiple extraction modules and fully connected layers connected in sequence. Each extraction module includes a first convolutional layer and a pooling layer. The pooling layer of the first extraction module is connected to the first convolutional layer of the second extraction module by a second convolutional layer. The multiple extraction modules include the first extraction module and the second extraction module, and the second extraction module is the next level extraction module of the first extraction module.
4. The method as described in claim 1, characterized in that, The process of determining the hand mesh of the depth image based on the hand joint position includes: Based on the hand joint positions, determine the positional relationships between the hand joint points; Based on the positional relationship, the number of hand joints is increased to obtain a preset number of hand grid coordinate points; Based on the hand grid coordinates, the hand grid of the depth image is determined.
5. An image recognition device, characterized in that, include: The first acquisition module is used to acquire the hand point cloud of the depth image; An input module is used to input the hand point cloud into a preset neural network model to obtain the hand joint positions output by the preset neural network model; The first determining module is used to determine the hand grid of the depth image based on the hand joint position; The second determining module is used to determine the target gesture that matches the hand joint position; The third determining module is used to determine the hand recognition result of the depth image based on the hand grid and the target gesture; The second determining module includes: The first acquisition unit is used to acquire a gesture template dataset, wherein the gesture template dataset includes multiple gesture templates; The registration unit is used to register the hand joint positions with the multiple gesture templates respectively, so as to obtain a transformation matrix corresponding to each of the multiple gesture templates. The second acquisition unit is used to acquire the matching scores of the hand joint positions with the multiple gesture templates respectively; The determining unit is used to determine a target gesture that matches the hand joint position based on the target gesture template and the transformation matrix corresponding to the target gesture template, wherein the target gesture template is the gesture template with the highest matching score among the plurality of gesture templates; The second acquisition unit includes: The first acquisition subunit is used to acquire the set of corresponding point pairs between the hand joint positions and the plurality of gesture templates; The second acquisition subunit is used to acquire the distance of each corresponding point pair in the corresponding point pair set; The third acquisition subunit is used to acquire the first corresponding point pair in the corresponding point pair set, wherein the distance between the first corresponding point pair is less than a first preset threshold. A subunit is determined for determining the matching score based on the number of the first corresponding point pairs.
6. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and running on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the image recognition method as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image recognition method as described in any one of claims 1 to 4.