Model training method, palm posture estimation method, electronic device, and storage medium

Abstract

Embodiments of the present application relate to the field of image recognition, and disclose a model training method, a palm posture estimation method, an electronic device and a storage medium. The model training method comprises: constructing a feature extraction model for generating a feature map of an image sample pair; wherein the image sample pair comprises a palm color image and a palm infrared image; matching the feature map of the palm color image and the feature map of the palm infrared image to determine a feature point pair of homonymous points; constructing a loss function based on the distance between the disparity value obtained based on the feature point pair and the reference disparity value obtained by monocular matching algorithm of the palm infrared image, and training the feature extraction model. When applied to a palm posture estimation process, the palm posture estimation accuracy is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of image recognition, and in particular to a model training method, a palm posture estimation method, an electronic device, and a storage medium. BACKGROUND

[0002] Palm posture estimation is an important part of palmprint and palm vein recognition systems. The detection results obtained inform the user to make corresponding adjustments to the palm posture, so as to better obtain palm images and thus ensure the recognition accuracy of palmprints and palm veins. Palm posture estimation can be divided into two categories according to the final generation method: a detection-based method and a regression-based method. The detection-based method generates a heat map to obtain predicted palm key points, and determines the palm posture according to the predicted palm key points. The regression-based method directly regresses the position coordinates of the key points, and obtains the three-dimensional posture of the palm according to the position coordinates of the key points.

[0003] However, the regression-based method lacks spatial generalization ability and loses spatial information, resulting in inaccurate posture estimation results. The detection-based method mostly uses a large amount of labeled hand data and a complex deep learning model to obtain accurate palm postures. However, the collection of labeled hand data requires a large amount of time and manpower, and the complex deep learning model is not conducive to transplantation to the embedded platform of the palm brushing system. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a model training method, a palm posture estimation method, an electronic device, and a storage medium. By combining the model training method and the feature detection matching method, the feature map obtained by the feature extraction model with a relatively simple structure is more accurate, thereby improving the accuracy of determining the palm posture according to the feature points of the feature map.

[0005] To solve the above technical problems, the embodiments of the present application provide a model training method, which comprises: constructing a feature extraction model for generating a feature map of an image sample pair; wherein the image sample pair comprises a palm color image and a palm infrared image; matching the feature map of the palm color image and the feature map of the palm infrared image to determine a feature point pair of homonymous points; constructing a loss function based on the distance between the parallax value obtained based on the feature point pair and the reference parallax value obtained by monocular matching algorithm of the palm infrared image, and training the feature extraction model.

[0006] The embodiments of this application also provide a hand pose estimation method, comprising: inputting a color image of the hand to be tested and an infrared image of the hand to be tested, obtained by capturing the same hand, into a trained feature extraction model to obtain feature maps of the color image of the hand to be tested and the infrared image of the hand to be tested; wherein the feature extraction model is obtained by the model training method described above; matching the feature maps of the color image of the hand to be tested and the feature maps of the infrared image of the hand to be tested to determine feature point pairs with corresponding points, and obtaining three-dimensional point cloud data of the feature points of the infrared image of the hand to be tested in the feature point pairs; calculating the covariance matrix of the three-dimensional point cloud data, and using the eigenvalue decomposition method to determine the eigenvalues ​​of the covariance matrix and the eigenvectors corresponding to the eigenvalues; combining the eigenvector corresponding to the maximum value of the eigenvalues, the eigenvalue corresponding to the second largest value, and the eigenvector corresponding to the third largest value as a hand pose rotation matrix, and determining the three-dimensional pose of the hand based on the hand pose rotation matrix.

[0007] Embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the model training method mentioned in the above embodiments, or to perform the hand pose estimation method mentioned in the above embodiments.

[0008] The embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the model training method mentioned in the above embodiments, or is capable of executing the hand pose estimation method mentioned in the above embodiments.

[0009] The model training method provided in this application inputs a color image and an infrared image of the palm into a feature extraction model to obtain feature maps of the color image and the infrared image of the palm. It calculates and determines feature point pairs that are identical in the feature maps of the color image and the infrared image of the palm. The distance between the disparity value obtained from the feature point pair and the disparity value obtained from the palm infrared image using a monocular matching algorithm is used as the loss function to calculate the feature extraction model. Since the entire model is only responsible for feature map extraction, the model architecture is relatively simple and does not require a large amount of labeled data for training. The model training is constrained by the distance between the disparity value of the feature point pair calculated by feature detection matching and the disparity value obtained from the palm infrared image using a monocular matching algorithm. In other words, this application combines image processing methods (feature detection matching) and deep learning methods (model training), making the feature maps of the color image and the infrared image of the palm obtained by the feature extraction model more accurate, thereby improving the accuracy of subsequent hand pose determination based on feature points in the feature maps.

[0010] The hand pose estimation method provided in this application extracts feature maps from a color image and an infrared image of the hand to be tested using a feature extraction model obtained through model training. These two feature maps are matched to determine feature point pairs with corresponding points. Three-dimensional point cloud data is then determined from these feature point pairs. Finally, the three-dimensional pose of the hand is determined based on the eigenvalues ​​of the covariance matrix of the three-dimensional point cloud data and the corresponding eigenvectors. The entire method only requires image processing algorithms to obtain accurate three-dimensional hand pose, making it simple, convenient, and highly portable, significantly improving the practicality of the hand pose estimation module in palmprint and palm vein recognition systems.

[0011] Furthermore, the model training method provided in this application matches the feature map of the hand color image and the feature map of the hand infrared image to determine feature point pairs that are identically named. This includes: calculating the feature vector magnitude of each pixel in the feature map of the hand color image and the feature map of the hand infrared image; using a preset search window, determining the maximum value of the feature vector magnitude within the search window in both the hand color image and the hand infrared image, and using the pixel corresponding to the maximum value as a feature point; and using a preset feature matching algorithm to match the feature points in the feature map of the hand color image and the feature points in the feature map of the hand infrared image to determine feature point pairs that are identically named. This application uses the pixel corresponding to the maximum value of the feature vector magnitude within the search window as the feature point, that is, selecting pixels with strong features based on the feature vector magnitude as pixels for subsequent calculation of the hand's three-dimensional pose. This avoids the problems of high computational load, excessive redundant information between pixels, and low efficiency when calculating the hand's three-dimensional pose using all pixels.

[0012] Furthermore, the model training method provided in this application, which involves constructing a loss function based on the distance between the disparity value obtained from the feature point pair and the reference disparity value obtained from the palm infrared image using a monocular matching algorithm, and then training the feature extraction model, includes: projecting the disparity map obtained from the palm infrared image using a monocular matching algorithm onto the infrared camera coordinate system, and performing downsampling processing to obtain a disparity map with the same size as the feature map in the infrared camera coordinate system; determining the disparity value of the pixel corresponding to each feature point in the feature map of the palm infrared image in the disparity map, and using the disparity value as a reference disparity value; and constructing a loss function based on the distance between the disparity value obtained from the feature point pair and the reference disparity value, and then training the feature extraction model. The feature map obtained by the feature extraction model in this application is generally smaller than the size of the original image (palm infrared image and palm color image). Therefore, by downsampling the disparity map obtained from the palm infrared image using a monocular matching algorithm to obtain a disparity map with the same size as the feature map, the corresponding reference disparity value can be quickly determined based on the coordinates of the feature points when constructing the loss function.

[0013] Furthermore, the hand pose estimation method provided in this application, wherein acquiring the three-dimensional point cloud data of feature points in the infrared image of the hand to be tested by aligning the feature points includes: obtaining the depth values ​​of the feature points in the infrared image of the hand to be tested by aligning the feature points using a depth calculation formula; and calculating the three-dimensional point cloud data of the feature points in the infrared image of the hand to be tested in the infrared camera coordinate system based on the image coordinates of the feature points, the depth values, and preset camera intrinsic parameters. This application obtains the three-dimensional point cloud data of the feature points by aligning the feature points in the infrared image of the hand to be tested by aligning the feature points, uses this data as hand feature points, and performs simple mathematical calculations on the three-dimensional point cloud data of these hand feature points to determine the three-dimensional pose of the hand. Attached Figure Description

[0014] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0015] Figure 1 This is a flowchart of the model training method provided in the embodiments of this application;

[0016] Figure 2 This is a flowchart of the hand pose estimation method provided in the embodiments of this application;

[0017] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of this application to enable the reader to better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0019] The following provides an example illustrating the implementation details of model training in this embodiment. The following implementation details are provided for ease of understanding only and are not essential for implementing this solution.

[0020] The embodiments of this application relate to a model training method, such as... Figure 1 As shown, it includes:

[0021] Step 101: Construct a feature extraction model for generating feature maps of image sample pairs; wherein the image sample pairs include a color image of a hand and an infrared image of a hand.

[0022] In this embodiment, the image sample pairs used for training comprise several sets. Each set of image sample pairs includes one color image and one infrared image of the palm, and both images are taken from the same palm. For example, there are 10 sets of image sample pairs used for training. These 10 sets of image sample pairs can be images of the same palm taken from different shooting positions, or images of different palms taken from the same shooting position, but each set of image sample pairs includes a color image and an infrared image of the same palm. Generally, a color camera and an infrared camera are used to simultaneously acquire color and infrared images of the palm, with the color camera and the infrared camera having different shooting viewpoints.

[0023] Step 102: Match the feature map of the color image of the palm with the feature map of the infrared image of the palm to determine feature point pairs that are identical.

[0024] In this embodiment, feature detection and feature matching are performed on the feature maps of the acquired color image and infrared image of the palm to determine feature point pairs that are identical in name. That is, a feature point pair includes two pixels with strong features. It is important to note that in the subsequent palm pose estimation process, this application determines the 3D pose of the palm based on pixels with strong features. On the one hand, not all pixels are helpful in determining the 3D pose of the palm; some pixel information may be unnecessary. On the other hand, there is redundant information between adjacent pixels. If all pixels in the palm image are used for palm pose estimation calculations, the computational load of the entire process will increase significantly, which is detrimental to improving the efficiency of palm pose estimation.

[0025] In one embodiment, step 102 specifically includes: calculating the feature vector magnitude of each pixel in the feature map of the color image of the palm and the feature map of the infrared image of the palm respectively; determining the maximum value point of the feature vector magnitude in the feature map of the color image of the palm and the feature map of the infrared image of the palm respectively using a preset search window, and taking the pixel point corresponding to the maximum value point as the feature point; and using a preset feature matching algorithm to match the feature points in the feature map of the color image of the palm and the feature points in the feature map of the infrared image of the palm to determine feature point pairs that are identical in name.

[0026] In this embodiment, the modulus of the feature vector of each pixel is calculated for the obtained feature map. The dimension of the feature vector of each pixel is related to the number of channels of the model. For example, if the feature extraction model has 1024 channels, then the feature vector of each pixel is a 1024-dimensional feature vector. Then, using a preset search window, the maximum value of the feature vector modulus within the feature map of the hand color image and the feature map of the hand infrared image is determined. Assuming the search window size is 3*3, the magnitudes of the nine feature vectors within the search window are compared, and the pixel corresponding to the maximum value of the feature vector modulus is taken as the feature point. Specifically, the size of the search window and the search step size can be set according to the accuracy of hand pose estimation and the needs of calculation speed. The preset image matching algorithm can be the Fast Nearest Neighbor (FLANN) algorithm, SIFT algorithm, SURF algorithm, PCA-SIFT algorithm, etc. Specifically, during the matching process, two matching pixels can be determined based on Euclidean distance, Mahalanobis distance, etc.

[0027] In addition, to improve the accuracy of matching calculations, the two cameras can be calibrated before acquiring image sample pairs to determine their intrinsic and extrinsic parameters. Based on these parameters, the Fusiello method is used to perform epipolar correction on the infrared and color images of the palm, achieving strict row alignment. This means the epipolar lines of the two images are exactly on the same horizontal line, and corresponding points (feature matching points) in the two images have the same row number. Specifically, the camera calibration process includes: placing a checkerboard calibration board in front of the infrared and color cameras, ensuring the calibration board is simultaneously and completely within the field of view of both cameras; and simultaneously acquiring several sets of color and infrared images of the calibration board. First, the intrinsic parameters of the two cameras are calibrated separately. Specifically, this involves extracting the coordinates of the calibration board corner points, estimating the initial intrinsic parameters, and using the least squares method to iteratively optimize the reprojection error. The obtained intrinsic parameters include: camera focal length, principal point coordinates, distortion coefficients, etc. Then, the intrinsic parameters of the two cameras are fixed, the initial extrinsic parameters are estimated, and the least squares method is used to iteratively optimize the reprojection error to obtain the extrinsic parameters of the two cameras, including rotation matrix, translation matrix, and corrected intrinsic parameter matrix.

[0028] Step 103: Construct a loss function based on the distance between the disparity value obtained from the feature point pair and the reference disparity value obtained from the palm infrared image through a monocular matching algorithm, and train the feature extraction model.

[0029] In this embodiment, the reference disparity value obtained from the hand infrared image using a monocular matching algorithm is used as the true value in the loss function, and the disparity value obtained from the feature pair is used as the predicted disparity value. It should be noted that the reference disparity value is obtained by adding a laser projector to one side of the infrared camera to construct a monocular structured light system, and then performing a monocular matching algorithm on the hand infrared image. It is understood that the laser projector mainly emits infrared laser light; therefore, the hand speckle image obtained by the infrared camera will be clearer than the hand speckle image obtained by the monocular structured light system composed of RGB cameras. Consequently, the disparity value determined by the monocular matching algorithm for the hand infrared speckle image and the reference infrared speckle image has higher accuracy and smaller error. Based on this, during model training, the disparity value calculated from the cross-modal feature point pairs (feature points from the feature map of the hand color image and the feature map of the hand infrared image) is made as close as possible to the high-precision reference disparity value, improving the disparity calculation accuracy of cross-modal images.

[0030] In one embodiment, step 103 specifically includes: projecting the disparity map obtained from the palm infrared image using a monocular matching algorithm onto the infrared camera coordinate system, and performing downsampling processing to obtain a disparity map with the same size as the feature map in the infrared camera coordinate system; determining the disparity value of each feature point in the feature map of the palm infrared image in the corresponding pixel point in the disparity map, and using the disparity value as a reference disparity value; constructing a loss function based on the distance between the disparity value obtained from the feature point pair and the reference disparity value, and training the feature extraction model.

[0031] In this embodiment, since the feature map obtained by the feature extraction model is generally smaller than the original image size, while the disparity map obtained from the hand infrared image using a monocular matching algorithm has the same size as the original image, it is necessary to downsample the disparity map during loss function calculation to obtain a disparity map with the same size as the feature map. In this way, the corresponding pixels in the disparity map can be determined based on the coordinates of each feature point in the feature map of the hand infrared image. That is, a correspondence is established between the feature points of the hand color image, the feature points of the hand infrared image, and the corresponding pixels in the disparity map.

[0032] Specifically, the loss function used by the feature extraction model in this embodiment during training is:

[0033]

[0034]

[0035] Where N is the number of feature point pairs. It is the reference disparity value obtained from the infrared image of the palm using a monocular matching algorithm. The x-coordinate of a feature point in the feature map of the infrared image of the palm. It is the x-coordinate of the feature point of the feature map of the color image of the palm.

[0036] It should be noted that the feature extraction model in this application can adopt the DenseNet structure of a dense convolutional network, or other network structures, such as residual networks, attention mechanism networks, etc. If the DenseNet structure is adopted, in order to better utilize the feature information of each convolutional layer, the mean pooling layer in the DenseNet-transition module can be modified to a max pooling layer to further highlight the feature response. The final output feature map resolution of the network is 1 / 4 of the original image, and the number of channels is 1024. The feature extraction model can be trained in two stages. In the first stage, pre-training is performed using an open-source dataset to obtain highly generalized network model parameters, giving the network an initial feature extraction capability. In the second stage, transfer training is performed on self-constructed image sample pairs. Specifically, the second stage training process can use the RMSProp optimizer with an initial learning rate of 0.001. During model training, the learning rate is dynamically adjusted, decreasing stepwise with the increase of the number of iterations.

[0037] The model training method provided in this application inputs a color image and an infrared image of the palm into a feature extraction model to obtain feature maps of the color image and the infrared image of the palm. It calculates and determines feature point pairs that are identical in the feature maps of the color image and the infrared image of the palm. The distance between the disparity value obtained from the feature point pair and the disparity value obtained from the palm infrared image using a monocular matching algorithm is used as the loss function to calculate the feature extraction model. Since the entire model is only responsible for feature map extraction, the model architecture is relatively simple and does not require a large amount of labeled data for training. The model training is constrained by the distance between the disparity value of the feature point pair calculated by feature detection matching and the disparity value obtained from the palm infrared image using a monocular matching algorithm. In other words, this application combines image processing methods (feature detection matching) and deep learning methods (model training), making the feature maps of the color image and the infrared image of the palm obtained by the feature extraction model more accurate, thereby improving the accuracy of subsequent hand pose determination based on feature points in the feature maps.

[0038] The embodiments of this application relate to a hand pose estimation method, such as... Figure 2 As shown, it includes:

[0039] Step 201: Input the color image and infrared image of the palm to be tested, obtained by capturing the same palm, into the trained feature extraction model to obtain the feature map of the color image and the feature map of the infrared image of the palm to be tested; wherein the feature extraction model is obtained by the model training method described above.

[0040] In this embodiment, the network structure, training method, and loss function of the feature extraction model can be referred to the specific implementation details of steps 101-103.

[0041] Step 202: Match the feature map of the color image of the palm to be tested with the feature map of the infrared image of the palm to be tested, determine the feature point pairs with the same name, and obtain the three-dimensional point cloud data of the feature points of the infrared image of the palm to be tested in the feature point pairs.

[0042] In this embodiment, the method for determining feature point pairs is the same as that for determining feature point pairs during the model training phase, and will not be described in detail here. Obtaining the 3D point cloud data of the feature points of the infrared image of the hand to be tested in the feature point pair includes: obtaining the depth values ​​of the feature points of the infrared image of the hand to be tested in the feature point pair using a depth calculation formula; and calculating the 3D point cloud data of the feature points of the infrared image of the hand to be tested in the infrared camera coordinate system based on the image coordinates of the feature points of the infrared image of the hand to be tested, the depth values, and preset camera intrinsic parameters.

[0043] Specifically, the depth calculation formula is as follows:

[0044]

[0045] Among them, u ir and u rgb , respectively, are the pixel coordinates of feature points in the infrared image and color image of the palm to be tested, f is the focal length pixel value in the intrinsic parameter matrix, and L is the baseline length in the intrinsic parameter matrix.

[0046] Specifically, the formula for calculating 3D point cloud data is:

[0047]

[0048] Where (X, Y, Z) are the three-dimensional coordinates of the feature point, (u ir v ir (c) represents the image coordinates of the feature points in the infrared image of the palm to be tested. x c y ) represents the principal point coordinates in the intrinsic parameter matrix, f represents the focal length pixel value in the intrinsic parameter matrix, and d represents the depth value corresponding to the feature point.

[0049] Step 203: Calculate the covariance matrix of the 3D point cloud data, and use the eigenvalue decomposition method to determine the eigenvalues ​​of the covariance matrix and the corresponding eigenvectors.

[0050] In this embodiment, the calculation process of the covariance matrix includes: decentralizing the 3D point cloud data, that is, subtracting the average value of the coordinates of the 3D point cloud data from the coordinate value of each point, and then calculating the covariance matrix on the decentralized 3D point cloud data. Assuming there are m n-dimensional data points, the original data are arranged into an n-row m-column matrix X. The formula for calculating the covariance is Cov = (1 / m) * X. T The eigenvalues ​​and corresponding eigenvectors of the calculated covariance matrix are determined using eigenvalue decomposition. Specifically, the number of eigenvalues ​​is equal to the degree of the characteristic equation.

[0051] Step 204: Combine the eigenvector corresponding to the maximum eigenvalue, the eigenvalue corresponding to the second largest eigenvalue, and the eigenvector corresponding to the third largest eigenvalue as the hand posture rotation matrix, and determine the three-dimensional posture of the hand based on the hand posture rotation matrix.

[0052] In this embodiment, the obtained eigenvalues ​​are arranged in descending order. The eigenvector corresponding to the largest value, the eigenvector corresponding to the second largest value, and the eigenvector corresponding to the third largest value are combined to form the hand posture rotation matrix. The three-dimensional posture of the hand is then determined based on the hand posture rotation matrix. Assuming a coordinate system is established with the center of the hand as the origin, the direction of the hand's forward movement is the x-axis, the direction perpendicular to the hand is the z-axis, and the direction perpendicular to both the x-axis and z-axis is the y-axis. Based on this, the eigenvector corresponding to the largest eigenvalue is in the x-axis direction, the eigenvector corresponding to the second largest value is in the y-axis direction, and the eigenvector corresponding to the third largest value is in the z-axis direction.

[0053] Specifically, the formula for calculating the three-dimensional pose of the hand is:

[0054]

[0055]

[0056] Where R is the hand pose rotation matrix, (θ x θ y θ z ( ) represents the three-dimensional pose of the hand.

[0057] In addition, before acquiring the 3D point cloud data of the feature points of the infrared image of the hand to be tested in the feature point pair, the method further includes: calculating the difference in the abscissa of the two feature points in the feature point pair; and removing feature point pairs whose abscissa difference exceeds a preset threshold from the determined feature point pairs. It should be noted that due to the network generalization problem, there may be mismatches. Therefore, after completing the feature matching, it is necessary to further filter the selected corresponding point pairs based on the epipolar constraint principle, and remove corresponding point pairs whose abscissa difference exceeds the threshold, so as to improve the matching accuracy of corresponding point pairs.

[0058] The hand pose estimation method provided in this application extracts feature maps from a color image and an infrared image of the hand to be tested using a feature extraction model obtained through model training. These two feature maps are matched to determine feature point pairs with corresponding points. Three-dimensional point cloud data is then determined from these feature point pairs. Finally, the three-dimensional pose of the hand is determined based on the eigenvalues ​​of the covariance matrix of the three-dimensional point cloud data and the corresponding eigenvectors. The entire method only requires image processing algorithms to obtain accurate three-dimensional hand pose, making it simple, convenient, and highly portable, significantly improving the practicality of the hand pose estimation module in palmprint and palm vein recognition systems.

[0059] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0060] The embodiments of this application relate to an electronic device, such as... Figure 3 As shown, it includes:

[0061] At least one processor 301; and a memory 302 communicatively connected to at least one processor 301; wherein the memory 302 stores instructions executable by at least one processor 301, the instructions being executed by at least one processor 301 to enable at least one processor 301 to perform the model training method mentioned in the above embodiments, or to perform the hand pose estimation method mentioned in the above embodiments.

[0062] The electronic device includes: one or more processors 301 and a memory 302. Figure 3 Taking a processor 301 as an example, the processor 301 and the memory 302 can be connected via a bus or other means. Figure 3Taking a bus connection as an example, memory 302, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. For example, in this embodiment, the algorithms corresponding to each processing strategy in the strategy space are stored in memory 302. Processor 301 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 302, thereby implementing the above-mentioned model training method or hand pose estimation method.

[0063] Memory 302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 302 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 302 may optionally include memory remotely located relative to processor 301, and these remote memories can be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0064] One or more modules are stored in memory 302. When executed by one or more processors 301, they can perform the model training method in any of the above embodiments, or can perform the hand pose estimation method mentioned in the above embodiments.

[0065] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0066] Embodiments of this application relate to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the above-described method embodiments.

[0067] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0068] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application.

Claims

1. A model training method, characterized in that, include: A feature extraction model is constructed to generate feature maps of image sample pairs; wherein the image sample pairs include a color image of a palm and an infrared image of a palm. The feature maps of the color image of the palm and the infrared image of the palm are matched to determine feature point pairs that are identical to each other. A loss function is constructed based on the distance between the disparity value obtained from the feature point pair and the reference disparity value obtained from the palm infrared image by a monocular matching algorithm, and the feature extraction model is trained accordingly. The step of matching the feature map of the color image of the palm and the feature map of the infrared image of the palm to determine feature point pairs that are identical in name includes: Calculate the feature vector magnitude of each pixel in the feature map of the color image of the palm and the feature map of the infrared image of the palm, respectively; Using a preset search window, determine the maximum value of the feature vector magnitude within the feature map of the color image of the palm and the feature map of the infrared image of the palm, respectively, and use the pixel corresponding to the maximum value as the feature point; A preset feature matching algorithm is used to match feature points in the feature map of the color image of the palm and feature points in the feature map of the infrared image of the palm to determine feature point pairs that are identical in name.

2. The model training method according to claim 1, characterized in that, The loss function is constructed based on the distance between the disparity value obtained from the feature point pair and the reference disparity value obtained from the palm infrared image through a monocular matching algorithm, and the feature extraction model is trained by the following steps: The disparity map obtained by the hand infrared image through a monocular matching algorithm is projected onto the infrared camera coordinate system, and downsampling is performed to obtain a disparity map with the same size as the feature map in the infrared camera coordinate system. Determine the disparity value of each feature point in the feature map of the palm infrared image and the corresponding pixel point in the disparity map, and use the disparity value as a reference disparity value; A loss function is constructed based on the distance between the disparity value obtained from the feature point pair and the reference disparity value, and the feature extraction model is trained.

3. The model training method according to claim 1, characterized in that, The loss function is expressed by the following formula: Build: Where N is the number of feature point pairs. It is the reference disparity value obtained from the infrared image of the palm using a monocular matching algorithm. The x-coordinate of a feature point in the feature map of the infrared image of the palm. It is the x-coordinate of the feature point of the feature map of the color image of the palm.

4. A method for estimating hand pose, characterized in that, include: The color image and infrared image of the palm to be tested, obtained by photographing the same palm, are input into a pre-trained feature extraction model to obtain feature maps of the color image and the infrared image of the palm to be tested; wherein the feature extraction model is obtained by the model training method as described in any one of claims 1 to 3 above. The feature map of the color image of the palm to be tested and the feature map of the infrared image of the palm to be tested are matched to determine the feature point pairs with the same name, and the three-dimensional point cloud data of the feature points of the infrared image of the palm to be tested in the feature point pairs are obtained. Calculate the covariance matrix of the 3D point cloud data, and use eigenvalue decomposition to determine the eigenvalues ​​of the covariance matrix and the eigenvectors corresponding to the eigenvalues; The eigenvector corresponding to the maximum value, the eigenvalue corresponding to the second largest value, and the eigenvector corresponding to the third largest value are combined to form the hand posture rotation matrix, and the three-dimensional posture of the hand is determined based on the hand posture rotation matrix.

5. The hand pose estimation method according to claim 4, characterized in that, The acquisition of the three-dimensional point cloud data of the feature points of the infrared image of the palm to be tested in the feature point pair includes: The depth values ​​of the feature points in the infrared image of the palm to be tested are obtained using a depth calculation formula; Based on the image coordinates of the feature points of the infrared image of the palm to be tested, the depth value, and the preset camera intrinsic parameters, the three-dimensional point cloud data of the feature points of the infrared image of the palm to be tested in the infrared camera coordinate system is calculated.

6. The hand pose estimation method according to claim 4, characterized in that, The three-dimensional pose of the hand is determined by the following formula: Where R is the hand pose rotation matrix, ( , , ( ) represents the three-dimensional pose of the hand.

7. The hand pose estimation method according to claim 4, characterized in that, Before acquiring the three-dimensional point cloud data of the feature points of the infrared image of the palm to be tested in the feature point pair, the method further includes: Calculate the difference in abscissa between the two feature points in the feature point pair; Feature point pairs whose abscissa difference exceeds a preset threshold are removed from the determined feature point pairs.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the model training method as described in any one of claims 1 to 3, or to perform the hand pose estimation method as described in any one of claims 4 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method as described in any one of claims 1 to 3, or is capable of performing the hand pose estimation method as described in any one of claims 4 to 7.

Images