Keypoint localization model training method, keypoint localization method and electronic equipment

Abstract

This application provides a keypoint localization model training method, a keypoint localization method, and an electronic device, relating to the field of deep learning technology. The method includes: inputting a first sample image containing a specified object into a first keypoint localization model to be trained, obtaining predicted coordinates representing keypoints of the specified object in the first sample image; obtaining the current prediction weight matrix of a specified network layer in the first keypoint localization model to be trained; for every two keypoints, calculating a prediction correlation coefficient between the two keypoints based on the columns corresponding to the two keypoints in the prediction weight matrix; calculating a first loss value based on the prediction correlation coefficient and the true correlation coefficient; and adjusting the model parameters based on the first loss value and the second loss value until a preset convergence condition is met, thus obtaining a trained first keypoint localization model. This allows for the acquisition of a keypoint localization model with higher accuracy, thereby improving the accuracy of the localization results.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a key point localization model training method, a key point localization method, and an electronic device. Background Technology

[0002] In the field of deep learning technology, a trained keypoint localization model can be used to locate the keypoints of a specified object in an image, that is, to obtain the coordinates of each keypoint in the image. For example, common keypoint localization models include: face keypoint localization models, finger keypoint localization models, and human pose keypoint localization models, etc.

[0003] However, during the training of the keypoint localization model for the initial structure, the specified objects in the sample images used are often in an unspecified state. This results in low accuracy of the localization results when the model locates keypoints in images containing specified objects in specified states. For example, in the training scenario of a face keypoint localization model, an unspecified state indicates that both eyes are either open or closed simultaneously, while a specified state indicates that one eye is open and the other eye is closed.

[0004] Therefore, how to obtain a more accurate key point localization model to improve the accuracy of localization results has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a keypoint localization model training method, a keypoint localization method, and an electronic device, so as to obtain a keypoint localization model with higher accuracy and improve the accuracy of localization results. The specific technical solution is as follows:

[0006] A first aspect of this application provides a key point localization model training method, the method comprising:

[0007] The first sample image containing the specified object is input into the first keypoint localization model to be trained, and the coordinates of the keypoints of the specified object in the first sample image are obtained as the predicted coordinates.

[0008] Obtain the current weight matrix of the specified network layer in the first keypoint localization model to be trained, as the prediction weight matrix; wherein, any column in the prediction weight matrix represents: the feature of the horizontal coordinate of a keypoint, or the feature of the vertical coordinate of a keypoint;

[0009] For each pair of key points, the correlation coefficient between the two key points is calculated based on the columns corresponding to the two key points in the prediction weight matrix, and is used as the prediction correlation coefficient.

[0010] Based on the predicted correlation coefficient and the true correlation coefficient, a first loss value is calculated; wherein, the true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image;

[0011] Based on the first loss value and the second loss value, the model parameters of the first keypoint localization model to be trained are adjusted until the preset convergence condition is met, and the trained first keypoint localization model is obtained; wherein, the second loss value is determined based on the predicted coordinates and the true coordinates of the keypoints of the specified object in the first sample image.

[0012] In some embodiments, the predictive correlation coefficient between any two key points includes: the predictive correlation coefficient between the horizontal coordinates of the two key points, and / or the predictive correlation coefficient between the vertical coordinates of the two key points.

[0013] In some embodiments, the prediction correlation coefficient between the horizontal coordinates of any two key points is obtained by processing the columns corresponding to the horizontal coordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0014] The predicted correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0015] In some embodiments, the true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image, combined with the physiological constraints of the parts to which the key points of the specified object belong.

[0016] In some embodiments, before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient, the method further includes:

[0017] Obtain the second sample image;

[0018] For each key point, the mean of the horizontal coordinate and the mean of the vertical coordinate in the true coordinates of the key point in each second sample image are calculated and used as the template coordinates of the key point.

[0019] For each pair of key points, the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points and the physiological constraints of the parts to which the two key points belong in the specified object.

[0020] In some embodiments, calculating the true correlation coefficient between two key points for each pair of key points, based on the distance between the template coordinates of the two key points and the physiological constraint relationship of the parts to which the two key points belong in the specified object, includes:

[0021] For every two key points, if the parts to which the two key points belong in the specified object do not have a physiological constraint relationship, then the true correlation coefficient between the two key points is determined to be 0.

[0022] If the parts to which the two key points belong in the specified object are physiologically constrained, and the distance between the template coordinates of the two key points is not less than the preset distance, then the true correlation coefficient between the two key points is determined to be 0.

[0023] If the parts to which the two key points belong in the specified object are physiologically constrained, and the distance between the template coordinates of the two key points is less than a preset distance, then the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points; wherein, the calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

[0024] In some embodiments, calculating the true correlation coefficient between the two key points based on the distance between their template coordinates includes:

[0025] According to the first preset formula, the first correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points; wherein, the first preset formula indicates that there is a physiological constraint relationship between the first part of the part to which the two key points belong and the second part of the part to which they belong.

[0026] The first preset formula is:

[0027] R1=1-d×Z1

[0028] R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight.

[0029] Based on the first correlation coefficient, the true correlation coefficient between the two key points is calculated.

[0030] In some embodiments, before calculating the true correlation coefficient between the two key points based on the first correlation coefficient, the method further includes:

[0031] According to the second preset formula, the second correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points; wherein, the second preset formula indicates that the second part and the first part have a physiological constraint relationship;

[0032] The second preset formula is:

[0033] R² = 1 - d × Z²

[0034] R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight.

[0035] The step of calculating the true correlation coefficient between the two key points based on the first correlation coefficient includes:

[0036] Calculate the average of the first correlation coefficient and the second correlation coefficient to obtain the true correlation coefficient between the two key points.

[0037] In some embodiments, obtaining the second sample image includes:

[0038] Obtain the third sample image;

[0039] Each third sample image is mapped to a unified image template to obtain a second sample image; wherein the distance between preset parts of a specified object in each second sample image is less than a preset threshold.

[0040] In some embodiments, before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient, the method further includes:

[0041] Obtain the current weight matrix of a specified network layer in the second keypoint localization model as the true weight matrix; wherein, the second keypoint localization model is trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: the feature of the horizontal coordinate of a keypoint, or the feature of the vertical coordinate of a keypoint;

[0042] For each pair of key points, the correlation coefficient between the two key points is calculated based on the columns corresponding to the two key points in the true weight matrix, and this coefficient is used as the true correlation coefficient.

[0043] In some embodiments, the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points, and / or the true correlation coefficient between the vertical coordinates of the two key points.

[0044] In some embodiments, the true correlation coefficient between the x-coordinates of any two key points is obtained by processing the columns corresponding to the x-coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm.

[0045] The true correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm.

[0046] In some embodiments, calculating the first loss value based on the predicted correlation coefficient and the actual correlation coefficient includes:

[0047] If the true correlation coefficient between any two key points includes both the true correlation coefficient between their x-coordinates and their y-coordinates, and the predicted correlation coefficient between any two key points includes both the predicted correlation coefficient between their x-coordinates and their y-coordinates, then a first loss value is calculated based on the first sub-loss value and the second sub-loss value. The first sub-loss value represents the difference between the predicted correlation coefficient between the x-coordinates and the true correlation coefficient between the x-coordinates; the second sub-loss value represents the difference between the predicted correlation coefficient between the y-coordinates and the true correlation coefficient between the y-coordinates.

[0048] If the true correlation coefficient between any two key points is the same as the true correlation coefficient between the x-coordinates of the two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the x-coordinates of the two key points, then the first sub-loss value is obtained as the first loss value.

[0049] If the true correlation coefficient between any two key points is the same as the true correlation coefficient between the ordinates of the two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the ordinates of the two key points, then the second sub-loss value is obtained as the first loss value.

[0050] In some embodiments, the key points of the specified object include at least one of the following: facial key points, finger key points, and human posture key points.

[0051] A second aspect of this application provides a key point localization method, the method comprising:

[0052] Get the image to be detected that contains the specified object;

[0053] The image to be detected is input into a trained third keypoint localization model to obtain the coordinates of key points of a specified object in the image to be detected; wherein, the third keypoint localization model is trained according to any of the above keypoint localization model training methods.

[0054] A third aspect of this application provides a key point localization model training apparatus, the apparatus comprising:

[0055] The prediction coordinate acquisition module is used to input a first sample image containing a specified object into a first keypoint localization model to be trained, and obtain the coordinates of the keypoints of the specified object in the first sample image as the prediction coordinates.

[0056] The prediction weight matrix acquisition module is used to acquire the current weight matrix of a specified network layer in the first keypoint localization model to be trained, as the prediction weight matrix; wherein, any column in the prediction weight matrix represents: the feature of the horizontal coordinate of a keypoint, or the feature of the vertical coordinate of a keypoint.

[0057] The prediction correlation coefficient acquisition module is used to calculate the correlation coefficient between two key points based on the columns corresponding to the two key points in the prediction weight matrix for each pair of key points, and use it as the prediction correlation coefficient.

[0058] The first loss value acquisition module is used to calculate a first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image;

[0059] The model parameter adjustment module is used to adjust the model parameters of the first keypoint localization model to be trained based on the first loss value and the second loss value until a preset convergence condition is reached to obtain the trained first keypoint localization model; wherein, the second loss value is determined based on the predicted coordinates and the true coordinates of the keypoints of the specified object in the first sample image.

[0060] In some embodiments, the predictive correlation coefficient between any two key points includes: the predictive correlation coefficient between the horizontal coordinates of the two key points, and / or the predictive correlation coefficient between the vertical coordinates of the two key points.

[0061] In some embodiments, the predicted correlation coefficient between the abscissas of any two key points is obtained by processing the columns corresponding to the abscissas of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm; the predicted correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0062] In some embodiments, the true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image, combined with the physiological constraints of the parts to which the key points of the specified object belong.

[0063] In some embodiments, the apparatus further includes: a second sample image acquisition module, configured to acquire the second sample image before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient; a template coordinate acquisition module, configured to calculate, for each key point, the mean of the horizontal coordinate and the mean of the vertical coordinate in the true coordinates of the key point in each second sample image, as the template coordinates of the key point; and a first true correlation coefficient acquisition module, configured to calculate, for every two key points, the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object.

[0064] In some embodiments, the first true correlation coefficient acquisition module includes: a first condition determination submodule, configured to determine the true correlation coefficient between two key points as 0 if, for every two key points, the parts to which the two key points belong in the specified object do not have a physiological constraint relationship; a second condition determination submodule, configured to determine the true correlation coefficient between two key points as 0 if, in the specified object, the parts to which the two key points belong have a physiological constraint relationship, and the distance between the template coordinates of the two key points is not less than a preset distance; and a third condition determination submodule, configured to calculate the true correlation coefficient between two key points based on the distance between the template coordinates of the two key points if, in the specified object, the parts to which the two key points belong have a physiological constraint relationship, and the distance between the template coordinates of the two key points is less than a preset distance; wherein the calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

[0065] In some embodiments, the third condition determination submodule includes: a first correlation coefficient calculation unit, configured to calculate a first correlation coefficient between the two key points using the distance between the template coordinates of the two key points according to a first preset formula; wherein, the first preset formula indicates that the first part of the part to which the two key points belong and the second part of the part to which they belong have a physiological constraint relationship; the first preset formula is: R1 = 1 - d × Z1, where R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight; and a true correlation coefficient calculation unit, configured to calculate the true correlation coefficient between the two key points based on the first correlation coefficient.

[0066] In some embodiments, the device further includes: a second correlation coefficient calculation module, configured to calculate a second correlation coefficient between the two key points according to a second preset formula, using the distance between the template coordinates of the two key points, before calculating the true correlation coefficient between the two key points based on the first correlation coefficient; wherein, the second preset formula indicates that the second part and the first part have a physiological constraint relationship; the second preset formula is: R2 = 1 - d × Z2, where R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight; the true correlation coefficient calculation unit is specifically configured to: calculate the average of the first correlation coefficient and the second correlation coefficient to obtain the true correlation coefficient between the two key points.

[0067] In some embodiments, the second sample image acquisition module includes: a third sample image acquisition submodule for acquiring a third sample image; and a mapping submodule for mapping each third sample image to a unified image template to obtain a second sample image; wherein the distance between preset parts of a specified object in each second sample image is less than a preset threshold.

[0068] In some embodiments, the apparatus further includes: a true weight matrix acquisition module, configured to acquire the current weight matrix of a specified network layer in the second keypoint localization model as the true weight matrix before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the second keypoint localization model is trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: the feature of the horizontal coordinate of a keypoint, or the feature of the vertical coordinate of a keypoint; and a second true correlation coefficient acquisition module, configured to calculate the correlation coefficient between two keypoints based on the columns corresponding to the two keypoints in the true weight matrix for each pair of keypoints, as the true correlation coefficient.

[0069] In some embodiments, the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points, and / or the true correlation coefficient between the vertical coordinates of the two key points.

[0070] In some embodiments, the true correlation coefficient between the x-coordinates of any two key points is obtained by processing the columns corresponding to the x-coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm; the true correlation coefficient between the y-coordinates of any two key points is obtained by processing the columns corresponding to the y-coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm.

[0071] In some embodiments, the first loss value acquisition module is specifically configured to: if the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, then calculate a first loss value based on a first sub-loss value and a second sub-loss value; wherein, the first sub-loss value represents: the difference between the predicted correlation coefficient between the horizontal coordinates and the true correlation coefficient between the horizontal coordinates; the second sub-loss value... This indicates the difference between the predicted correlation coefficient and the actual correlation coefficient between the vertical axes. If the actual correlation coefficient between any two key points is the same as the actual correlation coefficient between the horizontal axes of those two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the horizontal axes of those two key points, then the first sub-loss value is obtained as the first loss value. If the actual correlation coefficient between any two key points is the same as the actual correlation coefficient between the vertical axes of those two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the vertical axes of those two key points, then the second sub-loss value is obtained as the first loss value.

[0072] In some embodiments, the key points of the specified object include at least one of the following: facial key points, finger key points, and human posture key points.

[0073] A fourth aspect of the embodiments of this application provides a key point localization device, the device comprising:

[0074] The image acquisition module is used to acquire an image containing a specified object.

[0075] The localization module is used to input the image to be detected into a trained third keypoint localization model to obtain the coordinates of key points of a specified object in the image to be detected; wherein, the third keypoint localization model is trained according to any of the above keypoint localization model training methods.

[0076] A fifth aspect of this application provides an electronic device, including:

[0077] Memory, used to store computer programs;

[0078] When a processor executes a program stored in memory, it implements the keypoint localization model training method described in the first aspect above, or the keypoint localization method described in the second aspect.

[0079] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the key point localization model training method described in the first aspect, or the key point localization method described in the second aspect.

[0080] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the key point localization model training method described in the first aspect, or the key point localization method described in the second aspect.

[0081] Beneficial effects of the embodiments in this application:

[0082] This application provides a keypoint localization model training method, comprising: inputting a first sample image containing a specified object into a first keypoint localization model to be trained, obtaining coordinates representing keypoints of the specified object in the first sample image, as predicted coordinates; obtaining the current weight matrix of a specified network layer in the first keypoint localization model to be trained, as a predicted weight matrix; wherein any column in the predicted weight matrix represents: the feature of the horizontal coordinate of a keypoint, or the feature of the vertical coordinate of a keypoint; for every two keypoints, calculating the correlation coefficient between the two keypoints based on the columns corresponding to the two keypoints in the predicted weight matrix, as a predicted correlation coefficient; calculating a first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the true correlation coefficient is determined based on the true coordinates of the keypoints of the specified object in a second sample image; adjusting the model parameters of the first keypoint localization model to be trained based on the first loss value and the second loss value until a preset convergence condition is reached, thereby obtaining a trained first keypoint localization model; wherein the second loss value is determined based on the predicted coordinates and the true coordinates of the keypoints of the specified object in the first sample image.

[0083] Based on the above processing, during the training of the keypoint localization model, the current weight matrix (i.e., the predicted weight matrix) of a specified network layer in the keypoint localization model can be obtained. Since a column in the weight matrix can represent the features of the horizontal coordinate or the vertical coordinate of a keypoint, the predicted correlation coefficient obtained based on the predicted weight matrix for any two keypoints of a specified object can reflect the correlation between the two keypoints currently learned by the model. Furthermore, since the true correlation coefficient is determined based on the true coordinates of the keypoints of the specified object in the second sample image, the true correlation coefficient can reflect the true correlation between the two keypoints. Therefore, the first loss value can reflect the difference between the correlation between the two keypoints currently learned by the model and the true correlation between the two keypoints.

[0084] Furthermore, since the second loss value is determined based on the predicted coordinates and the true coordinates of key points of the specified object in the first sample image, adjusting the model parameters of the first keypoint localization model to be trained based on the first and second loss values ​​allows the trained model to obtain the coordinates of each key point of the specified object based on the true correlation between key points. This avoids deviations in the localization results of the specified object due to incorrect learned correlations between key points. Thus, a more accurate keypoint localization model can be obtained, improving the accuracy of the localization results.

[0085] For example, in a scenario involving facial landmark localization, the actual correlation between a facial landmark belonging to the left eye (referred to as the first landmark) and another facial landmark belonging to the right eye (referred to as the second landmark) might not be high. However, since the sample images used to train the facial landmark localization model typically contain objects in an unspecified state—for example, in an unspecified state, the first and second landmarks are nearly symmetrical—the correlation between these two landmarks learned by the facial landmark localization model might be high. Accordingly, adjusting the model parameters of the facial landmark localization model based on the first and second loss values ​​can enable the trained facial landmark localization model to learn the actual correlation between these two landmarks.

[0086] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0087] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0088] Figure 1 This is a first flowchart of a key point localization model training method provided in an embodiment of this application;

[0089] Figure 2(a) is a schematic diagram of facial key points provided in an embodiment of this application;

[0090] Figure 2(b) is another schematic diagram of facial key points provided in an embodiment of this application;

[0091] Figure 3 A schematic diagram illustrating the localization of 68 facial key points provided in an embodiment of this application;

[0092] Figure 4A first flowchart for obtaining the true correlation coefficient provided in this application embodiment;

[0093] Figure 5 A second flowchart for obtaining the true correlation coefficient is provided for embodiments of this application;

[0094] Figure 6 A third flowchart for obtaining the true correlation coefficient provided in this application embodiment;

[0095] Figure 7 A flowchart illustrating the training of a keypoint localization model provided in this application embodiment;

[0096] Figure 8 This is a first flowchart of a key point localization method provided in an embodiment of the present invention;

[0097] Figure 9 A flowchart for obtaining facial key points in a face sample image provided in this application embodiment;

[0098] Figure 10 A structural diagram of a key point localization model training device provided in an embodiment of this application;

[0099] Figure 11 A structural diagram of a key point positioning device provided in an embodiment of this application;

[0100] Figure 12 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0101] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0102] In the field of deep learning technology, a trained keypoint localization model can be used to locate the keypoints of a specified object in an image, that is, to obtain the coordinates of each keypoint in the image. For example, common keypoint localization models include: face keypoint localization models, finger keypoint localization models, and human pose keypoint localization models, etc.

[0103] However, during the training of the keypoint localization model for the initial structure, the specified objects in the sample images used are often in an unspecified state. This results in low accuracy of the localization results when the model locates keypoints in images containing specified objects in specified states. For example, in the training scenario of a face keypoint localization model, an unspecified state indicates that both eyes are either open or closed simultaneously, while a specified state indicates that one eye is open and the other eye is closed.

[0104] Therefore, how to obtain a highly accurate key point localization model has become an urgent problem to be solved.

[0105] This application provides a keypoint localization model training method, which can be applied to electronic devices. See also... Figure 1 , Figure 1 A first flowchart of a keypoint localization model training method provided in this application embodiment, the method includes the following steps:

[0106] S101: Input the first sample image containing the specified object into the first keypoint localization model to be trained, and obtain the coordinates of the keypoints of the specified object in the first sample image as the predicted coordinates.

[0107] S102: Obtain the current weight matrix of the specified network layer in the first keypoint localization model to be trained, and use it as the prediction weight matrix.

[0108] In the prediction weight matrix, any column represents either the x-coordinate feature of a key point or the y-coordinate feature of a key point.

[0109] S103: For every two key points, calculate the correlation coefficient between the two key points based on the columns corresponding to the two key points in the prediction weight matrix, and use it as the prediction correlation coefficient.

[0110] S104: Calculate the first loss value based on the predicted correlation coefficient and the actual correlation coefficient.

[0111] The true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image.

[0112] S105: Based on the first loss value and the second loss value, adjust the model parameters of the first keypoint localization model to be trained until the preset convergence condition is met, and obtain the trained first keypoint localization model.

[0113] The second loss value is determined based on the predicted coordinates and the true coordinates of the key points of the specified object in the first sample image.

[0114] Based on the above processing, during the training of the keypoint localization model, the current weight matrix (i.e., the predicted weight matrix) of a specified network layer in the keypoint localization model can be obtained. Since a column in the weight matrix can represent the features of the horizontal coordinate or the vertical coordinate of a keypoint, the predicted correlation coefficient obtained based on the predicted weight matrix for any two keypoints of a specified object can reflect the correlation between the two keypoints currently learned by the model. Furthermore, since the true correlation coefficient is determined based on the true coordinates of the keypoints of the specified object in the second sample image, the true correlation coefficient can reflect the true correlation between the two keypoints. Therefore, the first loss value can reflect the difference between the correlation between the two keypoints currently learned by the model and the true correlation between the two keypoints.

[0115] Furthermore, since the second loss value is determined based on the predicted coordinates and the true coordinates of key points of the specified object in the first sample image, adjusting the model parameters of the first keypoint localization model to be trained based on the first and second loss values ​​allows the trained model to obtain the coordinates of each key point of the specified object based on the true correlation between key points. This avoids deviations in the localization results of the specified object due to incorrect learned correlations between key points. Thus, a more accurate keypoint localization model can be obtained, improving the accuracy of the localization results.

[0116] For example, in a scenario involving facial landmark localization, the actual correlation between a facial landmark belonging to the left eye (referred to as the first landmark) and another facial landmark belonging to the right eye (referred to as the second landmark) might not be high. However, since the sample images used to train the facial landmark localization model typically contain objects in an unspecified state—for example, in an unspecified state, the first and second landmarks are nearly symmetrical—the correlation between these two landmarks learned by the facial landmark localization model might be high. Accordingly, adjusting the model parameters of the facial landmark localization model based on the first and second loss values ​​can enable the trained facial landmark localization model to learn the actual correlation between these two landmarks.

[0117] In related technologies, a keypoint localization model can be trained by acquiring a large number of sample images containing a specified object in a specified state and a large number of sample images containing a specified object in a non-specified state. However, existing sample sets typically only contain specified objects in non-specified states. Therefore, this approach requires significant annotation costs to annotate the sample images containing specified objects in specified states, thus increasing training costs. In contrast, the keypoint localization model training method provided in this application can constrain the model using true correlation coefficients. Therefore, it is not necessary to use a first sample image containing specified objects in specified states for training, yet the trained first keypoint localization model can still learn the true correlation between keypoints. Furthermore, if a second sample image contains specified objects in non-specified states, the true correlation between keypoints can be determined based on the true coordinates of the keypoints in the second sample image. Therefore, it is also not necessary to use sample images containing specified objects in specified states when obtaining the true correlation coefficient. This avoids incurring significant annotation costs and reduces training costs.

[0118] In addition, related technologies can also acquire a small number of sample images of a specified object in a specified state and sample images of a specified object in a non-specified state to train a keypoint localization model. However, due to the small number of sample images, the accuracy of the localization result obtained by the trained keypoint localization model is low when localizing sample images containing specified objects in non-specified states. In contrast, the keypoint localization model training method provided in this application can adjust the model parameters based on the second loss value determined by the predicted coordinates and true coordinates of the keypoints of the specified object in the sample images during the training process, thus ensuring the accuracy of the obtained localization result.

[0119] The keypoint localization model in this application can be a facial keypoint localization model, and correspondingly, the specified object represents a face. In this case, when the face is in a specified state, it can represent that the face is making an exaggerated expression. For example, when one eye of the face is closed and the other eye is open, the face is in the specified state.

[0120] Keypoint localization models can also be used for finger keypoint localization, where the specified object represents the human hand. In this case, when the human hand is in a specified state, it can represent the fingers making exaggerated gestures. For example, when using fingers to simulate animal shapes, the hand is in the specified state.

[0121] Keypoint localization models can also be used for human pose keypoint localization models, where a specified object represents the human body. Correspondingly, when the human body is in a specified state, it can represent exaggerated postures of various parts of the body. For example, when an actor performs a difficult dance move, the human body is in a specified state.

[0122] In this application, the number of key points of the specified object is preset according to actual needs. For ease of description, each key point of the specified object corresponds to a number. For example, the key points can be numbered starting from 1 in ascending order, and the largest number is the number of key points of the specified object.

[0123] Correspondingly, based on the physical meaning represented by each keypoint, the location of each keypoint within a specified object can be determined. For example, for a facial keypoint localization model, where the specified object is the face, the specified object's parts can be categorized as: facial contour, left eyebrow, right eyebrow, nose, left eye, right eye, and mouth, etc. For a finger keypoint localization model, where the specified object is the hand, the specified object's parts can be categorized as: thumb, index finger, middle finger, ring finger, and little finger, etc. For a human posture keypoint localization model, where the specified object is the human body, the specified object's parts can be categorized as: head, limbs, and torso, etc.

[0124] Referring to Figures 2(a) and 2(b), Figure 2(a) is a schematic diagram of facial key points provided in an embodiment of this application. Figure 2(b) is another schematic diagram of facial key points provided in an embodiment of this application. For an image of a specified object containing one eye open and the other eye closed, the points in Figure 2(a) represent facial key points of the image obtained based on the first key point localization model trained in this application, and the points in Figure 2(b) represent facial key points of the image obtained based on the first key point localization model to be trained in this application. It can be seen that the facial key points of the image obtained based on the first key point localization model trained in this application indicate that one eye of the face is open and the other eye is closed, which is consistent with the position of the actual facial key points in the image. The facial key points of the image obtained based on the first key point localization model to be trained in this application indicate that both eyes of the face are open, which is different from the position of the actual facial key points in the image.

[0125] like Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the localization of 68 facial landmarks as provided in an embodiment of this application. Based on Figure 3The method shown for setting facial keypoints can identify 68 keypoints (i.e., facial keypoints) in an image containing a face. These points are numbered 1-68, with 1-17 representing points on the facial contour, 18-22 representing points on the left eyebrow, 23-27 representing points on the right eyebrow, 28-36 representing points on the nose, 37-42 representing points on the left eye, 43-48 representing points on the right eye, and 49-68 representing points on the mouth.

[0126] Regarding step S101, the first keypoint localization model to be trained can be an untrained keypoint localization model or a pre-trained keypoint localization model. For example, a pre-trained keypoint localization model can be one trained based on sample images containing a specified object and corresponding sample labels, where the sample labels represent the coordinates of keypoints of the specified object in the corresponding sample images.

[0127] Understandably, for any keypoint of a specified object in a sample image, the coordinates of that keypoint include: the x-coordinate of the keypoint, which is the value of the keypoint mapped onto the horizontal axis (which can be called the X-axis) of the sample image, and the y-coordinate of the keypoint, which is the value of the keypoint mapped onto the vertical axis (which can be called the Y-axis) of the sample image. Accordingly, the coordinates of a keypoint can be represented as (x, y).

[0128] The first sample image is a sample image containing the specified object. Correspondingly, the sample label corresponding to the first sample image can also be obtained. The sample label corresponding to the first sample image represents the true coordinates of the key points of the specified object.

[0129] A first sample image containing a specified object is input into a first keypoint localization model to be trained. The first keypoint localization model can predict the coordinates (i.e., predicted coordinates) of keypoints representing the specified object in the first sample image. For example, in the above... Figure 3 In the scenario shown, the first sample image is a sample image containing a face, and the first key point localization model is a face key point localization model. The coordinates of 68 face key points can be obtained by following step S101.

[0130] Regarding step S102, the specified network layer can be the output layer of the first keypoint localization model to be trained, or it can be any other network layer, as long as its weight matrix can reflect the characteristics of the horizontal and vertical coordinates of each keypoint. All elements in the weight matrix represent the weights of the specified network layer. In this application, the keypoint localization model is a coordinate regression-based model, meaning that the keypoint localization model typically uses a fully connected layer as the output layer.

[0131] Since the coordinates of a keypoint include both its x-coordinate and y-coordinate, and each column of the output layer's weight matrix corresponds to either the x-coordinate or y-coordinate of a keypoint, if the number of keypoints for a given object is N, then the output dimension of the output layer is 2N. Furthermore, since the output dimension of the output layer is the same as the number of columns in its weight matrix, the weight matrix has 2N columns. Accordingly, any column in the weight matrix represents either the x-coordinate feature of a keypoint or the y-coordinate feature of a keypoint.

[0132] Furthermore, the number of rows in the weight matrix of the output layer is consistent with the output dimension of the previous network layer. For ease of description, the number of rows in the weight matrix of the output layer can be represented by M. That is, the weight matrix of the output layer is an M×2N matrix, and any column in the weight matrix is ​​an M×1 column vector.

[0133] It is understandable that during the training process of the keypoint localization model, S101-S105 above represent a process of adjusting the model parameters. During training using sample images, the model parameters of the keypoint localization model are continuously adjusted; that is, the weight matrix of a specified network layer of the keypoint localization model may also change. Therefore, the current weight matrix (i.e., the predicted weight matrix) of the specified network layer can be obtained, and the process of adjusting the model parameters can be completed based on the obtained current weight matrix.

[0134] For step S103, for any key point, the column corresponding to the horizontal coordinate of the key point and the column corresponding to the vertical coordinate of the key point can be obtained in the prediction weight matrix.

[0135] In some embodiments, the predictive correlation coefficient between any two key points includes: the predictive correlation coefficient between the horizontal coordinates of the two key points, and / or the predictive correlation coefficient between the vertical coordinates of the two key points.

[0136] In one implementation, for every two key points, the predictive correlation coefficient between the x-coordinates of the two key points can be calculated, and this coefficient can be used as the predictive correlation coefficient between the two key points. Furthermore, the predictive correlation coefficient between every two key points can be obtained.

[0137] Correspondingly, the predicted correlation coefficient between the x-coordinates of any two key points is obtained by processing the columns corresponding to the x-coordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0138] In the embodiments of this application, the preset correlation coefficient algorithm can be: linear correlation coefficient algorithm, Kendall correlation coefficient algorithm, and Spearman correlation coefficient algorithm, etc.

[0139] For each pair of key points, when calculating the predicted correlation coefficient between the x-coordinates of the two key points, the columns corresponding to the x-coordinates of each key point can be obtained from the prediction weight matrix. Then, based on a preset correlation coefficient algorithm, the columns corresponding to the obtained x-coordinates are processed to obtain the predicted correlation coefficient between the x-coordinates of the two key points.

[0140] For example, in the process of calculating the predicted correlation coefficient between the x-coordinates of facial key point 1 and facial key point 2, the column corresponding to the x-coordinate of facial key point 1 (which can be represented as C1) and the column corresponding to the x-coordinate of facial key point 2 (which can be represented as C2) can be obtained from the predicted weight matrix. In this case, the preset correlation coefficient algorithm is the linear correlation coefficient algorithm.

[0141] Correspondingly, the prediction correlation coefficient R between the abscissas of facial landmark 1 and facial landmark 2 can be calculated using formula (1):

[0142]

[0143] Among them, C1 i This represents the i-th element in column (C1) corresponding to the x-coordinate of facial landmark 1. C1 represents the mean of all elements in C1 (i.e., the mean of C1), and C2 represents the mean of all elements in C1. i This represents the i-th element in column (C2) corresponding to the x-coordinate of facial landmark 2. M represents the mean of all elements in C2 (i.e., the mean of C2), and M represents the number of rows in the weight matrix of the output layer.

[0144] In this way, the correlation between two key points can be measured based on the calculated predictive correlation coefficient between them.

[0145] If the predictive correlation coefficient between two key points only includes the predictive correlation coefficient between the x-coordinates of the two key points, a predictive correlation matrix (which can be called the X-axis predictive correlation matrix) can be obtained based on the predictive correlation coefficient between the x-coordinates of each pair of key points.

[0146] In this matrix, each row of the X-axis predictive correlation matrix corresponds to a keypoint, and each column also corresponds to a keypoint. Furthermore, the keypoint numbers in each column of the X-axis predictive correlation matrix, arranged from left to right, are consistent with the keypoint numbers in each row, arranged from top to bottom. Correspondingly, the number of rows and columns in the X-axis predictive correlation matrix are the same, both being the number of keypoints. The element in the i-th row and j-th column of the X-axis predictive correlation matrix represents the predicted correlation coefficient between the x-coordinates of the keypoints in the i-th and j-th rows. It can be understood that, based on the preset correlation coefficient algorithm, the value of the element in the i-th row and j-th column of the X-axis predictive correlation matrix is ​​consistent with the value of the element in the j-th row and i-th column, meaning the X-axis predictive correlation matrix is ​​a symmetric matrix. When i = j, the value of this element is 1, meaning that the values ​​of all elements on the main diagonal of the X-axis predictive correlation matrix are 1.

[0147] For example, in the above Figure 3 In the scenario shown, the obtained X-axis prediction correlation matrix is ​​a 68×68 matrix, where the i-th row and the i-th column of the matrix correspond to facial key points i, and the value of i ranges from 1 to 68.

[0148] In another implementation, for every two key points, the predictive correlation coefficient between the ordinates of those two key points can also be calculated, and this coefficient can be used as the predictive correlation coefficient between the two key points. Therefore, the predictive correlation coefficient between every two key points can be obtained.

[0149] Correspondingly, the predicted correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0150] In the embodiments of this application, the process of calculating the predictive correlation coefficient between the ordinates of any two key points is similar to the process of calculating the predictive correlation coefficient between the abscissas of any two key points, and will not be described in detail here.

[0151] If the predictive correlation coefficient between two key points only includes the predictive correlation coefficient between the ordinates of the two key points, a predictive correlation matrix (which can be called the Y-axis predictive correlation matrix) can be obtained based on the predictive correlation coefficient between the ordinates of each pair of key points.

[0152] The explanation of the Y-axis prediction correlation matrix is ​​similar to that of the X-axis prediction correlation matrix, and will not be repeated here.

[0153] In another implementation, for every two key points, the predicted correlation coefficient between their x-coordinates and their y-coordinates can also be calculated. This predicted correlation coefficient includes both the predicted correlation coefficient between their x-coordinates and their y-coordinates. Therefore, the predicted correlation coefficient between each pair of key points can be obtained.

[0154] In the embodiments of this application, the process of calculating the predictive correlation coefficient between the ordinates of any two key points and the process of calculating the predictive correlation coefficient between the abscissas of any two key points can refer to the above embodiments, and will not be repeated here.

[0155] When the predictive correlation coefficient between two key points includes the predictive correlation coefficient between the x-coordinates of the two key points and the predictive correlation coefficient between the y-coordinates of the two key points, an X-axis predictive correlation matrix and a Y-axis predictive correlation matrix can be obtained.

[0156] The process of obtaining the X-axis prediction correlation matrix and the Y-axis prediction correlation matrix can be referred to the above embodiments, and will not be repeated here.

[0157] For example, two key points belonging to the left eye in facial landmarks can be called L1 and L2. Two key points belonging to the right eye can be called R1 and R2. If the prediction correlation coefficient between two key points only includes the prediction correlation coefficient between the vertical coordinates of the two key points, in the scenarios of Figure 2(a) and Figure 2(b), for the first key point localization model trained in Figure 2(a), the corresponding partial matrices of L1, L2, R1 and R2 in the X-axis prediction correlation matrix are as shown in matrix (1):

[0158]

[0159] For example, in matrix (1), the value (1) of the element in the first row and first column is the predicted correlation coefficient between L1 and L2; the value (0.9083) of the element in the first row and second column is the predicted correlation coefficient between L1 and L2; the value (0.1497) of the element in the first row and third column is the predicted correlation coefficient between L1 and R1; and the value (0.1526) of the element in the first row and fourth column is the predicted correlation coefficient between L1 and R2. The predicted correlation coefficients represented by the elements in other positions in matrix (1) are similar to those represented by the elements in the first row, and will not be elaborated here.

[0160] For the first keypoint localization model to be trained corresponding to Figure 2(b), the corresponding partial matrices of L1, L2, R1 and R2 in the obtained X-axis prediction correlation matrix are shown in matrix (2):

[0161]

[0162] In matrix (1), the value (1) of the element in the first row and first column represents the predicted correlation coefficient between L1 and L2; the value (0.9662) of the element in the first row and second column represents the predicted correlation coefficient between L1 and L2; the value (0.8227) of the element in the first row and third column represents the predicted correlation coefficient between L1 and R1; and the value (0.7875) of the element in the first row and fourth column represents the predicted correlation coefficient between L1 and R2. The predicted correlation coefficients represented by the elements in other positions in matrix (2) are similar to those represented by the elements in the first row, and will not be elaborated here.

[0163] It can be seen that the prediction correlation coefficient between L1 and R1 in matrix (1) is much smaller than that between L1 and R1 in matrix (2), the prediction correlation coefficient between L1 and R2 in matrix (1) is much smaller than that between L1 and R2 in matrix (2), the prediction correlation coefficient between L2 and R1 in matrix (1) is much smaller than that between L2 and R1 in matrix (2), and the prediction correlation coefficient between L2 and R2 in matrix (1) is much smaller than that between L2 and R2 in matrix (2). Since in real-world scenarios, the correlation between key points belonging to the left eye and key points belonging to the right eye is low, the true correlation coefficients between L1 and R1, L1 and R2, L2 and R1, and L2 and R2 are also low. That is, the first key point localization model trained in this application can learn the true correlation between every two key points.

[0164] Based on the above processing, when the predicted correlation coefficient between two key points includes both the predicted correlation coefficient between their x-coordinates and their y-coordinates, the predicted correlation coefficient can reflect the correlation between the x-coordinates and y-coordinates of the two key points currently learned by the model. In this case, the obtained first loss value can more accurately reflect the difference between the correlation currently learned by the model between the two key points and the true correlation between the two key points. Thus, a more accurate key point localization model can be obtained to improve the accuracy of the localization results.

[0165] When the predicted correlation coefficient between two key points only includes the predicted correlation coefficient between the horizontal coordinates of the two key points, or only includes the predicted correlation coefficient between the vertical coordinates of the two key points, the computational cost of obtaining each predicted correlation coefficient is low. Correspondingly, the computational cost of obtaining the first loss value is also low. Thus, the training efficiency of the key point localization model can be improved.

[0166] For steps S104-S105, for any two key points of the specified object, since the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image, the true correlation coefficient can reflect the true correlation between the two key points.

[0167] After obtaining the true correlation coefficient between every two key points, a true correlation matrix can also be obtained based on the true correlation coefficient between every two key points. The specific process of obtaining the true correlation coefficient will be explained in subsequent embodiments.

[0168] Furthermore, a first loss value can be calculated based on a preset loss function (which can be called the first loss function) and the predicted correlation coefficient and the true correlation coefficient. For example, the first loss value can be calculated based on the difference between the predicted correlation matrix and the true correlation matrix, using the preset loss function. The first loss function can be an L2 loss function, an L1 loss function, or something similar.

[0169] Alternatively, a second loss value can be calculated based on a preset loss function (which can be called the second loss function), according to the predicted coordinates and the true coordinates of key points of a specified object in the first sample image. The second loss function can be an L2 loss function, an L1 loss function, or something similar.

[0170] Then, based on the first and second loss values, the model parameters of the first keypoint localization model to be trained are adjusted until the preset convergence condition is met, resulting in the trained first keypoint localization model. For example, a weighted sum of the first and second loss values ​​can be calculated as the total loss value. The model parameters of the first keypoint localization model are then adjusted using gradient descent.

[0171] In this application, the true correlation coefficient between any two key points can be obtained through any of the following methods:

[0172] Method 1: In some embodiments, the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image, combined with the physiological constraints of the parts to which the key points of the specified object belong.

[0173] In this embodiment of the application, for any two key points of a specified object, the true correlation coefficient between the two key points can be obtained based on the respective locations of the two key points and the true coordinates of the two key points in the second sample image.

[0174] It is understandable that if the two key points do not have a physiological constraint relationship with each other, then the two key points are unrelated.

[0175] For example, when the specified object is a human face, there is no physiological constraint relationship between the facial contour and the nose. For instance, when a person speaks, causing the facial contour to change, the position of the nose is usually not affected. Therefore, it can be determined that a key point belonging to the facial contour is unrelated to another key point belonging to the nose.

[0176] For example, when the object is a person's hand, the thumb and other fingers do not have a physiological constraint relationship. For instance, when the thumb is usually bent, the other fingers can be either extended or bent. Therefore, it can be determined that a key point belonging to the thumb is not related to another key point belonging to another finger.

[0177] For example, when the object is a person's hand, the front and back points of the thumb joint are physiologically constrained. For instance, normally, when the thumb is bent, the front point of the joint will protrude, while the back point will bend and fold.

[0178] Correspondingly, for two unrelated key points, the true correlation coefficient between the two key points can be determined to be 0.

[0179] If the two key points are physiologically related to each other, then the two key points are related.

[0180] For example, when the specified object is a human face, there is a physiological constraint relationship between the left eyebrow and the left eye. For instance, the position of the left eyebrow usually changes when the left eye goes from open to closed. Therefore, it can be determined that a key point belonging to the left eyebrow is related to another key point belonging to the left eye.

[0181] For example, when the object is a person's hand, the little finger and the ring finger have a physiological constraint relationship. For instance, when the little finger is bent, the ring finger is also bent. Therefore, it can be determined that a key point belonging to the little finger is related to another key point belonging to the ring finger.

[0182] Accordingly, the process of determining the true correlation coefficient between the two key points based on the true coordinates of the key points of the specified object in the second sample image can be found in subsequent embodiments.

[0183] Based on the above processing, since the true correlation coefficient is determined by combining the true coordinates of the key points of the specified object in the second sample image with the physiological constraints of the parts to which the key points belong in the specified object, the true correlation coefficient between two key points can reflect the correlation between the physiological positions of the specified object represented by the two key points. In this way, the key point localization model can learn the true correlation between the two key points.

[0184] Accordingly, see Figure 4 , Figure 4 A first flowchart for obtaining the true correlation coefficient provided in the embodiments of this application includes:

[0185] S401: Obtain the second sample image.

[0186] S402: For each key point, calculate the mean of the horizontal coordinate and the mean of the vertical coordinate in the true coordinates of the key point in each second sample image, and use them as the template coordinates of the key point.

[0187] S403: For each pair of key points, calculate the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points and the physiological constraints of the parts to which the two key points belong in the specified object.

[0188] In this embodiment of the application, the second sample image is a sample image containing a specified object.

[0189] It is understandable that, since the region proportion and pose of a specified object in the sample images of the existing sample set may vary, after obtaining the sample images in the sample set, the sample images can be aligned to obtain the aligned sample images (i.e., the second sample images).

[0190] In some embodiments, step S401 includes: acquiring a third sample image; mapping each third sample image to a unified image template to obtain a second sample image.

[0191] Among them, the distance between the preset parts of the specified objects in each second sample image is less than a preset threshold.

[0192] In this embodiment, a third sample image of a specified object can be obtained. For each third sample image, it can be mapped to a unified image template so that the distance between preset parts of the specified object in each mapped third sample image is less than a preset threshold. For example, when the specified object is a face, the preset parts can be the left eye, the right eye, and the mouth. This ensures that the positions of the eyes and mouths are approximately the same in each mapped third sample image. The unified image template can have a specific preset size, for example, 224×224. Furthermore, during the process of mapping the third sample image to the unified image template, the third sample image can also be cropped, parallelized, and / or linearly mapped to ensure that the mapped third sample image meets the conditions described above.

[0193] Based on the above processing, the region proportion and pose of the specified object in the third sample image can be adjusted so that the proportion and pose of the specified object in the obtained second sample image are more conducive to the processing of the key point localization model, thereby further improving the accuracy of the key point localization model.

[0194] In steps S402-S403, for each key point of a specified object, the mean of the horizontal coordinate and the mean of the vertical coordinate in the true coordinates of the key point in each second sample image can be calculated as the template coordinates of the key point.

[0195] For example, in the above Figure 3 In the scenario shown, for each facial keypoint, such as facial keypoint 1, the x-coordinate of facial keypoint 1 in the true coordinates of each second sample image can be obtained. Then, the mean of the obtained x-coordinates can be calculated and used as the x-coordinate value in the template coordinates of facial keypoint 1. Similarly, the mean of the obtained y-coordinates can be calculated and used as the y-coordinate value in the template coordinates of facial keypoint 1. Thus, the template coordinates of each facial keypoint can be obtained.

[0196] Furthermore, for each pair of key points, the distance between their template coordinates can be calculated. Generally, the larger the distance between the template coordinates of two key points, the greater the distance between the two key points in the physiological location of the specified object, and correspondingly, the lower the correlation between the two key points. Therefore, based on the distance between the template coordinates of the two key points and the physiological constraints of the parts to which the two key points belong in the specified object, the true correlation coefficient between the two key points can be calculated. The specific calculation process will be described in subsequent embodiments.

[0197] Based on the above processing, the template coordinates of each keypoint can represent the average level of each specified object in each second sample image. Calculating the true correlation coefficient between every two keypoints based on the template coordinates of each keypoint avoids interference from individual differences in the specified objects in the sample images, thus obtaining a more accurate keypoint localization model to improve the accuracy of the localization results.

[0198] In some embodiments, see Figure 5 , Figure 5 The second flowchart for obtaining the true correlation coefficient provided in this application embodiment is as follows: Figure 4 Based on this, step S403 includes:

[0199] S4031: For every two key points, if there is no physiological constraint relationship between the parts to which the two key points belong in the specified object, then the true correlation coefficient between the two key points is determined to be 0.

[0200] S4032: If the parts to which the two key points belong in the specified object have a physiological constraint relationship, and the distance between the template coordinates of the two key points is not less than the preset distance, then the true correlation coefficient between the two key points is determined to be 0.

[0201] S4033: If the parts to which the two key points belong in the specified object have a physiological constraint relationship, and the distance between the template coordinates of the two key points is less than the preset distance, then the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points.

[0202] The calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

[0203] In this embodiment, irrelevant parts of a specified object can be predetermined. For every two key points, if the parts to which the two key points belong in the specified object are unrelated, the true correlation coefficient between the two key points can be determined to be 0.

[0204] For example, based on prior physiological knowledge, when the specified object is a face, it can be determined that the facial contours and nose are independent of each other. Correspondingly, in the above... Figure 3 In the scenario shown, for facial key point 1 and facial key point 28, since facial key point 1 belongs to the facial contour and facial key point 28 belongs to the nose, it can be directly determined that the true correlation coefficient between facial key point 1 and facial key point 28 is 0.

[0205] If the locations of the two key points in the specified object are related, the true correlation coefficient between the two key points can be determined based on the distance between their template coordinates.

[0206] If the distance between the template coordinates of two key points is not less than the preset distance, it indicates that the distance between the template coordinates of the two key points is relatively large. Although the parts to which the two key points belong are related, the two key points themselves are not related. In this case, the true correlation coefficient between the two key points can be determined as 0. The preset distance can be determined based on the parts to which the two key points belong.

[0207] If the distance between the template coordinates of two key points is less than a preset distance, it indicates that the template coordinates of the two key points are close and that the two key points are correlated. In this case, the true correlation coefficient between the two key points is negatively correlated with the distance between their template coordinates. Therefore, the true correlation coefficient between the two key points can be calculated based on the distance between their template coordinates. The specific calculation process will be explained in subsequent embodiments.

[0208] Based on the above processing, for each pair of key points, it can be determined in advance whether the parts to which the two key points belong are related. If the parts to which the two key points belong are not related, it can be directly determined that the two key points are not related. This can reduce the amount of computation consumed in the training of the key point localization model and improve the training efficiency.

[0209] In addition, if the parts to which the two key points belong are related, it can be determined whether the distance between the template coordinates of the two key points is less than the preset distance. If it is not less than the preset distance, it means that the distance between the template coordinates of the two key points is far and the two key points are not related. This can further reduce the amount of computation consumed in the training of the key point localization model and improve the training efficiency.

[0210] In some embodiments, S4033:

[0211] Step 1: According to the first preset formula, calculate the first correlation coefficient between the two key points using the distance between the template coordinates of the two key points.

[0212] The first preset formula indicates that the first part and the second part of the corresponding body parts of the two key points have a physiological constraint relationship. The first preset formula is:

[0213] R1=1-d×Z1

[0214] R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight.

[0215] Step 2: Based on the first correlation coefficient, calculate the true correlation coefficient between the two key points.

[0216] In this embodiment of the application, the first preset weight corresponds to the first part of the parts to which the two key points belong. The first part can be any part on the specified object.

[0217] In one implementation, after obtaining the first correlation coefficient, the first correlation coefficient can be directly determined as the true correlation coefficient between the two key points.

[0218] In another implementation, the method also includes:

[0219] Step 3: According to the second preset formula, use the distance between the template coordinates of the two key points to calculate the second correlation coefficient between the two key points.

[0220] The second preset formula indicates that the second part and the first part have a physiological constraint relationship. The second preset formula is:

[0221] R² = 1 - d × Z²

[0222] R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight.

[0223] Step 2 above includes: calculating the average of the first correlation coefficient and the second correlation coefficient to obtain the true correlation coefficient between the two key points.

[0224] Correspondingly, the second preset weight corresponds to the second part of the region to which the two key points belong. In other words, when calculating the true correlation coefficient between key points belonging to two different regions, the weights in the formula used can be different or the same.

[0225] Correspondingly, for two key points, when the locations of these two key points are different (i.e., the first location and the second location are different), the calculated first correlation coefficient and second correlation coefficient may be different or the same. Furthermore, when the first correlation coefficient and the second correlation coefficient are different, the average of the first and second correlation coefficients can be calculated to obtain the true correlation coefficient between the two key points. This improves the accuracy of obtaining the true correlation coefficient between the two key points, thereby obtaining a more accurate key point localization model and improving the accuracy of the localization results.

[0226] For two key points, if the locations of the two key points are the same, the calculated first correlation coefficient and the second correlation coefficient will necessarily be the same. Therefore, after calculating the first correlation coefficient, the calculated first correlation coefficient can be directly determined as the true correlation coefficient between the two key points.

[0227] For example, in the above Figure 3 In the scenario shown, for facial keypoint 1 and facial keypoint 2, the location of facial keypoint 1 is the facial contour (i.e., the first part), and the location of facial keypoint 2 is also the facial contour (i.e., the second part). At this time, the first preset weight and the second preset weight are the same, and correspondingly, the calculated first correlation coefficient and the second correlation coefficient are also the same. Therefore, after calculating the first correlation coefficient, it can be directly determined as the true correlation coefficient between the two keypoints.

[0228] In this way, the amount of computation required to obtain the true correlation coefficient between two key points can be reduced, thus improving the efficiency of obtaining the true correlation coefficient.

[0229] For example, in a scenario where a facial landmark localization model is being trained, and the specified object's body parts include: facial contour, left eyebrow, right eyebrow, nose, left eye, right eye, and mouth, the true correlation coefficient between each pair of facial landmarks can be calculated using the following formula:

[0230]

[0231]

[0232]

[0233]

[0234]

[0235]

[0236] Where i represents facial landmark i, and j represents facial landmark j. d ij This represents the distance between the template coordinates of facial keypoint i and facial keypoint j. `outline` represents the outer contour; `eyebrow` contains `l_eyebrow` and `r_eyebrow`, where `l_eyebrow` represents the left eyebrow and `r_eyebrow` represents the right eyebrow; `eye` contains `l_eye` and `r_eye`, where `l_eye` represents the left eye and `r_eyebrow` represents the right eye; `nose` represents the nose; and `mouth` represents the mouth.

[0237] When the facial keypoint i belongs to the facial contour, it can be based on R outline The corresponding formula determines the true correlation coefficient between facial key point i and facial key point j. At this time, the preset weight is 0.0025. If the part to which facial key point j belongs is not the outer contour or mouth, or if the distance between the template coordinates of facial key point i and facial key point j is not less than 120, then the true correlation coefficient between facial key point i and facial key point j is 0.

[0238] When the facial landmark i belongs to either the right or left eyebrow, it can be based on R. eyebrow The corresponding formula determines the true correlation coefficient between facial key point i and facial key point j. At this point, the preset weight is 0.005. If the location of facial key point j is not the right eyebrow, left eyebrow, left eye, or right eye, or if the distance between the template coordinates of facial key point i and facial key point j is not less than 60, then the true correlation coefficient between facial key point i and facial key point j is 0.

[0239] When the facial landmark i is located in the left eye area, it can be based on R l _ eye The corresponding formula determines the true correlation coefficient between facial key point i and facial key point j. At this time, the preset weight is 0.005. If the part to which facial key point j belongs is not the left eyebrow, left eye, or nose, or if the distance between the template coordinates of facial key point i and facial key point j is not less than 60, then the true correlation coefficient between facial key point i and facial key point j is 0.

[0240] When the facial landmark i is located at the right eye, it can be based on R r _ eye The corresponding formula determines the true correlation coefficient between facial key point i and facial key point j. At this time, the preset weight is 0.005. If the part to which facial key point j belongs is not the right eyebrow, right eye, or nose, or if the distance between the template coordinates of facial key point i and facial key point j is not less than 60, then the true correlation coefficient between facial key point i and facial key point j is 0.

[0241] When the facial landmark i belongs to the nose, it can be based on R nose The corresponding formula determines the true correlation coefficient between facial key point i and facial key point j. At this point, the preset weight is 0.0025. If the location of facial key point j is not the right eyebrow, left eyebrow, left eye, right eye, nose, or mouth, or if the distance between the template coordinates of facial key point i and facial key point j is not less than 120, then the true correlation coefficient between facial key point i and facial key point j is 0.

[0242] When the facial keypoint i belongs to the mouth, it can be based on R. mouth The corresponding formula determines the true correlation coefficient between facial key point i and facial key point j. At this time, the preset weight is 0.005. If the part to which facial key point j belongs is not the facial contour, nose or mouth, or if the distance between the template coordinates of facial key point i and facial key point j is not less than 60, then the true correlation coefficient between facial key point i and facial key point j is 0.

[0243] Understandably, according to Method 1, the true correlation coefficient between any two key points is unique. That is, for any two key points, there is no distinction between the true correlation coefficient between their horizontal axes and their vertical axes. In this case, a true correlation matrix can be obtained according to Method 1 described in the above implementation.

[0244] Subsequently, if the predicted correlation coefficient between two key points only includes the predicted correlation coefficient between the x-coordinates of the two key points, the first loss value can be calculated based on the difference between the true correlation matrix and the predicted correlation matrix on the x-axis.

[0245] If the predicted correlation coefficient between two key points only includes the predicted correlation coefficient between the y-axis of the two key points, the first loss value can be calculated based on the difference between the true correlation matrix and the Y-axis predicted correlation matrix.

[0246] When the predicted correlation coefficient between two key points only includes the predicted correlation coefficient between the x-axis of the two key points, the first loss value can be calculated based on the difference between the true correlation matrix and the X-axis predicted correlation matrix, and the difference between the true correlation matrix and the Y-axis predicted correlation matrix.

[0247] Method 2: Based on a pre-trained keypoint localization model, the true correlation coefficient between every two keypoints can be obtained. See [link / reference] for details. Figure 6 , Figure 6 A third flowchart for obtaining the true correlation coefficient provided in the embodiments of this application includes:

[0248] S601: Obtain the current weight matrix of the specified network layer in the second keypoint localization model, and use it as the true weight matrix.

[0249] The second keypoint localization model is trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents either the feature of the horizontal coordinate of a keypoint or the feature of the vertical coordinate of a keypoint.

[0250] S602: For every two key points, calculate the correlation coefficient between the two key points based on the columns corresponding to the two key points in the true weight matrix, and use it as the true correlation coefficient.

[0251] In this embodiment, the second sample image consists of a small number of sample images containing specified objects in a specified state and a small number of sample images containing specified objects in a non-specified state. Since the second sample image contains specified objects in both non-specified and specified states, the true correlation between any two key points can be determined based on the true coordinates of key points in the second sample images of specified objects in different states, thereby improving the accuracy of the predicted correlation coefficient between any two key points learned by the model.

[0252] Accordingly, the second keypoint localization model is trained based on the second sample image and the corresponding second sample label. The process of obtaining the current weight matrix of the specified network layer in the second keypoint localization model as the true weight matrix is ​​similar to the process of obtaining the predicted weight matrix. For the specific processing procedure, please refer to the relevant description of step S102 above, which will not be repeated here.

[0253] Furthermore, after obtaining the true weight matrix, the true correlation coefficient between each pair of key points can be calculated based on the true weight matrix.

[0254] In step S602, for any key point, the column corresponding to the horizontal coordinate of the key point and the column corresponding to the vertical coordinate of the key point can be obtained from the real weight matrix.

[0255] Based on the above processing, the true correlation coefficient between each pair of key points can be obtained through model learning. Compared with the first method, this reduces the computational cost of obtaining the true correlation coefficient and improves the training efficiency of the key point localization model.

[0256] In some embodiments, the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points, and / or the true correlation coefficient between the vertical coordinates of the two key points.

[0257] Correspondingly, the true correlation coefficient between the x-coordinates of any two key points is obtained by processing the columns corresponding to the x-coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm. Similarly, the true correlation coefficient between the y-coordinates of any two key points is obtained by processing the columns corresponding to the y-coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm.

[0258] In one implementation, for every two key points, only the true correlation coefficient between the x-coordinates of the two key points can be obtained as the true correlation coefficient between the two key points.

[0259] The calculation of the true correlation coefficient between the x-coordinates of the two key points is similar to the calculation of the predicted correlation coefficient between the x-coordinates of the two key points, and will not be elaborated here.

[0260] At this point, a true correlation matrix (which can be called the X-axis true correlation matrix) can also be obtained based on the true correlation coefficient between the x-coordinates of every two key points. The process of obtaining the X-axis true correlation matrix is ​​similar to that of obtaining the X-axis predicted correlation matrix, and will not be elaborated here.

[0261] In another implementation, for each pair of key points, only the true correlation coefficient between the ordinates of the two key points can be obtained as the true correlation coefficient between the two key points.

[0262] The method for calculating the true correlation coefficient between the ordinates of the two key points is similar to that for calculating the predicted correlation coefficient between the ordinates of the two key points, and will not be elaborated here.

[0263] At this point, a true correlation matrix (which can be called the Y-axis true correlation matrix) can also be obtained based on the true correlation coefficient between the ordinates of every two key points. The process of obtaining the Y-axis true correlation matrix is ​​similar to that of obtaining the Y-axis predicted correlation matrix, and will not be elaborated here.

[0264] In another implementation, for every two key points, the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points can be obtained to obtain the true correlation coefficient between the two key points.

[0265] At this point, the true correlation matrix of the X-axis can be obtained based on the true correlation coefficient between the horizontal coordinates of every two key points, and the true correlation matrix of the Y-axis can be obtained based on the true correlation coefficient between the vertical coordinates of every two key points.

[0266] In some embodiments, step S104 includes:

[0267] In one implementation, if the true correlation coefficient between any two key points includes the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points includes the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, then a first loss value is calculated based on the first sub-loss value and the second sub-loss value.

[0268] The first sub-loss value represents the difference between the predicted correlation coefficient and the true correlation coefficient between the horizontal axes; the second sub-loss value represents the difference between the predicted correlation coefficient and the true correlation coefficient between the vertical axes.

[0269] In the embodiments of this application, if the true correlation coefficient between any two key points includes the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, then the predicted correlation coefficient between any two key points also includes the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points.

[0270] Therefore, the first sub-loss value can be calculated based on the true correlation coefficient between each horizontal axis and the predicted correlation coefficient between each horizontal axis. For example, the first sub-loss value can be calculated based on the difference between the true correlation matrix and the predicted correlation matrix on the X-axis. And the second sub-loss value can be calculated based on the true correlation coefficient between each vertical axis and the predicted correlation coefficient between each vertical axis. For example, the second sub-loss value can be calculated based on the difference between the true correlation matrix and the predicted correlation matrix on the Y-axis.

[0271] Furthermore, the first loss value can be calculated based on the first sub-loss value and the second sub-loss value. For example, the weighted sum of the first and second sub-loss values ​​can be calculated as the first loss value.

[0272] Based on the above processing, when the true correlation coefficient between two key points includes both the true correlation coefficient between their x-coordinates and their y-coordinates, the true correlation coefficient can reflect the true correlation between the x-coordinates and y-coordinates of each pair of key points. In this case, the first loss value can more accurately reflect the difference between the correlation currently learned by the model between the two key points and the true correlation between them. Thus, a more accurate keypoint localization model can be obtained to improve the accuracy of the localization results.

[0273] Another implementation method is to obtain the first sub-loss value as the first loss value if the true correlation coefficient between any two key points is the same as the true correlation coefficient between the x-coordinates of the two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the x-coordinates of the two key points.

[0274] In the embodiments of this application, if the true correlation coefficient between any two key points includes the true correlation coefficient between the horizontal coordinates of the two key points, then the predicted correlation coefficient between any two key points also includes the predicted correlation coefficient between the horizontal coordinates of the two key points.

[0275] Therefore, the first sub-loss value can be calculated based on the true correlation coefficient between each horizontal axis and the predicted correlation coefficient between each horizontal axis. For example, the loss value between the true correlation matrix on the X-axis and the predicted correlation matrix on the X-axis can be calculated as the first sub-loss value. Furthermore, the first sub-loss value can be directly used as the first loss value.

[0276] Another implementation method is to obtain the second sub-loss value as the first loss value if the true correlation coefficient between any two key points is the same as the true correlation coefficient between the ordinates of the two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the ordinates of the two key points.

[0277] In the embodiments of this application, if the true correlation coefficient between any two key points includes the true correlation coefficient between the ordinates of the two key points, then the predicted correlation coefficient between any two key points also includes the predicted correlation coefficient between the ordinates of the two key points.

[0278] Therefore, the second sub-loss value can be calculated based on the true correlation coefficient between each y-axis and the predicted correlation coefficient between each y-axis. For example, the loss value between the true correlation matrix and the predicted correlation matrix on the y-axis can be calculated as the second sub-loss value. Furthermore, the second sub-loss value can be directly used as the first loss value.

[0279] When the true correlation coefficient between two key points only includes the true correlation coefficient between the horizontal coordinates of the two key points, or only includes the true correlation coefficient between the vertical coordinates of the two key points, the computational cost of obtaining each true correlation coefficient is low. Correspondingly, the computational cost of obtaining the first loss value is also low. Thus, the training efficiency of the key point localization model can be improved.

[0280] See Figure 7 , Figure 7 A flowchart illustrating the training keypoint localization model provided in this application embodiment.

[0281] S701: Start.

[0282] S702: Obtain a training set of key points for normal human faces.

[0283] That is, obtain the first sample image. The first sample image contains at least one specified object in a non-specified state.

[0284] S703: Training the keypoint regression model.

[0285] That is, the first sample image is input into the key point localization model to be trained (i.e., the first key point localization model to be trained).

[0286] S704: Calculate the feature correlation weights for each point.

[0287] That is, obtain the current weight matrix of a specified network layer in the first keypoint localization model to be trained, and use it as the prediction weight matrix. For every two keypoints, calculate the correlation coefficient between the two keypoints based on the columns corresponding to the two keypoints in the prediction weight matrix, and use it as the prediction correlation coefficient.

[0288] S705: Obtain the correlation of features of each constrained point.

[0289] That is, obtain the true correlation coefficient. The true correlation coefficient can be obtained through the learning method described above, or it can be calculated using the formula described above.

[0290] S706: Facial landmark model feature decoupling learning.

[0291] That is, based on the predicted correlation coefficient and the true correlation coefficient, a first loss value is calculated; based on the first loss value and the second loss value, the model parameters of the first keypoint localization model to be trained are adjusted until the preset convergence condition is met, thus obtaining the trained first keypoint localization model. The second loss value is determined based on the predicted coordinates and the true coordinates of the keypoints of the specified object in the first sample image.

[0292] S707: Facial landmark model prediction.

[0293] That is, the image to be detected is input into the first key point localization model that has been trained.

[0294] S708: Outputs the coordinates of facial landmarks.

[0295] That is, the coordinates of key points of a specified object in the image to be detected are obtained.

[0296] Based on the same inventive concept, this application provides a key point localization method, see [link to relevant documentation]. Figure 8 , Figure 8 A first flowchart of a key point localization method provided in an embodiment of the present invention, the method comprising:

[0297] S801: Obtain the image to be detected that contains the specified object.

[0298] S802: Input the image to be detected into the trained third keypoint localization model to obtain the coordinates of the keypoints of the specified object in the image to be detected.

[0299] The third keypoint localization model is obtained by training according to the training method of any of the above keypoint localization models.

[0300] In this embodiment of the application, the image to be detected is the image for which key point localization is currently required.

[0301] The image to be detected is input into a trained third keypoint localization model to obtain the coordinates of keypoints of a specified object in the image. The third keypoint localization model is trained using any of the keypoint localization model training methods described above.

[0302] Based on the above processing, during the training of the keypoint localization model, the current weight matrix (i.e., the predicted weight matrix) of the output layer of the keypoint localization model can be obtained. Since one column in the weight matrix can represent the features of the horizontal coordinate of a keypoint, or the features of the vertical coordinate of a keypoint, the predicted correlation coefficient obtained based on the predicted weight matrix for any two keypoints of a specified object can reflect the correlation currently learned by the model between the two keypoints. Furthermore, since the true correlation coefficient is determined based on the true coordinates of the keypoints of the specified object in the second sample image, the true correlation coefficient can reflect the true correlation between the two keypoints. Therefore, the first loss value can reflect the difference between the correlation currently learned by the model between the two keypoints and the true correlation between the two keypoints. Accordingly, adjusting the model parameters of the first keypoint localization model to be trained based on the first loss value enables the trained first keypoint localization model to learn the true correlation between every two keypoints, thus obtaining a keypoint localization model with higher accuracy and improving the accuracy of the localization results.

[0303] See Figure 9 , Figure 9 This is a flowchart for obtaining facial key points in a face sample image provided in an embodiment of this application.

[0304] S901: Obtain the input image.

[0305] S902: Input the input image to the face detector. If a face is detected, proceed to step S903; otherwise, return to step S901.

[0306] S903: Acquire face region image.

[0307] That is, the image region corresponding to the smallest bounding rectangle of the face is determined from the input image and used as the face region image.

[0308] S904: Input the face region image into the face landmark localization model.

[0309] The facial landmark localization model is trained according to the landmark localization model training method provided in this application.

[0310] S905: Obtain the coordinates of each facial key point in the face region image.

[0311] In the technical solution of this application, the acquisition, storage, use, processing, transmission, provision and disclosure of user personal information (facial information, human hand information, human body information, etc.) are all carried out with the user's authorization.

[0312] It should be noted that the image containing the specified object in this embodiment is not an image of the specified object corresponding to a specific user, and does not reflect the personal information of a specific user.

[0313] It should be noted that the images containing faces, hands, and bodies in this embodiment are from publicly available datasets.

[0314] Based on the same inventive concept, embodiments of this application provide a key point localization model training device. See also Figure 10 , Figure 10 This application provides a structural diagram of a key point localization model training device, which includes:

[0315] The prediction coordinate acquisition module 1001 is used to input a first sample image containing a specified object into a first keypoint localization model to be trained, and obtain the coordinates of the keypoints of the specified object in the first sample image as prediction coordinates.

[0316] The prediction weight matrix acquisition module 1002 is used to acquire the current weight matrix of a specified network layer in the first key point localization model to be trained, as the prediction weight matrix; wherein, any column in the prediction weight matrix represents: the feature of the horizontal coordinate of a key point, or the feature of the vertical coordinate of a key point.

[0317] The prediction correlation coefficient acquisition module 1003 is used to calculate the correlation coefficient between two key points based on the columns corresponding to the two key points in the prediction weight matrix for each pair of key points, and use it as the prediction correlation coefficient.

[0318] The first loss value acquisition module 1004 is used to calculate a first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image;

[0319] The model parameter adjustment module 1005 is used to adjust the model parameters of the first keypoint localization model to be trained based on the first loss value and the second loss value until a preset convergence condition is reached, so as to obtain the trained first keypoint localization model; wherein, the second loss value is determined based on the predicted coordinates and the true coordinates of the keypoints of the specified object in the first sample image.

[0320] In some embodiments, the predictive correlation coefficient between any two key points includes: the predictive correlation coefficient between the horizontal coordinates of the two key points, and / or the predictive correlation coefficient between the vertical coordinates of the two key points.

[0321] In some embodiments, the predicted correlation coefficient between the abscissas of any two key points is obtained by processing the columns corresponding to the abscissas of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm; the predicted correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0322] In some embodiments, the true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image, combined with the physiological constraints of the parts to which the key points of the specified object belong.

[0323] In some embodiments, the apparatus further includes: a second sample image acquisition module, configured to acquire the second sample image before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient; a template coordinate acquisition module, configured to calculate, for each key point, the mean of the horizontal coordinate and the mean of the vertical coordinate in the true coordinates of the key point in each second sample image, as the template coordinates of the key point; and a first true correlation coefficient acquisition module, configured to calculate, for every two key points, the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object.

[0324] In some embodiments, the first true correlation coefficient acquisition module includes: a first condition determination submodule, configured to determine the true correlation coefficient between two key points as 0 if, for every two key points, the parts to which the two key points belong in the specified object do not have a physiological constraint relationship; a second condition determination submodule, configured to determine the true correlation coefficient between two key points as 0 if, in the specified object, the parts to which the two key points belong have a physiological constraint relationship, and the distance between the template coordinates of the two key points is not less than a preset distance; and a third condition determination submodule, configured to calculate the true correlation coefficient between two key points based on the distance between the template coordinates of the two key points if, in the specified object, the parts to which the two key points belong have a physiological constraint relationship, and the distance between the template coordinates of the two key points is less than a preset distance; wherein the calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

[0325] In some embodiments, the third condition determination submodule includes: a first correlation coefficient calculation unit, configured to calculate a first correlation coefficient between the two key points using the distance between the template coordinates of the two key points according to a first preset formula; wherein, the first preset formula indicates that the first part of the part to which the two key points belong and the second part of the part to which they belong have a physiological constraint relationship; the first preset formula is: R1 = 1 - d × Z1, where R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight; and a true correlation coefficient calculation unit, configured to calculate the true correlation coefficient between the two key points based on the first correlation coefficient.

[0326] In some embodiments, the device further includes: a second correlation coefficient calculation module, configured to calculate a second correlation coefficient between the two key points according to a second preset formula, using the distance between the template coordinates of the two key points, before calculating the true correlation coefficient between the two key points based on the first correlation coefficient; wherein, the second preset formula indicates that the second part and the first part have a physiological constraint relationship; the second preset formula is: R2 = 1 - d × Z2, where R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight; the true correlation coefficient calculation unit is specifically configured to: calculate the average of the first correlation coefficient and the second correlation coefficient to obtain the true correlation coefficient between the two key points.

[0327] In some embodiments, the second sample image acquisition module includes: a third sample image acquisition submodule for acquiring a third sample image; and a mapping submodule for mapping each third sample image to a unified image template to obtain a second sample image; wherein the distance between preset parts of a specified object in each second sample image is less than a preset threshold.

[0328] In some embodiments, the apparatus further includes: a true weight matrix acquisition module, configured to acquire the current weight matrix of a specified network layer in the second keypoint localization model as the true weight matrix before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the second keypoint localization model is trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: the feature of the horizontal coordinate of a keypoint, or the feature of the vertical coordinate of a keypoint; and a second true correlation coefficient acquisition module, configured to calculate the correlation coefficient between two keypoints based on the columns corresponding to the two keypoints in the true weight matrix for each pair of keypoints, as the true correlation coefficient.

[0329] In some embodiments, the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points, and / or the true correlation coefficient between the vertical coordinates of the two key points.

[0330] In some embodiments, the true correlation coefficient between the x-coordinates of any two key points is obtained by processing the columns corresponding to the x-coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm; the true correlation coefficient between the y-coordinates of any two key points is obtained by processing the columns corresponding to the y-coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm.

[0331] In some embodiments, the first loss value acquisition module 1004 is specifically configured to: if the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, then calculate a first loss value based on a first sub-loss value and a second sub-loss value; wherein, the first sub-loss value represents: the difference between the predicted correlation coefficient between the horizontal coordinates and the true correlation coefficient between the horizontal coordinates; the second ... The loss value represents the difference between the predicted correlation coefficient between the vertical axes and the true correlation coefficient between the vertical axes. If the true correlation coefficient between any two key points is the same as the true correlation coefficient between the horizontal axes of the two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the horizontal axes of the two key points, then the first sub-loss value is obtained as the first loss value. If the true correlation coefficient between any two key points is the same as the true correlation coefficient between the vertical axes of the two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the vertical axes of the two key points, then the second sub-loss value is obtained as the first loss value.

[0332] In some embodiments, the key points of the specified object include at least one of the following: facial key points, finger key points, and human posture key points.

[0333] Based on the same inventive concept, embodiments of this application provide a key point positioning device. See also Figure 11 , Figure 11 A structural diagram of a key point localization device provided in this application embodiment is shown. The device includes:

[0334] The image acquisition module 1101 is used to acquire an image to be detected containing a specified object;

[0335] The positioning module 1102 is used to input the image to be detected into a trained third keypoint positioning model to obtain the coordinates of key points of a specified object in the image to be detected; wherein, the third keypoint positioning model is trained according to any keypoint positioning model training method in the above embodiments.

[0336] This application also provides an electronic device, such as... Figure 12 As shown, it includes:

[0337] Memory 1201 is used to store computer programs;

[0338] When the processor 1202 executes the program stored in the memory 1201, it implements any of the key point localization model training methods or key point localization method steps in the above embodiments.

[0339] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1202, the communication interface, and the memory 1201 communicating with each other via the communication bus.

[0340] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0341] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0342] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0343] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0344] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the above-described key point localization model training methods, or the steps of the key point localization method.

[0345] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the key point localization model training methods or key point localization methods in the above embodiments.

[0346] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0347] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0348] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and computer-readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0349] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A key point localization model training method, characterized in that, The method includes: The first sample image containing the specified object is input into the first keypoint localization model to be trained, and the coordinates of the keypoints of the specified object in the first sample image are obtained as the predicted coordinates. Obtain the current weight matrix of the specified network layer in the first keypoint localization model to be trained, as the prediction weight matrix; wherein, any column in the prediction weight matrix represents: the feature of the horizontal coordinate of a keypoint or the feature of the vertical coordinate of a keypoint; For each pair of key points, the correlation coefficient between the two key points is calculated based on the columns corresponding to the two key points in the prediction weight matrix, and is used as the prediction correlation coefficient. Based on the predicted correlation coefficient and the true correlation coefficient, a first loss value is calculated; wherein, the true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image; Based on the first loss value and the second loss value, the model parameters of the first keypoint localization model to be trained are adjusted until the preset convergence condition is met, and the trained first keypoint localization model is obtained; wherein, the second loss value is determined based on the predicted coordinates and the true coordinates of the keypoints of the specified object in the first sample image.

2. The method according to claim 1, characterized in that, The predictive correlation coefficient between any two key points includes: the predictive correlation coefficient between the x-coordinates of the two key points and / or the predictive correlation coefficient between the y-coordinates of the two key points.

3. The method according to claim 2, characterized in that, The predicted correlation coefficient between the x-coordinates of any two key points is obtained by processing the columns corresponding to the x-coordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm. The predicted correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

4. The method according to claim 1, characterized in that, The true correlation coefficient is determined based on the true coordinates of key points of a specified object in the second sample image, combined with the physiological constraints of the parts to which the key points of the specified object belong.

5. The method according to claim 4, characterized in that, Before calculating the first loss value based on the predicted correlation coefficient and the actual correlation coefficient, the method further includes: Obtain the second sample image; For each key point, the mean of the horizontal coordinate and the mean of the vertical coordinate in the true coordinates of the key point in each second sample image are calculated and used as the template coordinates of the key point. For each pair of key points, the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points and the physiological constraints of the parts to which the two key points belong in the specified object.

6. The method according to claim 5, characterized in that, For each pair of key points, based on the distance between the template coordinates of the two key points and the physiological constraints of the locations to which the two key points belong in the specified object, the true correlation coefficient between the two key points is calculated, including: For every two key points, if the parts to which the two key points belong in the specified object do not have a physiological constraint relationship, then the true correlation coefficient between the two key points is determined to be 0. If the parts to which the two key points belong in the specified object are physiologically constrained, and the distance between the template coordinates of the two key points is not less than the preset distance, then the true correlation coefficient between the two key points is determined to be 0. If the parts to which the two key points belong in the specified object are physiologically constrained, and the distance between the template coordinates of the two key points is less than a preset distance, then the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points; wherein, the calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

7. The method according to claim 6, characterized in that, The calculation of the true correlation coefficient between the two key points based on the distance between their template coordinates includes: According to the first preset formula, the first correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points; wherein, the first preset formula indicates that there is a physiological constraint relationship between the first part of the part to which the two key points belong and the second part of the part to which they belong. The first preset formula is: This represents the first correlation coefficient. This represents the distance between the template coordinates of the two key points. Indicates the first preset weight; Based on the first correlation coefficient, the true correlation coefficient between the two key points is calculated.

8. The method according to claim 7, characterized in that, Before calculating the true correlation coefficient between the two key points based on the first correlation coefficient, the method further includes: According to the second preset formula, the second correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points; wherein, the second preset formula indicates that the second part and the first part have a physiological constraint relationship; The second preset formula is: This represents the second correlation coefficient. This represents the distance between the template coordinates of the two key points. Indicates the second preset weight; The step of calculating the true correlation coefficient between the two key points based on the first correlation coefficient includes: Calculate the average of the first correlation coefficient and the second correlation coefficient to obtain the true correlation coefficient between the two key points.

9. The method according to claim 5, characterized in that, The acquisition of the second sample image includes: Obtain the third sample image; Each third sample image is mapped to a unified image template to obtain a second sample image; wherein the distance between preset parts of a specified object in each second sample image is less than a preset threshold.

10. The method according to claim 1, characterized in that, Before calculating the first loss value based on the predicted correlation coefficient and the actual correlation coefficient, the method further includes: Obtain the current weight matrix of a specified network layer in the second keypoint localization model as the true weight matrix; wherein, the second keypoint localization model is trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: the feature of the horizontal coordinate of a keypoint or the feature of the vertical coordinate of a keypoint; For each pair of key points, the correlation coefficient between the two key points is calculated based on the columns corresponding to the two key points in the true weight matrix, and this coefficient is used as the true correlation coefficient.

11. The method according to claim 10, characterized in that, The true correlation coefficient between any two key points includes: the true correlation coefficient between the x-coordinates of the two key points and / or the true correlation coefficient between the y-coordinates of the two key points.

12. The method according to claim 11, characterized in that, The true correlation coefficient between the x-coordinates of any two key points is obtained by processing the columns corresponding to the x-coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm. The true correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm.

13. The method according to claim 11, characterized in that, The calculation of the first loss value based on the predicted correlation coefficient and the actual correlation coefficient includes: If the true correlation coefficient between any two key points includes both the true correlation coefficient between their x-coordinates and their y-coordinates, and the predicted correlation coefficient between any two key points includes both the predicted correlation coefficient between their x-coordinates and their y-coordinates, then a first loss value is calculated based on the first sub-loss value and the second sub-loss value. The first sub-loss value represents the difference between the predicted correlation coefficient between the x-coordinates and the true correlation coefficient between the x-coordinates; the second sub-loss value represents the difference between the predicted correlation coefficient between the y-coordinates and the true correlation coefficient between the y-coordinates. If the true correlation coefficient between any two key points is the same as the true correlation coefficient between the x-coordinates of the two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the x-coordinates of the two key points, then the first sub-loss value is obtained as the first loss value. If the true correlation coefficient between any two key points is the same as the true correlation coefficient between the ordinates of the two key points, and the predicted correlation coefficient between any two key points is the same as the predicted correlation coefficient between the ordinates of the two key points, then the second sub-loss value is obtained as the first loss value.

14. The method according to any one of claims 1-13, characterized in that, The key points of the specified object include at least one of the following: facial key points, finger key points, and human posture key points.

15. A key point localization method, characterized in that, The method includes: Get the image to be detected that contains the specified object; The image to be detected is input into a trained third keypoint localization model to obtain the coordinates of key points of a specified object in the image to be detected; wherein the third keypoint localization model is trained according to any one of the methods described in claims 1-14.

16. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-14 or 15.

Images