Key point positioning model training method, key point positioning method and electronic equipment

Abstract

The embodiment of the invention provides a key point positioning model training method, a key point positioning method and electronic equipment, and relates to the technical field of deep learning. The method comprises the following steps: inputting a first sample image containing a specified object into a to-be-trained first key point positioning model to obtain predicted coordinates representing key points of the specified object in the first sample image; obtaining a current prediction weight matrix of a specified network layer in a to-be-trained first key point positioning model; for every two key points, based on columns corresponding to the two key points in the prediction weight matrix, calculating a prediction correlation coefficient between the two key points; calculating a first loss value based on the predicted correlation coefficient and the real correlation coefficient; and based on the first loss value and the second loss value, adjusting the model parameters until a preset convergence condition is reached, and obtaining a trained first key point positioning model. Therefore, a key point positioning model with relatively high accuracy can be obtained to improve the accuracy of a positioning result.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a key point positioning model training method, a key point positioning method and an electronic device. Background Art

[0002] In the field of deep learning technology, the key points of a specified object in an image can be located using a trained key point localization model, that is, the coordinates of each key point in the image can be obtained. For example, common key point localization models include: face key point localization model, finger key point localization model, and human body posture key point localization model.

[0003] However, in the process of training the key point location model of the initial structure, the designated object in the sample image used is usually in a non-designated state, resulting in the model locating the key points in the image containing the designated object in the designated state, and the positioning result obtained is not accurate. For example, in the scenario of training the key point location model of the face, the non-designated state means that the eyes of the person are both open or closed, and correspondingly, the designated state means that one eye is open and the other eye is closed.

[0004] Therefore, how to obtain a key point positioning model with higher accuracy to improve the accuracy of positioning results has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a key point positioning model training method, a key point positioning method and an electronic device to obtain a key point positioning model with high accuracy to improve the accuracy of the positioning result. The specific technical solution is as follows:

[0006] In a first aspect of an embodiment of the present application, a key point positioning model training method is first provided, the method comprising:

[0007] Inputting a first sample image containing a specified object into a first key point positioning model to be trained, and obtaining coordinates of key points representing the specified object in the first sample image as predicted coordinates;

[0008] Obtaining the current weight matrix of the specified network layer in the first key point positioning model to be trained as a prediction weight matrix; wherein any column in the prediction weight matrix represents: a feature of the horizontal coordinate of a key point, or a feature of the vertical coordinate of a key point;

[0009] For every two key points, based on the columns corresponding to the two key points in the prediction weight matrix, calculate the correlation coefficient between the two key points as the prediction correlation coefficient;

[0010] Calculating a first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the true correlation coefficient is determined based on the true coordinates of the key point of the specified object in the second sample image;

[0011] Based on the first loss value and the second loss value, the model parameters of the first key point positioning model to be trained are adjusted until a preset convergence condition is reached, thereby obtaining a trained first key point positioning model; wherein the second loss value is determined based on the predicted coordinates and the real coordinates of the key point of the specified object in the first sample image.

[0012] In some embodiments, the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points, and / or the predicted correlation coefficient between the vertical coordinates of the two key points.

[0013] In some embodiments, the prediction correlation coefficient between the horizontal coordinates of any two key points is obtained by processing the columns corresponding to the horizontal coordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm;

[0014] The prediction correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0015] In some embodiments, the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image and in combination with the physiological constraint relationship of the part to which the key points in the specified object belong.

[0016] In some embodiments, before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient, the method further includes:

[0017] acquiring the second sample image;

[0018] For each key point, calculate the mean of the horizontal coordinate and the mean of the vertical coordinate of the real coordinates of the key point in each second sample image as the template coordinates of the key point;

[0019] For every two key points, the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object.

[0020] In some embodiments, for each two key points, based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object, calculating the true correlation coefficient between the two key points includes:

[0021] For each two key points, if there is no physiological constraint relationship between the parts to which the two key points belong in the specified object, then the true correlation coefficient between the two key points is determined to be 0;

[0022] If the parts to which the two key points belong in the specified object have a physiological constraint relationship, and the distance between the template coordinates of the two key points is not less than a preset distance, then determining that the true correlation coefficient between the two key points is 0;

[0023] If the parts to which the two key points belong in the specified object have a physiological constraint relationship, and the distance between the template coordinates of the two key points is less than the preset distance, then the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points; wherein the calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

[0024] In some embodiments, the calculating the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points includes:

[0025] According to a first preset formula, a first correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points; wherein the first preset formula indicates that a first part of the parts to which the two key points belong and a second part of the parts to which the two key points belong have a physiological constraint relationship;

[0026] The first preset formula is:

[0027] R1=1-d×Z1

[0028] R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight;

[0029] Based on the first correlation coefficient, a true correlation coefficient between the two key points is calculated.

[0030] In some embodiments, before calculating the true correlation coefficient between the two key points based on the first correlation coefficient, the method further includes:

[0031] According to a second preset formula, a second correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points; wherein the second preset formula indicates that there is a physiological constraint relationship between the second part and the first part;

[0032] The second preset formula is:

[0033] R2=1-d×Z2

[0034] R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight;

[0035] The calculating the true correlation coefficient between the two key points based on the first correlation coefficient includes:

[0036] An average value of the first correlation coefficient and the second correlation coefficient is calculated to obtain a true correlation coefficient between the two key points.

[0037] In some embodiments, acquiring the second sample image includes:

[0038] acquiring a third sample image;

[0039] Each third sample image is mapped to a unified image template to obtain a second sample image; wherein the distance between the preset parts of the specified object in each second sample image is less than a preset threshold.

[0040] In some embodiments, before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient, the method further includes:

[0041] Obtaining the current weight matrix of the specified network layer in the second key point positioning model as the true weight matrix; wherein the second key point positioning model is: obtained by training based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: a feature of the horizontal coordinate of a key point, or a feature of the vertical coordinate of a key point;

[0042] For every two key points, based on the columns corresponding to the two key points in the true weight matrix, a correlation coefficient between the two key points is calculated as the true correlation coefficient.

[0043] In some embodiments, the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points, and / or the true correlation coefficient between the vertical coordinates of the two key points.

[0044] In some embodiments, the true correlation coefficient between the horizontal coordinates of any two key points is obtained by processing the columns corresponding to the horizontal coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm;

[0045] The true correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm.

[0046] In some embodiments, the calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient includes:

[0047] If the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, then based on the first sub-loss value and the second sub-loss value, calculate the first loss value; wherein the first sub-loss value represents: the difference between the predicted correlation coefficient between the horizontal coordinates and the true correlation coefficient between the horizontal coordinates; the second sub-loss value represents: the difference between the predicted correlation coefficient between the vertical coordinates and the true correlation coefficient between the vertical coordinates;

[0048] If the true correlation coefficient between any two key points is the true correlation coefficient between the horizontal coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the horizontal coordinates of the two key points, then obtaining the first sub-loss value as the first loss value;

[0049] If the true correlation coefficient between any two key points is the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the vertical coordinates of the two key points, then the second sub-loss value is obtained as the first loss value.

[0050] In some embodiments, the key points of the designated object include at least one of the following: face key points, finger key points, and human body posture key points.

[0051] A second aspect of an embodiment of the present application provides a key point positioning method, the method comprising:

[0052] Get the image to be detected containing the specified object;

[0053] The image to be detected is input into a trained third key point positioning model to obtain the coordinates of the key points of the specified object in the image to be detected; wherein the third key point positioning model is trained according to any of the above-mentioned key point positioning model training methods.

[0054] In a third aspect of an embodiment of the present application, a key point positioning model training device is provided, the device comprising:

[0055] A prediction coordinate acquisition module, used for inputting a first sample image containing a specified object into a first key point positioning model to be trained, and obtaining coordinates of key points representing the specified object in the first sample image as prediction coordinates;

[0056] A prediction weight matrix acquisition module is used to obtain the current weight matrix of the specified network layer in the first key point positioning model to be trained as a prediction weight matrix; wherein any column in the prediction weight matrix represents: a feature of the horizontal coordinate of a key point, or a feature of the vertical coordinate of a key point;

[0057] A prediction correlation coefficient acquisition module, used for calculating, for each two key points, the correlation coefficient between the two key points based on the columns corresponding to the two key points in the prediction weight matrix as the prediction correlation coefficient;

[0058] A first loss value acquisition module, used to calculate a first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the true correlation coefficient is determined based on the true coordinates of the key point of the specified object in the second sample image;

[0059] A model parameter adjustment module is used to adjust the model parameters of the first key point positioning model to be trained based on the first loss value and the second loss value until a preset convergence condition is reached to obtain a trained first key point positioning model; wherein the second loss value is determined based on the predicted coordinates and the real coordinates of the key point of the specified object in the first sample image.

[0060] In some embodiments, the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points, and / or the predicted correlation coefficient between the vertical coordinates of the two key points.

[0061] In some embodiments, the predicted correlation coefficient between the horizontal coordinates of any two key points is: based on a preset correlation coefficient algorithm, the columns corresponding to the horizontal coordinates of the two key points in the prediction weight matrix are processed; the predicted correlation coefficient between the vertical coordinates of any two key points is: based on a preset correlation coefficient algorithm, the columns corresponding to the vertical coordinates of the two key points in the prediction weight matrix are processed.

[0062] In some embodiments, the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image and in combination with the physiological constraint relationship of the part to which the key points in the specified object belong.

[0063] In some embodiments, the device also includes: a second sample image acquisition module, used to acquire the second sample image before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient; a template coordinate acquisition module, used to calculate, for each key point, the mean of the horizontal coordinate and the mean of the vertical coordinate in the true coordinates of the key point in each second sample image as the template coordinates of the key point; a first true correlation coefficient acquisition module, used to calculate, for each two key points, the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object.

[0064] In some embodiments, the first true correlation coefficient acquisition module includes: a first condition determination submodule, for determining, for each two key points, that the true correlation coefficient between the two key points is 0 if there is no physiological constraint relationship between the parts to which the two key points belong in the specified object; a second condition determination submodule, for determining that the true correlation coefficient between the two key points is 0 if there is a physiological constraint relationship between the parts to which the two key points belong in the specified object and the distance between the template coordinates of the two key points is not less than a preset distance; a third condition determination submodule, for calculating the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points if there is a physiological constraint relationship between the parts to which the two key points belong in the specified object and the distance between the template coordinates of the two key points is less than a preset distance; wherein the calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

[0065] In some embodiments, the third condition determination submodule includes: a first correlation coefficient calculation unit, which is used to calculate the first correlation coefficient between the two key points according to a first preset formula using the distance between the template coordinates of the two key points; wherein the first preset formula indicates that there is a physiological constraint relationship between the first part of the part to which the two key points belong and the second part of the part to which they belong; the first preset formula is: R1=1-d×Z1, R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight; a true correlation coefficient calculation unit, which is used to calculate the true correlation coefficient between the two key points based on the first correlation coefficient.

[0066] In some embodiments, the device also includes: a second correlation coefficient calculation module, which is used to calculate the second correlation coefficient between the two key points according to a second preset formula using the distance between the template coordinates of the two key points before calculating the true correlation coefficient between the two key points based on the first correlation coefficient; wherein the second preset formula indicates that there is a physiological constraint relationship between the second part and the first part; the second preset formula is: R2=1-d×Z2, R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight; the true correlation coefficient calculation unit is specifically used to calculate the average of the first correlation coefficient and the second correlation coefficient to obtain the true correlation coefficient between the two key points.

[0067] In some embodiments, the second sample image acquisition module includes: a third sample image acquisition submodule, used to acquire a third sample image; a mapping submodule, used to map each third sample image to a unified image template to obtain a second sample image; wherein the distance between the preset parts of the specified object in each second sample image is less than a preset threshold.

[0068] In some embodiments, the device also includes: a true weight matrix acquisition module, which is used to obtain the current weight matrix of the specified network layer in the second key point positioning model as the true weight matrix before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the second key point positioning model is: trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: the feature of the horizontal coordinate of a key point, or, the feature of the vertical coordinate of a key point; the second true correlation coefficient acquisition module is used to calculate the correlation coefficient between the two key points for every two key points based on the columns corresponding to the two key points in the true weight matrix, as the true correlation coefficient.

[0069] In some embodiments, the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points, and / or the true correlation coefficient between the vertical coordinates of the two key points.

[0070] In some embodiments, the true correlation coefficient between the horizontal coordinates of any two key points is: based on a preset correlation coefficient algorithm, the columns corresponding to the horizontal coordinates of the two key points in the real weight matrix are processed; the true correlation coefficient between the vertical coordinates of any two key points is: based on a preset correlation coefficient algorithm, the columns corresponding to the vertical coordinates of the two key points in the real weight matrix are processed.

[0071] In some embodiments, the first loss value acquisition module is specifically used to: if the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, then based on the first sub-loss value and the second sub-loss value, calculate the first loss value; wherein the first sub-loss value represents: the difference between the predicted correlation coefficient between the horizontal coordinates and the true correlation coefficient between the horizontal coordinates; the second sub-loss value Represents: the difference between the predicted correlation coefficient between the vertical coordinates and the true correlation coefficient between the vertical coordinates; if the true correlation coefficient between any two key points is the true correlation coefficient between the horizontal coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the horizontal coordinates of the two key points, then the first sub-loss value is obtained as the first loss value; if the true correlation coefficient between any two key points is the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the vertical coordinates of the two key points, then the second sub-loss value is obtained as the first loss value.

[0072] In some embodiments, the key points of the designated object include at least one of the following: face key points, finger key points, and human body posture key points.

[0073] In a fourth aspect of the embodiments of the present application, a key point positioning device is provided, the device comprising:

[0074] The module for acquiring the image to be detected is used to acquire the image to be detected containing the specified object;

[0075] A positioning module is used to input the image to be detected into a trained third key point positioning model to obtain the coordinates of the key points of the specified object in the image to be detected; wherein the third key point positioning model is trained according to any of the above-mentioned key point positioning model training methods.

[0076] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, including:

[0077] Memory, used to store computer programs;

[0078] The processor is used to implement the key point positioning model training method described in the first aspect, or the key point positioning method described in the second aspect when executing the program stored in the memory.

[0079] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the key point positioning model training method described in the first aspect, or the key point positioning method described in the second aspect.

[0080] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the key point positioning model training method described in the first aspect, or the key point positioning method described in the second aspect.

[0081] Beneficial effects of the embodiments of the present application:

[0082] An embodiment of the present application provides a key point positioning model training method, the method comprising: inputting a first sample image containing a specified object into a first key point positioning model to be trained, obtaining coordinates of key points representing the specified object in the first sample image as predicted coordinates; obtaining a current weight matrix of a specified network layer in the first key point positioning model to be trained as a predicted weight matrix; wherein any column in the predicted weight matrix represents: a feature of a horizontal coordinate of a key point, or a feature of a vertical coordinate of a key point; for every two key points, based on the columns corresponding to the two key points in the predicted weight matrix, calculating a correlation coefficient between the two key points as a predicted correlation coefficient; calculating a first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image; adjusting model parameters of the first key point positioning model to be trained based on the first loss value and the second loss value until a preset convergence condition is reached, thereby obtaining a trained first key point positioning model; wherein the second loss value is determined based on the predicted coordinates and the true coordinates of the key points of the specified object in the first sample image.

[0083] Based on the above processing, during the training process of the key point positioning model, the current weight matrix (i.e., the predicted weight matrix) of the specified network layer in the key point positioning model can be obtained. Since a column in the weight matrix can represent the characteristics of the horizontal coordinate of a key point, or the characteristics of the vertical coordinate of a key point, for any two key points of the specified object, the predicted correlation coefficient obtained based on the predicted weight matrix can reflect the correlation between the two key points currently learned by the model. And since the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image, the true correlation coefficient can reflect the true correlation between the two key points. Furthermore, the first loss value can reflect the difference between the correlation between the two key points currently learned by the model and the true correlation between the two key points.

[0084] In addition, since the second loss value is determined based on the predicted coordinates and the real coordinates of the key points of the specified object in the first sample image, the model parameters of the first key point positioning model to be trained are adjusted based on the first loss value and the second loss value, so that the trained first key point positioning model can obtain the coordinates of each key point of the specified object based on the real correlation between the key points, avoiding the deviation of the positioning results of each key point of the specified object due to incorrect correlation between the learned key points. In this way, a key point positioning model with high accuracy can be obtained to improve the accuracy of the positioning results.

[0085] For example, in a scenario where facial key points are located, the actual correlation between a facial key point belonging to the left eye (which may be referred to as the first key point) and another facial key point belonging to the right eye (which may be referred to as the second key point) may not be high. However, since the designated object contained in the sample image used in training the facial key point location model is usually in an undesignated state, for example, the first key point and the second key point are nearly symmetrical in the undesignated state, the correlation between the two key points learned by the facial key point location model may be high. Accordingly, adjusting the model parameters of the facial key point location model based on the first loss value and the second loss value can enable the trained facial key point location model to learn the actual correlation between the two key points.

[0086] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0088] Figure 1 A first flow chart of the key point positioning model training method provided in an embodiment of the present application;

[0089] FIG. 2( a ) is a schematic diagram of key points of a face provided in an embodiment of the present application;

[0090] FIG2( b ) is another schematic diagram of key points of a human face provided in an embodiment of the present application;

[0091] Figure 3 A schematic diagram of locating 68 facial key points provided in an embodiment of the present application;

[0092] Figure 4A first flow chart for obtaining a true correlation coefficient provided in an embodiment of the present application;

[0093] Figure 5 A second flow chart for obtaining a true correlation coefficient provided in an embodiment of the present application;

[0094] Figure 6 A third flow chart for obtaining a true correlation coefficient provided in an embodiment of the present application;

[0095] Figure 7 A flowchart of a training key point positioning model provided in an embodiment of the present application;

[0096] Figure 8 A first flow chart of a key point positioning method provided by an embodiment of the present invention;

[0097] Fig. 9 A flowchart for obtaining key points of a human face in a sample image containing a human face provided in an embodiment of the present application;

[0098] Fig.10 A structural diagram of a key point positioning model training device provided in an embodiment of the present application;

[0099] Fig.11 A structural diagram of a key point positioning device provided in an embodiment of the present application;

[0100] Fig.12 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0102] In the field of deep learning technology, the key points of a specified object in an image can be located using a trained key point localization model, that is, the coordinates of each key point in the image can be obtained. For example, common key point localization models include: face key point localization model, finger key point localization model, and human body posture key point localization model.

[0103] However, in the process of training the key point location model of the initial structure, the designated object in the sample image used is usually in a non-designated state, resulting in the model locating the key points in the image containing the designated object in the designated state, and the positioning result obtained is not accurate. For example, in the scenario of training the key point location model of the face, the non-designated state means that the eyes of the person are both open or closed, and correspondingly, the designated state means that one eye is open and the other eye is closed.

[0104] Therefore, how to obtain a key point positioning model with high accuracy has become an urgent problem to be solved.

[0105] The present application embodiment provides a key point positioning model training method, which can be applied to electronic devices. Figure 1 , Figure 1 The first flow chart of the key point positioning model training method provided in the embodiment of the present application comprises the following steps:

[0106] S101: Input a first sample image containing a specified object into a first key point positioning model to be trained, and obtain coordinates of key points representing the specified object in the first sample image as predicted coordinates.

[0107] S102: Obtain the current weight matrix of the specified network layer in the first key point positioning model to be trained as the prediction weight matrix.

[0108] Among them, any column in the prediction weight matrix represents: the feature of the horizontal coordinate of a key point, or the feature of the vertical coordinate of a key point.

[0109] S103: For every two key points, based on the columns corresponding to the two key points in the prediction weight matrix, calculate the correlation coefficient between the two key points as the prediction correlation coefficient.

[0110] S104: Calculate a first loss value based on the predicted correlation coefficient and the true correlation coefficient.

[0111] The true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image.

[0112] S105: Based on the first loss value and the second loss value, the model parameters of the first key point positioning model to be trained are adjusted until a preset convergence condition is reached, thereby obtaining a trained first key point positioning model.

[0113] The second loss value is determined based on the predicted coordinates and the real coordinates of the key points of the specified object in the first sample image.

[0114] Based on the above processing, during the training process of the key point positioning model, the current weight matrix (i.e., the predicted weight matrix) of the specified network layer in the key point positioning model can be obtained. Since a column in the weight matrix can represent the characteristics of the horizontal coordinate of a key point, or the characteristics of the vertical coordinate of a key point, for any two key points of the specified object, the predicted correlation coefficient obtained based on the predicted weight matrix can reflect the correlation between the two key points currently learned by the model. And since the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image, the true correlation coefficient can reflect the true correlation between the two key points. Furthermore, the first loss value can reflect the difference between the correlation between the two key points currently learned by the model and the true correlation between the two key points.

[0115] In addition, since the second loss value is determined based on the predicted coordinates and the real coordinates of the key points of the specified object in the first sample image, the model parameters of the first key point positioning model to be trained are adjusted based on the first loss value and the second loss value, so that the trained first key point positioning model can obtain the coordinates of each key point of the specified object based on the real correlation between the key points, avoiding the deviation of the positioning results of each key point of the specified object due to incorrect correlation between the learned key points. In this way, a key point positioning model with high accuracy can be obtained to improve the accuracy of the positioning results.

[0116] For example, in a scenario where facial key points are located, the actual correlation between a facial key point belonging to the left eye (which may be referred to as the first key point) and another facial key point belonging to the right eye (which may be referred to as the second key point) may not be high. However, since the designated object contained in the sample image used in training the facial key point location model is usually in an undesignated state, for example, the first key point and the second key point are nearly symmetrical in the undesignated state, the correlation between the two key points learned by the facial key point location model may be high. Accordingly, adjusting the model parameters of the facial key point location model based on the first loss value and the second loss value can enable the trained facial key point location model to learn the actual correlation between the two key points.

[0117] In the related art, a sample image containing a large number of designated objects in a designated state and a sample image containing a large number of designated objects in a non-designated state can be obtained to train the key point positioning model. However, the sample images of the existing sample set usually only contain designated objects in a non-designated state. Therefore, this method requires a large amount of annotation costs to annotate the sample images of the designated objects in the designated state, which will increase the cost of training. Compared with this method, the key point positioning model training method provided by the present application, because the model can be constrained by the real correlation coefficient, therefore, there is no need to use the first sample image containing the designated object in the designated state for training, and the first key point positioning model obtained by training can also learn the real correlation between the key points. In addition, if a second sample image contains a designated object in a non-designated state, the real correlation between the key points can be determined based on the real coordinates of the key points in the second sample image. Therefore, when obtaining the real correlation coefficient, there is no need to use the sample image containing the designated object in the designated state. In addition, it can avoid consuming a large amount of annotation costs, and it can also reduce the cost of training.

[0118] In addition, in the related art, a small number of sample images of designated objects in designated states and sample images of designated objects in non-designated states can also be obtained to train the key point positioning model. However, due to the small number of sample images, the key point positioning model obtained by training has low accuracy in positioning results when positioning sample images containing designated objects in non-designated states. Compared with this approach, the key point positioning model training method provided in the present application can adjust the model parameters based on the second loss value determined by the predicted coordinates and the true coordinates of the key points of the designated objects in the sample images during the training process of the key point positioning model, thereby ensuring the accuracy of the positioning results obtained.

[0119] The key point positioning model in the present application may be a face key point positioning model, and accordingly, the specified object represents a face. At this time, when the face is in a specified state, it may represent that the face is making an exaggerated expression. For example, when one eye of the face is closed and the other eye is open, the face is in a specified state.

[0120] The key point positioning model can also be a finger key point positioning model, and accordingly, the designated object represents a human hand. In this case, when the human hand is in the designated state, it can represent an exaggerated gesture of the fingers. For example, when the fingers simulate some animal shapes, the hand is in the designated state.

[0121] The key point positioning model can also be a human body posture key point positioning model, and accordingly, the specified object represents the human body. Accordingly, when the human body is in a specified state, it can represent that various parts of the human body present exaggerated postures. For example, when an actor performs a difficult dance move, the human body is in a specified state.

[0122] In the present application, the number of key points of a specified object is preset according to actual needs. For ease of description, each key point of a specified object corresponds to a number. For example, the numbers of the key points can be obtained from 1 in ascending order, and the largest number is the number of the key points of the specified object.

[0123] Accordingly, the part of each key point in the specified object can be determined according to the physical meaning represented by each key point. For example, for the face key point positioning model, the specified object is the face, and the parts of the specified object can be divided into: facial contour, left eyebrow, right eyebrow, nose, left eye, right eye and mouth, etc. For the finger key point positioning model, the specified object is the human hand, and the parts of the specified object can be divided into: thumb, index finger, middle finger, ring finger, little finger, etc. For the human posture key point positioning model, the specified object is the human body, and the parts of the specified object can be divided into: head, limbs, torso, etc.

[0124] Referring to Figures 2(a) and 2(b), Figure 2(a) is a schematic diagram of a facial key point provided in an embodiment of the present application. Figure 2(b) is another schematic diagram of a facial key point provided in an embodiment of the present application. For an image of a specified object including one eye open and the other eye closed, the points in Figure 2(a) represent the facial key points of the image obtained based on the first key point positioning model trained in the present application, and the points in Figure 2(b) represent the facial key points of the image obtained based on the first key point positioning model to be trained in the present application. It can be seen that the facial key points of the image obtained based on the first key point positioning model trained in the present application indicate that one eye in the face is open and the other eye is closed, which is consistent with the position of the real facial key points of the image. The facial key points obtained based on the facial key points of the image obtained based on the first key point positioning model to be trained in the present application indicate that both eyes in the face are open, which is different from the position of the real facial key points of the image.

[0125] like Figure 3 As shown, Figure 3 A schematic diagram of locating 68 facial key points provided in an embodiment of the present application. Figure 3The method of setting facial key points shown in the figure can determine 68 key points (i.e., facial key points) in an image containing a face. The points are numbered 1-68, 1-17 represent points on the facial contour, 18-22 represent points on the left eyebrow, 23-27 represent points on the right eyebrow, 28-36 represent points on the nose, 37-42 represent points on the left eye, 43-48 represent points on the right eye, and 49-68 represent points on the mouth.

[0126] For step S101, the first key point positioning model to be trained may be an untrained key point positioning model or a pre-trained key point positioning model. For example, the pre-trained key point positioning model may be: a model trained based on a sample image containing a specified object and a corresponding sample label, wherein the sample label represents the coordinates of the key point of the specified object in the corresponding sample image.

[0127] It can be understood that for any key point of a specified object in a sample image, the coordinates of the key point include: the horizontal coordinate of the key point, i.e., the value of the key point mapped to the horizontal axis of the sample image (which can be called the X axis), and the vertical coordinate of the key point, i.e., the value of the key point mapped to the vertical axis of the sample image (which can be called the Y axis). Accordingly, the coordinates of a key point can be expressed as (x, y).

[0128] The first sample image is a sample image containing a specified object. Accordingly, a sample label corresponding to the first sample image can also be obtained. The sample label corresponding to the first sample image represents the real coordinates of the key points of the specified object.

[0129] The first sample image containing the specified object is input into the first key point positioning model to be trained, and the first key point positioning model to be trained can predict the coordinates of the key points representing the specified object in the first sample image (i.e., the predicted coordinates). Figure 3 In the scenario shown, the first sample image is a sample image containing a face, and the first key point positioning model is a face key point positioning model. According to step S101, the coordinates of 68 face key points can be obtained.

[0130] For step S102, the designated network layer can be the output layer of the first key point positioning model to be trained, or it can be other network layers, as long as its weight matrix can reflect the characteristics of the horizontal coordinates and the vertical coordinates of each key point. All elements in the weight matrix represent the weights of the designated network layer. In the present application, the key point positioning model is a model based on coordinate regression, that is, the key point positioning model usually uses a fully connected layer as the output layer.

[0131] Since the coordinates of a key point include the horizontal coordinate and the vertical coordinate of the key point, one column of the weight matrix of the output layer corresponds to the horizontal coordinate of a key point or the vertical coordinate of a key point. Therefore, if the number of key points of the specified object is N, the output dimension of the output layer is 2N. And since the output dimension of the output layer is consistent with the number of columns of the weight matrix of the output layer, that is, the weight matrix of the output layer has 2N columns. Accordingly, any column in the weight matrix represents: the feature of the horizontal coordinate of a key point, or the feature of the vertical coordinate of a key point.

[0132] In addition, the number of rows of the weight matrix of the output layer is consistent with the output dimension of the previous network layer of the output layer. For the convenience of description, the number of rows of the weight matrix of the output layer can be represented by M. That is, the weight matrix of the output layer is an M×2N matrix, and any column in the weight matrix is ​​an M×1 column vector.

[0133] It is understandable that, during the training process of the key point positioning model, the above S101-S105 represents a process of adjusting the model parameters. During the training process using sample images, since the model parameters of the key point positioning model are constantly adjusted, that is, the weight matrix of the specified network layer of the key point positioning model may also change accordingly. Therefore, the current weight matrix (i.e., the predicted weight matrix) of the specified network layer can be obtained, and the process of adjusting the model parameters can be completed based on the obtained current weight matrix.

[0134] With respect to step S103 , for any key point, the column corresponding to the abscissa of the key point and the column corresponding to the ordinate of the key point can be obtained in the prediction weight matrix.

[0135] In some embodiments, the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points, and / or the predicted correlation coefficient between the vertical coordinates of the two key points.

[0136] In one implementation, for every two key points, the prediction correlation coefficient between the horizontal coordinates of the two key points can be calculated as the prediction correlation coefficient between the two key points. Further, the prediction correlation coefficient between every two key points can be obtained.

[0137] Correspondingly, the prediction correlation coefficient between the horizontal coordinates of any two key points is obtained by processing the columns corresponding to the horizontal coordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0138] In the embodiment of the present application, the preset correlation coefficient algorithm may be: a linear correlation coefficient algorithm, a Kendall correlation coefficient algorithm, a Spearman correlation coefficient algorithm, and the like.

[0139] For each two key points, when calculating the prediction correlation coefficient between the horizontal coordinates of the two key points, the columns corresponding to the horizontal coordinates of the two key points can be obtained in the prediction weight matrix. Then, based on the preset correlation coefficient algorithm, the columns corresponding to the obtained horizontal coordinates are processed to obtain the prediction correlation coefficient between the horizontal coordinates of the two key points.

[0140] For example, in the above process of calculating the prediction correlation coefficient between the horizontal coordinates of face key point 1 and face key point 2, the column corresponding to the horizontal coordinate of face key point 1 (which can be expressed as C1) and the column corresponding to the horizontal coordinate of face key point 2 (which can be expressed as C2) can be obtained in the prediction weight matrix. At this time, the preset correlation coefficient algorithm is a linear correlation coefficient algorithm.

[0141] Correspondingly, the predicted correlation coefficient R between the horizontal coordinates of face key point 1 and face key point 2 can be calculated by formula (1):

[0142]

[0143] Among them, C1 i Represents the i-th element in the column (i.e. C1) corresponding to the horizontal coordinate of facial key point 1, represents the mean of each element in C1 (i.e. the mean of C1), C2 i Represents the i-th element in the column (i.e. C2) corresponding to the horizontal coordinate of facial key point 2, represents the mean of each element in C2 (i.e., the mean of C2), and M represents the number of rows of the weight matrix of the output layer.

[0144] In this way, the correlation between the two key points can be measured based on the calculated prediction correlation coefficient between the two key points.

[0145] When the prediction correlation coefficient between two key points only includes the prediction correlation coefficient between the horizontal coordinates of the two key points, a prediction correlation matrix (which may be called an X-axis prediction correlation matrix) may be obtained based on the prediction correlation coefficient between the horizontal coordinates of every two key points.

[0146] Among them, one row of the X-axis prediction correlation matrix corresponds to one key point, and one column of the X-axis prediction correlation matrix also corresponds to one key point. And in the X-axis prediction correlation matrix, the number of key points corresponding to each column in the matrix in the order of arrangement from left to right is consistent with the number of key points corresponding to each row in the matrix in the order of arrangement from top to bottom. Correspondingly, the number of rows and columns of the X-axis prediction correlation matrix is ​​consistent with the number of columns, both of which are the number of key points. The element in the i-th row and j-th column of the X-axis prediction correlation matrix represents: the predicted correlation coefficient between the horizontal coordinates of the key point corresponding to the i-th row and the key point corresponding to the j-th row. It can be understood that based on the preset correlation coefficient algorithm, the value of the element in the i-th row and j-th column of the X-axis prediction correlation matrix is ​​consistent with the value of the element in the j-th row and i-th column of the X-axis prediction correlation matrix, that is, the X-axis prediction correlation matrix is ​​a symmetric matrix. When i=j, the value of the element is 1, that is, the values ​​of the elements on the main diagonal of the X-axis prediction correlation matrix are all 1.

[0147] For example, in the above Figure 3 In the scenario shown, the obtained X-axis prediction correlation matrix is ​​a 68×68 matrix, wherein the i-th row and the i-th column of the matrix correspond to the facial key point i, and the value range of i is 1 to 68.

[0148] In another implementation, for every two key points, the prediction correlation coefficient between the ordinates of the two key points can also be calculated as the prediction correlation coefficient between the two key points. Further, the prediction correlation coefficient between every two key points can be obtained.

[0149] Correspondingly, the prediction correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

[0150] In the embodiment of the present application, the process of calculating the predicted correlation coefficient between the vertical coordinates of any two key points is similar to the process of calculating the predicted correlation coefficient between the horizontal coordinates of any two key points, and will not be repeated here.

[0151] When the prediction correlation coefficient between two key points only includes the prediction correlation coefficient between the vertical coordinates of the two key points, a prediction correlation matrix (which may be called a Y-axis prediction correlation matrix) may be obtained based on the prediction correlation coefficient between the vertical coordinates of every two key points.

[0152] Among them, the description of the Y-axis prediction correlation matrix is ​​similar to the above-mentioned X-axis prediction correlation matrix and will not be repeated here.

[0153] In another implementation, for every two key points, the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points can also be calculated, and the predicted correlation coefficient between the two key points includes the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points. Thus, the predicted correlation coefficient between every two key points can be obtained.

[0154] In the embodiment of the present application, the process of calculating the predicted correlation coefficient between the vertical coordinates of any two key points, and the process of calculating the predicted correlation coefficient between the horizontal coordinates of any two key points can refer to the above embodiment and will not be repeated here.

[0155] When the prediction correlation coefficient between two key points includes the prediction correlation coefficient between the horizontal coordinates of the two key points and the prediction correlation coefficient between the vertical coordinates of the two key points, an X-axis prediction correlation matrix and a Y-axis prediction correlation matrix can be obtained.

[0156] The process of obtaining the X-axis prediction correlation matrix and the Y-axis prediction correlation matrix may refer to the above embodiment and will not be described in detail here.

[0157] For example, the two key points on the left eye of the face key points can be called L1 and L2. The two key points on the right eye can be called R1 and R2. If the prediction correlation coefficient between two key points only includes the prediction correlation coefficient between the ordinates of the two key points, in the scenarios of Figure 2(a) and Figure 2(b), for the first key point positioning model corresponding to Figure 2(a), the corresponding partial matrices of L1, L2, R1 and R2 in the X-axis prediction correlation matrix are shown in Matrix (1):

[0158]

[0159] For example, the value (1) of the first row and first column of the matrix (1) is the predicted correlation coefficient between L1 and L1, the value (0.9083) of the first row and second column is the predicted correlation coefficient between L1 and L2, the value (0.1497) of the first row and third column is the predicted correlation coefficient between L1 and R1, and the value (0.1526) of the first row and fourth column is the predicted correlation coefficient between L1 and R2. The predicted correlation coefficients represented by the elements in other positions of the matrix (1) are similar to the predicted correlation coefficients represented by the elements in the first row, and are not described here.

[0160] For the first key point positioning model to be trained corresponding to FIG2(b), the corresponding partial matrices of L1, L2, R1 and R2 in the obtained X-axis prediction correlation matrix are shown in matrix (2):

[0161]

[0162] Among them, the value (1) of the element in the first row and first column of the matrix (1) is: the predicted correlation coefficient between L1 and L1, the value (0.9662) of the element in the first row and second column is: the predicted correlation coefficient between L1 and L2, the value (0.8227) of the element in the first row and third column is: the predicted correlation coefficient between L1 and R1, and the value (0.7875) of the element in the first row and fourth column is: the predicted correlation coefficient between L1 and R2. The predicted correlation coefficients represented by the elements in other positions of the matrix (2) are similar to the predicted correlation coefficients represented by the elements in the first row, and are not repeated here.

[0163] It can be seen that the predicted correlation coefficient between L1 and R1 in matrix (1) is much smaller than the predicted correlation coefficient between L1 and R1 in matrix (2), the predicted correlation coefficient between L1 and R2 in matrix (1) is much smaller than the predicted correlation coefficient between L1 and R2 in matrix (2), the predicted correlation coefficient between L2 and R1 in matrix (1) is much smaller than the predicted correlation coefficient between L2 and R1 in matrix (2), and the predicted correlation coefficient between L2 and R2 in matrix (1) is much smaller than the predicted correlation coefficient between L2 and R2 in matrix (2). In real scenes, the correlation between the key points on the left eye and the key points on the right eye among the key points of the face is low, so the real correlation coefficient between L1 and R1, the real correlation coefficient between L1 and R2, the real correlation coefficient between L2 and R1, and the real correlation coefficient between L2 and R2 are also low, that is, the first key point positioning model trained by the present application can learn the real correlation between every two key points.

[0164] Based on the above processing, when the predicted correlation coefficient between two key points includes the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, the predicted correlation coefficient can reflect the correlation between the horizontal coordinates of the two key points and the correlation between the vertical coordinates of the two key points currently learned by the model. In this case, obtaining the first loss value can more accurately reflect the difference between the correlation between the two key points currently learned by the model and the actual correlation between the two key points. In this way, a key point positioning model with higher accuracy can be further obtained to improve the accuracy of the positioning results.

[0165] When the predicted correlation coefficient between two key points only includes the predicted correlation coefficient between the horizontal coordinates of the two key points, or only includes the predicted correlation coefficient between the vertical coordinates of the two key points, the amount of calculation required to obtain each predicted correlation coefficient is low, and correspondingly, the amount of calculation required to obtain the first loss value is also low. In this way, the training efficiency of the key point positioning model can be improved.

[0166] With respect to steps S104-S105, for any two key points of the specified object, since the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image, the true correlation coefficient can reflect the true correlation between the two key points.

[0167] After obtaining the true correlation coefficient between every two key points, a true correlation matrix can also be obtained based on the true correlation coefficient between every two key points. The specific process of obtaining the true correlation coefficient will be described in the subsequent embodiments.

[0168] Furthermore, a first loss value may be calculated based on a preset loss function (which may be referred to as a first loss function) according to the predicted correlation coefficient and the true correlation coefficient. For example, the first loss value may be calculated based on the difference between the predicted correlation matrix and the true correlation matrix based on a preset loss function. The first loss function may be an L2 loss function, an L1 loss function, or the like.

[0169] In addition, the second loss value may be calculated based on a preset loss function (which may be referred to as a second loss function) according to the predicted coordinates and the real coordinates of the key points of the specified object in the first sample image. The second loss function may be an L2 loss function, an L1 loss function, or the like.

[0170] Then, based on the first loss value and the second loss value, the model parameters of the first key point positioning model to be trained are adjusted until the preset convergence condition is reached, and the trained first key point positioning model is obtained. For example, the weighted sum of the first loss value and the second loss value can be calculated as the total loss value. The model parameters of the first key point positioning model are adjusted in a gradient descent manner.

[0171] In this application, the true correlation coefficient between every two key points can be obtained by any of the following methods:

[0172] Method 1: In some embodiments, the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image and in combination with the physiological constraint relationship of the part to which the key points in the specified object belong.

[0173] In the embodiment of the present application, for any two key points of a specified object, the true correlation coefficient between the two key points can be obtained according to the parts to which the two key points respectively belong and the true coordinates of the two key points in the second sample image.

[0174] It can be understood that if there is no physiological constraint relationship between the parts to which the two key points belong, it means that the two key points are unrelated.

[0175] For example, when the specified object is a human face, there is no physiological constraint relationship between the facial contour and the nose. For example, usually, when a person speaks and the facial contour changes, the position of the nose is often not affected. Therefore, it can be determined that a key point belonging to the facial contour is not related to another key point belonging to the nose.

[0176] For another example, when the designated object is a human hand, there is no physiological constraint relationship between the thumb and other fingers. For example, usually, when the thumb is in a bent state, the other fingers can be in an extended state or in a bent state. Therefore, it can be determined that a key point belonging to the thumb is not related to another key point belonging to another finger.

[0177] For another example, when the designated object is a human hand, there is a physiological constraint relationship between the front joint point and the back joint point on the thumb. For example, usually, when the thumb is in a bent state, the front joint point will protrude and the back joint point will bend and fold.

[0178] Correspondingly, for two unrelated key points, it can be determined that the true correlation coefficient between the two key points is 0.

[0179] If the parts to which the two key points belong have a physiological constraint relationship, it means that the two key points are related.

[0180] For example, when the specified object is a human face, there is a physiological constraint relationship between the left eyebrow and the left eye. For example, usually, when the left eye changes from an open state to a closed state, the position of the left eyebrow tends to change as well. Therefore, it can be determined that a key point belonging to the left eyebrow is related to another key point belonging to the left eye.

[0181] For another example, when the designated object is a human hand, the little finger and the ring finger have a physiological constraint relationship. For example, usually, when the little finger is in a bent state, the ring finger is also in a bent state. Therefore, it can be determined that a key point belonging to the little finger is related to another key point belonging to the ring finger.

[0182] Correspondingly, the process of determining the true correlation coefficient between the two key points based on the true coordinates of the key points of the designated object in the second sample image may refer to the subsequent embodiments.

[0183] Based on the above processing, since the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image and combined with the physiological constraint relationship of the part to which the key points in the specified object belong, the true correlation coefficient between the two key points can reflect the correlation between the physiological positions of the specified objects represented by the two key points. In this way, the key point positioning model can learn the true correlation between the two key points.

[0184] Accordingly, see Figure 4 , Figure 4 The first flow chart for obtaining the true correlation coefficient provided in the embodiment of the present application includes:

[0185] S401: Acquire a second sample image.

[0186] S402: For each key point, calculate the mean of the horizontal coordinate and the mean of the vertical coordinate in the real coordinates of the key point in each second sample image as the template coordinates of the key point.

[0187] S403: For every two key points, based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object, calculate the true correlation coefficient between the two key points.

[0188] In the embodiment of the present application, the second sample image is a sample image containing a specified object.

[0189] It is understandable that since the area percentage and posture of the specified object in the sample images in the existing sample set may be different, after obtaining the sample images in the sample set, the sample images can be aligned to obtain the aligned sample images (i.e., the second sample images).

[0190] In some embodiments, step S401 includes: acquiring third sample images; and mapping each third sample image to a unified image template to obtain a second sample image.

[0191] The distance between the preset parts of the designated objects in each second sample image is smaller than a preset threshold.

[0192] In an embodiment of the present application, a third sample image of a specified object can be obtained, and for each third sample image, the third sample image can be mapped to a unified image template so that the distance between the preset parts of the specified object in each mapped third sample image is less than a preset threshold. For example, when the specified object is a face, the preset parts can be the left eye, the right eye and the mouth. For example, the positions of the eyes in each mapped third sample image are roughly the same, and the positions of the mouths in each mapped third sample image are roughly the same. The unified image template can have a specific preset size, for example, it can be 224×224. In addition, in the process of mapping the third sample image to the unified image template, the third sample image can also be cropped, parallelized and / or linearly mapped, etc., so that the mapped third sample image meets the conditions described above.

[0193] Based on the above processing, the area proportion and posture of the specified object in the third sample image can be adjusted so that the area proportion and posture of the specified object in the obtained second sample image are more conducive to the key point positioning model for processing, further improving the accuracy of the key point positioning model.

[0194] In steps S402-S403, for each key point of the designated object, the mean of the abscissa and the mean of the ordinate in the real coordinates of the key point in each second sample image may be calculated as the template coordinates of the key point.

[0195] For example, in the above Figure 3 In the scenario shown, for each facial key point, such as facial key point 1, the horizontal coordinate of the real coordinate of the facial key point 1 in each second sample image can be obtained. Then, the mean of each obtained horizontal coordinate can be calculated as the value of the horizontal coordinate in the template coordinate of the facial key point 1. Similarly, the mean of each obtained vertical coordinate can be calculated as the value of the vertical coordinate in the template coordinate of the facial key point 1. Then, the template coordinate of each facial key point can be obtained.

[0196] Furthermore, for every two key points, the distance between the template coordinates of the two key points can be calculated. Generally, the larger the distance between the template coordinates of the two key points, the larger the distance between the two key points in the physiological position of the specified object, and correspondingly, the lower the correlation between the two key points. Therefore, the true correlation coefficient between the two key points can be calculated based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object. The specific calculation process will be described in the subsequent embodiments.

[0197] Based on the above processing, the template coordinates of each key point can represent the average level of each specified object in each second sample image. Based on the template coordinates of each key point, the true correlation coefficient between every two key points is calculated, which can avoid the interference of individual differences of the specified objects in the sample image on the calculation results, and thus a key point positioning model with higher accuracy can be obtained to improve the accuracy of the positioning results.

[0198] In some embodiments, see Figure 5 , Figure 5 The second flow chart for obtaining the true correlation coefficient provided in the embodiment of the present application is as follows: Figure 4 On the basis of, step S403 comprises:

[0199] S4031: For every two key points, if there is no physiological constraint relationship between the parts to which the two key points belong in the specified object, then the true correlation coefficient between the two key points is determined to be 0.

[0200] S4032: If the parts to which the two key points belong in the specified object have a physiological constraint relationship, and the distance between the template coordinates of the two key points is not less than a preset distance, then determine that the true correlation coefficient between the two key points is 0.

[0201] S4033: If the parts to which the two key points belong in the specified object have a physiological constraint relationship, and the distance between the template coordinates of the two key points is less than a preset distance, then based on the distance between the template coordinates of the two key points, the true correlation coefficient between the two key points is calculated.

[0202] The calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

[0203] In the embodiment of the present application, irrelevant parts of the specified object can be determined in advance. For each two key points, if the parts to which the two key points belong in the specified object are irrelevant, the true correlation coefficient between the two key points can be determined to be 0.

[0204] For example, based on physiological prior knowledge, when the specified object is a human face, it can be determined that the facial contour and the nose in various parts of the face are not related to each other. Figure 3 In the scenario shown, for facial key point 1 and facial key point 28, since facial key point 1 belongs to the facial contour and facial key point 28 belongs to the nose, it can be directly determined that the true correlation coefficient between facial key point 1 and facial key point 28 is 0.

[0205] If the parts to which the two key points belong in the specified object are correlated, further, the true correlation coefficient between the two key points can be determined according to the distance between the template coordinates of the two key points.

[0206] If the distance between the template coordinates of the two key points is not less than the preset distance, it indicates that the distance between the template coordinates of the two key points is far, and although the parts to which the two key points belong are related, the two key points are not related to each other. At this time, it can be determined that the true correlation coefficient between the two key points is 0. The preset distance may be determined according to the parts to which the two key points belong.

[0207] If the distance between the template coordinates of the two key points is less than the preset distance, it indicates that the distance between the template coordinates of the two key points is close, and the two key points are correlated. At this time, the true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points. Furthermore, the true correlation coefficient between the two key points can be calculated based on the distance between the template coordinates of the two key points. The specific calculation process will be described in the subsequent embodiments.

[0208] Based on the above processing, for every two key points, it can be determined in advance whether the parts to which the two key points belong are related. If the parts to which the two key points belong are not related, it can be directly determined that the two key points are not related. This can reduce the amount of computation consumed in the training process of the key point positioning model and improve the training efficiency.

[0209] In addition, if the parts to which the two key points belong are related, it can be determined whether the distance between the template coordinates of the two key points is less than the preset distance. If not, it means that the distance between the template coordinates of the two key points is far and the two key points are not related. This can further reduce the amount of computation consumed in the training process of the key point positioning model and improve the training efficiency.

[0210] In some embodiments, S4033:

[0211] Step 1: According to a first preset formula, a first correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points.

[0212] The first preset formula indicates that there is a physiological constraint relationship between the first part of the parts to which the two key points belong and the second part of the parts to which they belong. The first preset formula is:

[0213] R1=1-d×Z1

[0214] R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight.

[0215] Step 2: Based on the first correlation coefficient, calculate the true correlation coefficient between the two key points.

[0216] In the embodiment of the present application, the first preset weight corresponds to the first part of the parts to which the two key points belong, wherein the first part can be any part on the designated object.

[0217] In one implementation, after the first correlation coefficient is obtained, the first correlation coefficient can be directly determined as the true correlation coefficient between the two key points.

[0218] In another implementation, the method further includes:

[0219] Step 3: According to a second preset formula, the second correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points.

[0220] The second preset formula is used to express that there is a physiological constraint relationship between the second part and the first part. The second preset formula is:

[0221] R2=1-d×Z2

[0222] R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight.

[0223] The above step 2 includes: calculating the average value of the first correlation coefficient and the second correlation coefficient to obtain the true correlation coefficient between the two key points.

[0224] Correspondingly, the second preset weight corresponds to the second part of the parts to which the two key points belong. That is, when calculating the true correlation coefficient between key points belonging to two different parts, the weights in the formula used can be different or the same.

[0225] Accordingly, for two key points, when the parts to which the two key points belong are different (i.e., the first part and the second part are different), the calculated first correlation coefficient and the second correlation coefficient may be different or the same. In addition, when the first correlation coefficient and the second correlation coefficient are different, the average value of the first correlation coefficient and the second correlation coefficient may be calculated to obtain the true correlation coefficient between the two key points. In this way, the accuracy of obtaining the true correlation coefficient between the two key points can be improved, and then, a key point positioning model with higher accuracy is obtained to improve the accuracy of the positioning result.

[0226] For two key points, when the two key points belong to the same part, the calculated first correlation coefficient and the second correlation coefficient must be the same. Therefore, after calculating the first correlation coefficient, the calculated first correlation coefficient can be directly determined as the true correlation coefficient between the two key points.

[0227] For example, in the above Figure 3 In the scenario shown, for face key point 1 and face key point 2, the part to which face key point 1 belongs is the face contour (i.e., the first part), and the part to which face key point 2 belongs is also the face contour (i.e., the second part). At this time, the first preset weight is consistent with the second preset weight, and accordingly, the calculated first correlation coefficient and the second correlation coefficient are also consistent. Therefore, after the first correlation coefficient is calculated, the first correlation coefficient can be directly determined as the true correlation coefficient between the two key points.

[0228] In this way, the amount of calculation required to obtain the true correlation coefficient between two key points can be reduced, and the efficiency of obtaining the true correlation coefficient can be improved.

[0229] For example, in the scenario of training a facial key point positioning model, and the specified object parts include: facial contour, left eyebrow, right eyebrow, nose, left eye, right eye and mouth, the true correlation coefficient between every two facial key points can be calculated by the following formula:

[0230]

[0231]

[0232]

[0233]

[0234]

[0235]

[0236] Among them, i represents the key point i of the face, and j represents the key point j of the face. ij Indicates the distance between the template coordinates of face key point i and face key point j. outline indicates the outer contour, eyebrow contains l_eyebrow and r_eyebrow, l_eyebrow indicates the left eyebrow, r_eyebrow indicates the right eyebrow, eye contains l_eye and r_eye, l_eye indicates the left eye, r_eyebrow indicates the right eye, nose indicates the nose, and mouth indicates the mouth.

[0237] When the part to which the facial key point i belongs is the facial contour, it can be based on R outline The corresponding formula determines the true correlation coefficient between face key point i and face key point j. At this time, the preset weight is 0.0025. If the part to which face key point j belongs is not the external contour or the mouth, or the distance between the template coordinates of face key point i and face key point j is not less than 120, then the true correlation coefficient between face key point i and face key point j is 0.

[0238] When the facial key point i belongs to the right eyebrow or the left eyebrow, it can be based on R eyebrow The corresponding formula determines the true correlation coefficient between face key point i and face key point j. At this time, the preset weight is 0.005. If the part to which face key point j belongs is not the right eyebrow, the left eyebrow, the left eye or the right eye, or the distance between the template coordinates of face key point i and face key point j is not less than 60, then the true correlation coefficient between face key point i and face key point j is 0.

[0239] When the facial key point i belongs to the left eye, it can be based on R l _ eye The corresponding formula determines the true correlation coefficient between face key point i and face key point j. At this time, the preset weight is 0.005. If the part to which face key point j belongs is not the left eyebrow, left eye or nose, or the distance between the template coordinates of face key point i and face key point j is not less than 60, then the true correlation coefficient between face key point i and face key point j is 0.

[0240] When the facial key point i belongs to the right eye, it can be based on R r _ eye The corresponding formula determines the true correlation coefficient between the face key point i and the face key point j. At this time, the preset weight is 0.005. If the part to which the face key point j belongs is not the right eyebrow, the right eye or the nose, or the distance between the template coordinates of the face key point i and the face key point j is not less than 60, then the true correlation coefficient between the face key point i and the face key point j is 0.

[0241] When the facial key point i belongs to the nose, it can be based on R nose The corresponding formula determines the true correlation coefficient between the face key point i and the face key point j. At this time, the preset weight is 0.0025. If the part to which the face key point j belongs is not the right eyebrow, the left eyebrow, the left eye, the right eye, the nose or the mouth, or the distance between the template coordinates of the face key point i and the face key point j is not less than 120, then the true correlation coefficient between the face key point i and the face key point j is 0.

[0242] When the facial key point i belongs to the mouth, it can be based on R mouth The corresponding formula determines the true correlation coefficient between face key point i and face key point j. At this time, the preset weight is 0.005. If the part to which face key point j belongs is not the face contour, nose or mouth, or the distance between the template coordinates of face key point i and face key point j is not less than 60, then the true correlation coefficient between face key point i and face key point j is 0.

[0243] It is understandable that, according to the first method, the true correlation coefficient between each two key points is unique, that is, for each two key points, the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points are not distinguished. At this time, a true correlation matrix can be obtained according to the first method recorded in the above implementation.

[0244] Subsequently, when the predicted correlation coefficient between two key points only includes the predicted correlation coefficient between the horizontal coordinates of the two key points, the first loss value can be calculated based on the difference between the true correlation matrix and the X-axis predicted correlation matrix.

[0245] In the case where the predicted correlation coefficient between two key points only includes the predicted correlation coefficient between the vertical coordinates of the two key points, the first loss value can be calculated based on the difference between the true correlation matrix and the Y-axis predicted correlation matrix.

[0246] When the predicted correlation coefficient between two key points only includes the predicted correlation coefficient between the horizontal coordinates of the two key points, the first loss value can be calculated based on the difference between the true correlation matrix and the X-axis predicted correlation matrix, and the difference between the true correlation matrix and the Y-axis predicted correlation matrix.

[0247] Method 2: Based on the pre-trained key point positioning model, the true correlation coefficient between every two key points can be obtained. Figure 6 , Figure 6 The third flow chart for obtaining the true correlation coefficient provided in the embodiment of the present application includes:

[0248] S601: Obtain the current weight matrix of the specified network layer in the second key point positioning model as the real weight matrix.

[0249] Among them, the second key point positioning model is: trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: the feature of the horizontal coordinate of a key point, or, the feature of the vertical coordinate of a key point.

[0250] S602: For every two key points, based on the columns corresponding to the two key points in the true weight matrix, calculate the correlation coefficient between the two key points as the true correlation coefficient.

[0251] In the embodiment of the present application, the second sample image is composed of a small number of sample images containing the designated object in the designated state and a small number of sample images containing the designated object in the non-designated state. Since the second sample image contains the designated object in the non-designated state and the designated object in the designated state, the real correlation between each two key points can be determined based on the real coordinates of the key points in the second sample image of the designated object in different states, thereby improving the accuracy of the predicted correlation coefficient between each two key points learned by the model.

[0252] Correspondingly, the second key point positioning model is trained based on the second sample image and the corresponding second sample label. The process of obtaining the current weight matrix of the specified network layer in the second key point positioning model as the real weight matrix is ​​similar to the process of obtaining the predicted weight matrix. The specific processing process can refer to the relevant description of the above step S102, which will not be repeated here.

[0253] Furthermore, after obtaining the true weight matrix, the true correlation coefficient between every two key points can be calculated based on the true weight matrix.

[0254] In step S602, for any key point, the column corresponding to the horizontal coordinate of the key point and the column corresponding to the vertical coordinate of the key point can be obtained in the true weight matrix.

[0255] Based on the above processing, the true correlation coefficient between every two key points can be obtained through model learning. Compared with the above method 1, the amount of calculation for obtaining the true correlation coefficient can be reduced, thereby improving the training efficiency of the key point positioning model.

[0256] In some embodiments, the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points, and / or the true correlation coefficient between the vertical coordinates of the two key points.

[0257] Correspondingly, the true correlation coefficient between the horizontal coordinates of any two key points is obtained by processing the columns corresponding to the horizontal coordinates of the two key points in the real weight matrix based on the preset correlation coefficient algorithm. The true correlation coefficient between the vertical coordinates of any two key points is obtained by processing the columns corresponding to the vertical coordinates of the two key points in the real weight matrix based on the preset correlation coefficient algorithm.

[0258] In one implementation, for every two key points, only the true correlation coefficient between the horizontal coordinates of the two key points may be obtained as the true correlation coefficient between the two key points.

[0259] The method of calculating the true correlation coefficient between the horizontal coordinates of the two key points is similar to the method of calculating the predicted correlation coefficient between the horizontal coordinates of the two key points, which will not be described in detail here.

[0260] At this time, a real correlation matrix (which may be called an X-axis real correlation matrix) may also be obtained based on the real correlation coefficient between the horizontal coordinates of each two key points. The process of obtaining the X-axis real correlation matrix is ​​similar to the process of obtaining the X-axis predicted correlation matrix, which will not be described in detail here.

[0261] In another implementation manner, for every two key points, only the true correlation coefficient between the longitudinal coordinates of the two key points may be obtained as the true correlation coefficient between the two key points.

[0262] The method of calculating the true correlation coefficient between the ordinates of the two key points is similar to the method of calculating the predicted correlation coefficient between the ordinates of the two key points, which will not be described in detail here.

[0263] At this time, a real correlation matrix (which may be called a Y-axis real correlation matrix) may also be obtained based on the real correlation coefficient between the ordinates of each two key points. The process of obtaining the Y-axis real correlation matrix is ​​similar to the process of obtaining the Y-axis predicted correlation matrix, which will not be described in detail here.

[0264] In another implementation, for every two key points, the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points may be obtained to obtain the true correlation coefficient between the two key points.

[0265] At this time, the X-axis real correlation matrix can be obtained based on the real correlation coefficient between the horizontal coordinates of every two key points, and the Y-axis real correlation matrix can be obtained based on the real correlation coefficient between the vertical coordinates of every two key points.

[0266] In some embodiments, step S104 includes:

[0267] In one implementation method, if the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, then the first loss value is calculated based on the first sub-loss value and the second sub-loss value.

[0268] Among them, the first sub-loss value represents: the difference between the predicted correlation coefficient between the horizontal coordinates and the true correlation coefficient between the horizontal coordinates; the second sub-loss value represents: the difference between the predicted correlation coefficient between the vertical coordinates and the true correlation coefficient between the vertical coordinates.

[0269] In an embodiment of the present application, if the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, then the predicted correlation coefficient between any two key points also includes: the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points.

[0270] Therefore, the first sub-loss value can be calculated based on the true correlation coefficient between each horizontal coordinate and the predicted correlation coefficient between each horizontal coordinate. For example, the first sub-loss value can be calculated based on the difference between the X-axis true correlation matrix and the X-axis predicted correlation matrix. And the second sub-loss value can be calculated based on the true correlation coefficient between each vertical coordinate and the predicted correlation coefficient between each vertical coordinate. For example, the second sub-loss value is calculated based on the difference between the Y-axis true correlation matrix and the Y-axis predicted correlation matrix.

[0271] Furthermore, the first loss value may be calculated based on the first sub-loss value and the second sub-loss value. For example, a weighted sum of the first sub-loss value and the second sub-loss value may be calculated as the first loss value.

[0272] Based on the above processing, when the true correlation coefficient between two key points includes the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, the true correlation coefficient can reflect the true correlation between the horizontal coordinates of each two key points and the true correlation between the vertical coordinates of each two key points. In this case, obtaining the first loss value can more accurately reflect the difference between the correlation between the two key points currently learned by the model and the true correlation between the two key points. In this way, a key point positioning model with higher accuracy can be further obtained to improve the accuracy of the positioning result.

[0273] Another implementation method is that if the true correlation coefficient between any two key points is the true correlation coefficient between the horizontal coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the horizontal coordinates of the two key points, then the first sub-loss value is obtained as the first loss value.

[0274] In an embodiment of the present application, if the true correlation coefficient between any two key points includes the true correlation coefficient between the horizontal coordinates of the two key points, then the predicted correlation coefficient between any two key points also includes the predicted correlation coefficient between the horizontal coordinates of the two key points.

[0275] Therefore, the first sub-loss value can be calculated based on the true correlation coefficient between each horizontal coordinate and the predicted correlation coefficient between each horizontal coordinate. For example, the loss value between the X-axis true correlation matrix and the X-axis predicted correlation matrix can be calculated as the first sub-loss value. Furthermore, the first sub-loss value can be directly used as the first loss value.

[0276] In another implementation method, if the true correlation coefficient between any two key points is the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the vertical coordinates of the two key points, then the second sub-loss value is obtained as the first loss value.

[0277] In an embodiment of the present application, if the true correlation coefficient between any two key points includes the true correlation coefficient between the vertical coordinates of the two key points, then the predicted correlation coefficient between any two key points also includes the predicted correlation coefficient between the vertical coordinates of the two key points.

[0278] Therefore, the second sub-loss value can be calculated based on the true correlation coefficient between each ordinate and the predicted correlation coefficient between each ordinate. For example, the loss value between the Y-axis true correlation matrix and the Y-axis predicted correlation matrix can be calculated as the second sub-loss value. Furthermore, the second sub-loss value can be directly used as the first loss value.

[0279] When the true correlation coefficient between two key points only includes the true correlation coefficient between the horizontal coordinates of the two key points, or only includes the true correlation coefficient between the vertical coordinates of the two key points, the amount of calculation required to obtain each true correlation coefficient is low, and correspondingly, the amount of calculation required to obtain the first loss value is also low. In this way, the training efficiency of the key point positioning model can be improved.

[0280] See also Figure 7 , Figure 7 A flowchart of a key point positioning model for training provided in an embodiment of the present application.

[0281] S701: Start.

[0282] S702: Obtain a normal human face key point training set.

[0283] That is, a first sample image is acquired, wherein the first sample image at least includes a designated object in a non-designated state.

[0284] S703: Train a key point regression model.

[0285] That is, the first sample image is input into the key point positioning model to be trained (ie, the first key point positioning model to be trained).

[0286] S704: Calculate the feature correlation weight of each point.

[0287] That is, the current weight matrix of the specified network layer in the first key point positioning model to be trained is obtained as the prediction weight matrix. For every two key points, based on the columns corresponding to the two key points in the prediction weight matrix, the correlation coefficient between the two key points is calculated as the prediction correlation coefficient.

[0288] S705: Obtain the correlation of the features of each constraint point.

[0289] That is, the real correlation coefficient is obtained. The real correlation coefficient can be obtained by learning the above model or by calculating the above formula.

[0290] S706: Decoupling learning of facial key point model features.

[0291] That is, based on the predicted correlation coefficient and the true correlation coefficient, the first loss value is calculated; based on the first loss value and the second loss value, the model parameters of the first key point positioning model to be trained are adjusted until the preset convergence condition is reached, and the trained first key point positioning model is obtained. The second loss value is determined based on the predicted coordinates and the true coordinates of the key point of the specified object in the first sample image.

[0292] S707: Predicting facial key point model.

[0293] That is, the image to be detected is input into the trained first key point positioning model.

[0294] S708: Output the coordinates of facial key points.

[0295] That is, the coordinates of the key points of the specified object in the image to be detected are obtained.

[0296] Based on the same inventive concept, the present application embodiment provides a key point positioning method, see Figure 8 , Figure 8 A first flow chart of a key point positioning method provided by an embodiment of the present invention, the method comprising:

[0297] S801: Acquire an image to be detected containing a specified object.

[0298] S802: Input the image to be detected into the trained third key point positioning model to obtain the coordinates of the key points of the specified object in the image to be detected.

[0299] Among them, the third key point positioning model is trained according to any of the above-mentioned key point positioning model training methods.

[0300] In the embodiment of the present application, the image to be detected is an image that currently needs to perform key point positioning.

[0301] The image to be detected is input into the trained third key point positioning model to obtain the coordinates of the key points of the specified object in the image to be detected. The third key point positioning model is trained according to any of the above key point positioning model training methods.

[0302] Based on the above processing, during the training process of the key point positioning model, the current weight matrix (i.e., the prediction weight matrix) of the output layer of the key point positioning model can be obtained. Since a column in the weight matrix can represent the characteristics of the horizontal coordinate of a key point, or the characteristics of the vertical coordinate of a key point, for any two key points of the specified object, the prediction correlation coefficient obtained based on the prediction weight matrix can reflect the correlation currently learned by the model between the two key points. And since the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image, the true correlation coefficient can reflect the true correlation between the two key points. Furthermore, the first loss value can reflect: the difference between the correlation currently learned by the model between the two key points and the true correlation between the two key points. Accordingly, adjusting the model parameters of the first key point positioning model to be trained based on the first loss value can enable the first key point positioning model to learn the true correlation between each two key points, so that a key point positioning model with higher accuracy can be obtained to improve the accuracy of the positioning result.

[0303] See also Fig. 9 , Fig. 9 A flowchart for obtaining facial key points in a sample image containing a face is provided in an embodiment of the present application.

[0304] S901: Obtain an input image.

[0305] S902: Input the input image to the face detector. If a face is detected, execute step S903; if no face is detected, return to execute step S901.

[0306] S903: Acquire a face region image.

[0307] That is, an image area corresponding to the minimum circumscribed rectangle corresponding to the face is determined from the input image as the face area image.

[0308] S904: Inputting the face region image into the face key point positioning model.

[0309] Among them, the facial key point positioning model is obtained by training according to the key point positioning model training method provided in this application.

[0310] S905: Obtain the coordinates of each facial key point in the facial region image.

[0311] In the technical solution of the present application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing user personal information (facial information, hand information, body information, etc.) are all carried out with the user's authorization.

[0312] It should be noted that the image containing the designated object in this embodiment is not an image of the designated object corresponding to a specific user, and cannot reflect the personal information of a specific user.

[0313] It should be noted that the images containing human faces, images containing human hands, and images containing human bodies in this embodiment are from public data sets.

[0314] Based on the same inventive concept, the present application embodiment provides a key point positioning model training device. Fig.10 , Fig.10 A structural diagram of a key point positioning model training device provided in an embodiment of the present application, the device comprising:

[0315] The prediction coordinate acquisition module 1001 is used to input a first sample image containing a specified object into a first key point positioning model to be trained, and obtain the coordinates of the key points representing the specified object in the first sample image as the prediction coordinates;

[0316] The prediction weight matrix acquisition module 1002 is used to obtain the current weight matrix of the specified network layer in the first key point positioning model to be trained as the prediction weight matrix; wherein any column in the prediction weight matrix represents: the feature of the horizontal coordinate of a key point, or the feature of the vertical coordinate of a key point;

[0317] A prediction correlation coefficient acquisition module 1003 is used to calculate, for each two key points, the correlation coefficient between the two key points based on the columns corresponding to the two key points in the prediction weight matrix as the prediction correlation coefficient;

[0318] A first loss value acquisition module 1004 is used to calculate a first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image;

[0319] The model parameter adjustment module 1005 is used to adjust the model parameters of the first key point positioning model to be trained based on the first loss value and the second loss value until the preset convergence condition is reached to obtain the trained first key point positioning model; wherein the second loss value is determined based on the predicted coordinates and the real coordinates of the key point of the specified object in the first sample image.

[0320] In some embodiments, the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points, and / or the predicted correlation coefficient between the vertical coordinates of the two key points.

[0321] In some embodiments, the predicted correlation coefficient between the horizontal coordinates of any two key points is: based on a preset correlation coefficient algorithm, the columns corresponding to the horizontal coordinates of the two key points in the prediction weight matrix are processed; the predicted correlation coefficient between the vertical coordinates of any two key points is: based on a preset correlation coefficient algorithm, the columns corresponding to the vertical coordinates of the two key points in the prediction weight matrix are processed.

[0322] In some embodiments, the true correlation coefficient is determined based on the true coordinates of the key points of the specified object in the second sample image and in combination with the physiological constraint relationship of the part to which the key points in the specified object belong.

[0323] In some embodiments, the device also includes: a second sample image acquisition module, used to acquire the second sample image before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient; a template coordinate acquisition module, used to calculate, for each key point, the mean of the horizontal coordinate and the mean of the vertical coordinate in the true coordinates of the key point in each second sample image as the template coordinates of the key point; a first true correlation coefficient acquisition module, used to calculate, for each two key points, the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object.

[0324] In some embodiments, the first true correlation coefficient acquisition module includes: a first condition determination submodule, for determining, for each two key points, that the true correlation coefficient between the two key points is 0 if there is no physiological constraint relationship between the parts to which the two key points belong in the specified object; a second condition determination submodule, for determining that the true correlation coefficient between the two key points is 0 if there is a physiological constraint relationship between the parts to which the two key points belong in the specified object and the distance between the template coordinates of the two key points is not less than a preset distance; a third condition determination submodule, for calculating the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points if there is a physiological constraint relationship between the parts to which the two key points belong in the specified object and the distance between the template coordinates of the two key points is less than a preset distance; wherein the calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

[0325] In some embodiments, the third condition determination submodule includes: a first correlation coefficient calculation unit, which is used to calculate the first correlation coefficient between the two key points according to a first preset formula using the distance between the template coordinates of the two key points; wherein the first preset formula indicates that there is a physiological constraint relationship between the first part of the part to which the two key points belong and the second part of the part to which they belong; the first preset formula is: R1=1-d×Z1, R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight; a true correlation coefficient calculation unit, which is used to calculate the true correlation coefficient between the two key points based on the first correlation coefficient.

[0326] In some embodiments, the device also includes: a second correlation coefficient calculation module, which is used to calculate the second correlation coefficient between the two key points according to a second preset formula using the distance between the template coordinates of the two key points before calculating the true correlation coefficient between the two key points based on the first correlation coefficient; wherein the second preset formula indicates that there is a physiological constraint relationship between the second part and the first part; the second preset formula is: R2=1-d×Z2, R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight; the true correlation coefficient calculation unit is specifically used to calculate the average of the first correlation coefficient and the second correlation coefficient to obtain the true correlation coefficient between the two key points.

[0327] In some embodiments, the second sample image acquisition module includes: a third sample image acquisition submodule, used to acquire a third sample image; a mapping submodule, used to map each third sample image to a unified image template to obtain a second sample image; wherein the distance between the preset parts of the specified object in each second sample image is less than a preset threshold.

[0328] In some embodiments, the device also includes: a true weight matrix acquisition module, which is used to obtain the current weight matrix of the specified network layer in the second key point positioning model as the true weight matrix before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the second key point positioning model is: trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: the feature of the horizontal coordinate of a key point, or, the feature of the vertical coordinate of a key point; the second true correlation coefficient acquisition module is used to calculate the correlation coefficient between the two key points for every two key points based on the columns corresponding to the two key points in the true weight matrix, as the true correlation coefficient.

[0329] In some embodiments, the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points, and / or the true correlation coefficient between the vertical coordinates of the two key points.

[0330] In some embodiments, the true correlation coefficient between the horizontal coordinates of any two key points is: based on a preset correlation coefficient algorithm, the columns corresponding to the horizontal coordinates of the two key points in the real weight matrix are processed; the true correlation coefficient between the vertical coordinates of any two key points is: based on a preset correlation coefficient algorithm, the columns corresponding to the vertical coordinates of the two key points in the real weight matrix are processed.

[0331] In some embodiments, the first loss value acquisition module 1004 is specifically used to: if the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, then calculate the first loss value based on the first sub-loss value and the second sub-loss value; wherein the first sub-loss value represents: the difference between the predicted correlation coefficient between the horizontal coordinates and the true correlation coefficient between the horizontal coordinates; the second sub-loss value represents: the difference between the predicted correlation coefficient between the horizontal coordinates and the true correlation coefficient between the horizontal coordinates; The loss value represents: the difference between the predicted correlation coefficient between the vertical coordinates and the true correlation coefficient between the vertical coordinates; if the true correlation coefficient between any two key points is the true correlation coefficient between the horizontal coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the horizontal coordinates of the two key points, then the first sub-loss value is obtained as the first loss value; if the true correlation coefficient between any two key points is the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the vertical coordinates of the two key points, then the second sub-loss value is obtained as the first loss value.

[0332] In some embodiments, the key points of the designated object include at least one of the following: face key points, finger key points, and human body posture key points.

[0333] Based on the same inventive concept, the present application embodiment provides a key point positioning device. Fig.11 , Fig.11 A structural diagram of a key point positioning device provided in an embodiment of the present application, the device comprising:

[0334] The to-be-detected image acquisition module 1101 is used to acquire the to-be-detected image containing the specified object;

[0335] The positioning module 1102 is used to input the image to be detected into a trained third key point positioning model to obtain the coordinates of the key points of the specified object in the image to be detected; wherein the third key point positioning model is trained according to any key point positioning model training method in the above embodiments.

[0336] The present application also provides an electronic device, such as Fig.12 As shown, including:

[0337] Memory 1201, used for storing computer programs;

[0338] The processor 1202 is used to implement any key point positioning model training method or key point positioning method steps in the above embodiments when executing the program stored in the memory 1201.

[0339] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1202, the communication interface, and the memory 1201 communicate with each other via the communication bus.

[0340] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0341] The communication interface is used for communication between the above electronic device and other devices.

[0342] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0343] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0344] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned key point positioning model training methods, or the steps of the key point positioning method.

[0345] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any key point positioning model training method or key point positioning method in the above-mentioned embodiments.

[0346] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0347] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0348] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0349] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A key point positioning model training method, characterized in that: The method comprises: Inputting a first sample image containing a specified object into a first key point positioning model to be trained, and obtaining coordinates of key points representing the specified object in the first sample image as predicted coordinates; Obtaining the current weight matrix of the specified network layer in the first key point positioning model to be trained as a prediction weight matrix; wherein any column in the prediction weight matrix represents: a feature of the horizontal coordinate of a key point, or a feature of the vertical coordinate of a key point; For every two key points, based on the columns corresponding to the two key points in the prediction weight matrix, calculate the correlation coefficient between the two key points as the prediction correlation coefficient; Calculating a first loss value based on the predicted correlation coefficient and the true correlation coefficient; wherein the true correlation coefficient is determined based on the true coordinates of the key point of the specified object in the second sample image; Based on the first loss value and the second loss value, the model parameters of the first key point positioning model to be trained are adjusted until a preset convergence condition is reached, thereby obtaining a trained first key point positioning model; wherein the second loss value is determined based on the predicted coordinates and the real coordinates of the key point of the specified object in the first sample image.

2. The method according to claim 1, characterized in that The prediction correlation coefficient between any two key points includes: the prediction correlation coefficient between the horizontal coordinates of the two key points, and / or the prediction correlation coefficient between the vertical coordinates of the two key points.

3. The method according to claim 2, characterized in that The prediction correlation coefficient between the horizontal coordinates of any two key points is obtained by processing the columns corresponding to the horizontal coordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm; The prediction correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the prediction weight matrix based on a preset correlation coefficient algorithm.

4. The method according to claim 1, characterized in that: The true correlation coefficient is determined based on the true coordinates of the key points of the designated object in the second sample image and in combination with the physiological constraint relationship of the part to which the key points in the designated object belong.

5. The method according to claim 4, characterized in that Before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient, the method further includes: acquiring the second sample image; For each key point, calculate the mean of the horizontal coordinate and the mean of the vertical coordinate of the real coordinates of the key point in each second sample image as the template coordinates of the key point; For every two key points, the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object.

6. The method according to claim 5, characterized in that The calculating, for each two key points, of the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points and the physiological constraint relationship between the parts to which the two key points belong in the specified object comprises: For each two key points, if there is no physiological constraint relationship between the parts to which the two key points belong in the specified object, then the true correlation coefficient between the two key points is determined to be 0; If the parts to which the two key points belong in the specified object have a physiological constraint relationship, and the distance between the template coordinates of the two key points is not less than a preset distance, then determining that the true correlation coefficient between the two key points is 0; If the parts to which the two key points belong in the specified object have a physiological constraint relationship, and the distance between the template coordinates of the two key points is less than the preset distance, then the true correlation coefficient between the two key points is calculated based on the distance between the template coordinates of the two key points; wherein the calculated true correlation coefficient between the two key points is negatively correlated with the distance between the template coordinates of the two key points.

7. The method according to claim 6, characterized in that The calculating the true correlation coefficient between the two key points based on the distance between the template coordinates of the two key points includes: According to a first preset formula, a first correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points; wherein the first preset formula indicates that a first part of the parts to which the two key points belong and a second part of the parts to which the two key points belong have a physiological constraint relationship; The first preset formula is: R1=1-d×Z1 R1 represents the first correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z1 represents the first preset weight; Based on the first correlation coefficient, a true correlation coefficient between the two key points is calculated.

8. The method according to claim 7, characterized in that Before calculating the true correlation coefficient between the two key points based on the first correlation coefficient, the method further includes: According to a second preset formula, a second correlation coefficient between the two key points is calculated using the distance between the template coordinates of the two key points; wherein the second preset formula indicates that there is a physiological constraint relationship between the second part and the first part; The second preset formula is: R2=1-d×Z2 R2 represents the second correlation coefficient, d represents the distance between the template coordinates of the two key points, and Z2 represents the second preset weight; The calculating the true correlation coefficient between the two key points based on the first correlation coefficient includes: An average value of the first correlation coefficient and the second correlation coefficient is calculated to obtain a true correlation coefficient between the two key points.

9. The method according to claim 5, characterized in that The acquiring the second sample image comprises: acquiring a third sample image; Each third sample image is mapped to a unified image template to obtain a second sample image; wherein the distance between the preset parts of the specified object in each second sample image is less than a preset threshold.

10. The method according to claim 1, characterized in that Before calculating the first loss value based on the predicted correlation coefficient and the true correlation coefficient, the method further includes: Obtaining the current weight matrix of the specified network layer in the second key point positioning model as the true weight matrix; wherein the second key point positioning model is: trained based on the second sample image and the corresponding second sample label; the second sample image contains a specified object in a specified state; any column in the true weight matrix represents: a feature of the horizontal coordinate of a key point, or a feature of the vertical coordinate of a key point; For every two key points, based on the columns corresponding to the two key points in the true weight matrix, a correlation coefficient between the two key points is calculated as the true correlation coefficient.

11. The method according to claim 10, characterized in that The true correlation coefficient between any two key points includes: the true correlation coefficient between the abscissas of the two key points, and / or the true correlation coefficient between the ordinates of the two key points.

12. The method according to claim 11, characterized in that The true correlation coefficient between the horizontal coordinates of any two key points is obtained by processing the columns corresponding to the horizontal coordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm; The true correlation coefficient between the ordinates of any two key points is obtained by processing the columns corresponding to the ordinates of the two key points in the true weight matrix based on a preset correlation coefficient algorithm.

13. The method according to claim 11, characterized in that The calculating a first loss value based on the predicted correlation coefficient and the true correlation coefficient comprises: If the true correlation coefficient between any two key points includes: the true correlation coefficient between the horizontal coordinates of the two key points and the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points includes: the predicted correlation coefficient between the horizontal coordinates of the two key points and the predicted correlation coefficient between the vertical coordinates of the two key points, then based on the first sub-loss value and the second sub-loss value, calculate the first loss value; wherein the first sub-loss value represents: the difference between the predicted correlation coefficient between the horizontal coordinates and the true correlation coefficient between the horizontal coordinates; the second sub-loss value represents: the difference between the predicted correlation coefficient between the vertical coordinates and the true correlation coefficient between the vertical coordinates; If the true correlation coefficient between any two key points is the true correlation coefficient between the horizontal coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the horizontal coordinates of the two key points, then obtaining the first sub-loss value as the first loss value; If the true correlation coefficient between any two key points is the true correlation coefficient between the vertical coordinates of the two key points, and the predicted correlation coefficient between any two key points is the predicted correlation coefficient between the vertical coordinates of the two key points, then the second sub-loss value is obtained as the first loss value.

14. The method according to any one of claims 1 to 13, characterized in that: The key points of the designated object include at least one of the following: face key points, finger key points, and body posture key points.

15. A key point positioning method, characterized in that: The method comprises: Get the image to be detected containing the specified object; The image to be detected is input into a trained third key point positioning model to obtain the coordinates of the key points of the specified object in the image to be detected; wherein the third key point positioning model is trained according to any one of the methods described in claims 1-14.

16. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-14 or 15 when executing a program stored in a memory.

Images