A key point detection method, system and related device

Abstract

The present application discloses a key point detection method, system and related device, the method comprising: in response to obtaining an image to be detected, obtaining significant key points from the facial area in the image to be detected, and determining multiple reference key points in the image to be detected based on the significant key points; wherein the positions of the significant key points of multiple specifications are predefined; based on the reference key points, the image to be detected is divided into multiple areas to be detected; the coordinates of multiple key points in each area to be detected are predicted in turn to obtain multiple target key points in each area to be detected. In the above manner, the present application can improve the efficiency and accuracy of key point detection.

Description

Technical Field

[0001] The present application relates to the technical field of key point detection, and in particular to a key point detection method, system and related devices. Background Art

[0002] With the continuous improvement of intelligence level, the realization of more and more functions depends on the recognition of facial key points, such as the facial optimization function, expression tracking or special effects function of smart cameras. Among them, the accuracy of the detected facial key point position will directly affect the realization effect of the corresponding function.

[0003] At present, the traditional facial key point detection function is often implemented through smart terminals or embedded devices with lower power consumption. When faced with scenes such as occlusion or facial tilt, traditional facial key point detection methods usually have problems such as excessive consumption of computing resources and poor detection accuracy. In view of this, how to accurately detect facial key points by consuming less computing resources has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a key point detection method, system and related devices, which can improve the efficiency and accuracy of key point detection.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a key point detection method, comprising: in response to obtaining an image to be detected, obtaining significant key points from the facial area in the image to be detected, and determining multiple reference key points in the image to be detected based on the significant key points; wherein the positions of the multiple significant key points are predefined; based on the reference key points, the image to be detected is divided into multiple areas to be detected; the coordinates of multiple key points in each of the areas to be detected are predicted in turn to obtain multiple target key points in each of the areas to be detected.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide a key point detection system, including: a processing module, in response to obtaining an image to be detected, the processing module is used to obtain significant key points from the facial area in the image to be detected, and determine multiple reference key points in the image to be detected based on the significant key points; wherein the positions of the multiple significant key points are predefined; a division module, used to divide the image to be detected into multiple areas to be detected based on the reference key points; a detection module, used to predict the coordinates of multiple key points in each of the areas to be detected in turn, and obtain multiple target key points in each of the areas to be detected.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising: a memory and a processor coupled to each other, the memory storing program data, and the processor calling the program data to execute the key point detection method mentioned in the above technical solution.

[0008] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium on which program data is stored, and when the program data is executed by a processor, the key point detection method mentioned in the above technical solution is implemented.

[0009] The beneficial effect of the present application is as follows: Different from the prior art, the key point detection method proposed in the present application first determines the significant key points of the facial area in the image to be detected, and then efficiently determines the reference key points in the image to be detected according to the positions of the significant key points in the image to be detected. Thus, the image to be detected can be divided into multiple areas to be detected according to the distribution of the reference key points in the image, so that there is no need to predict the key point coordinates of the complete image to be detected later, that is, the key point coordinates are predicted for the area to be detected containing a single feature information each time, which saves the computing resources required for the prediction and improves the efficiency of obtaining the target key points; at the same time, it also avoids the influence of the difference in feature information between different areas to be detected on the prediction results, and improves the accuracy of the obtained target key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0011] Figure 1 It is a flow chart of an implementation method of a key point detection method of the present application;

[0012] Figure 2 is a flowchart of an implementation method corresponding to step S101;

[0013] Figure 3 is a flowchart of step S103 corresponding to another implementation manner;

[0014] Figure 4 is a schematic diagram of an implementation method corresponding to the detection model of the present application;

[0015] Figure 5 It is a flowchart of an implementation method of a training method of a detection model of the present application;

[0016] Figure 6It is a flowchart of another implementation method of the training method of the detection model of the present application;

[0017] Figure 7 is a schematic diagram of another implementation of the detection model of the present application;

[0018] Figure 8 It is a structural schematic diagram of an implementation method of a key point detection system of the present application;

[0019] Fig. 9 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0020] Fig.10 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] See also Figure 1 , Figure 1 : is a flow chart of an implementation method of a key point detection method of the present application, the method comprising:

[0023] S101: In response to obtaining an image to be detected, obtaining salient key points from a facial region in the image to be detected, and determining a plurality of reference key points in the image to be detected based on the salient key points, wherein positions of the plurality of salient key points are predefined.

[0024] In one embodiment, see Figure 2 , Figure 2 is a flow chart of an implementation method corresponding to step S101. Specifically, step S101 includes:

[0025] S1011: Determine a facial region in the image to be detected, and based on the positions of all pre-defined salient key points, obtain all visible salient key points from the facial region.

[0026] In one embodiment, the implementation process of step S1011 includes: after obtaining the image to be detected that needs to be detected for facial key points, identifying the facial area in the image to be detected, and determining all pre-defined significant key points in the facial area. Among them, the significant key points are often located in the area containing the significant information of the face, and the recognition difficulty is relatively low; for example, the nose tip, the corners of the eyes and the corners of the mouth of the face. In addition, since the facial features of different people have certain similarities, the position of the facial significant key points is less affected by the differences in the faces of the people.

[0027] In one implementation scenario, face detection is performed on the obtained image to be detected. Since the face detection algorithm pays more attention to the significant features of the face in the image during detection, some significant key points can be predicted during the face detection process. Among them, the face detection method can be directly implemented using an existing open source algorithm, and this application does not limit this.

[0028] In another implementation scenario, after obtaining multiple salient key points of the face, it is determined whether the facial area at each salient key point is occluded, and based on the determination result, the salient key points corresponding to the unoccluded facial area are retained and the remaining key points are deleted to improve the accuracy of the obtained salient key points.

[0029] In a specific application scenario, after performing face detection on the image to be detected, facial feature information in the image to be detected is obtained, and the left eye area, right eye area, nose area, lip area, etc. are determined based on the facial feature information, and the center points or edge points of these areas are used as significant key points.

[0030] It should be noted that the specific number of significant key points in this application can be set according to actual conditions. Of course, in order to improve the accuracy of the subsequent process implementation, the number of significant key points should be no less than 5.

[0031] S1012: Adjust the universal face model based on the acquired salient key points to obtain a reference face model, wherein the universal face model is obtained based on multiple sample face models.

[0032] In one embodiment, the implementation process of step S1012 includes: obtaining multiple sample face models, and constructing a universal face model based on the facial shape information and facial texture information in the multiple sample face models. Wherein, each sample face model is pre-set with multiple facial key points, so the universal face model obtained based on the multiple sample face models also includes multiple facial key points.

[0033] Furthermore, the general face model is adjusted according to a plurality of significant key points in the image to be detected.

[0034] In one implementation scenario, multiple reference samples with high similarity to the image to be detected are determined based on feature information of the area around the significant key points in the image to be detected. Sample features of the reference samples are obtained, and a reference face model corresponding to the facial area in the image to be detected is obtained based on multiple sample features and a universal face model. The reference sample can be a partial facial area screened from multiple sample face models, or a partial area obtained from other face models different from the sample face models, for example, the tip of the nose, the corners of the eyes, the corners of the mouth, and other areas of the face.

[0035] In another embodiment, after obtaining the reference face model, the angle information of the facial region in the image to be detected may be determined according to a plurality of significant key points in the image to be detected, so as to adjust the angle of the reference face model according to the angle information.

[0036] In a specific application scenario, the reference face model can be obtained by the following formula:

[0037]

[0038] Among them, S new represents the reference face model corresponding to the facial area in the image to be detected, represents a universal face model, P represents the sample features of the reference sample, and H represents the weighting coefficient of the sample features.

[0039] S1013: Mapping facial key points in the reference face model to the image to be detected to obtain a plurality of reference key points in the image to be detected.

[0040] In one implementation, the specific implementation process of step S1013 includes the following steps:

[0041] A: Obtain all pre-set facial key points in the reference face model, and determine the control key points that match the salient key points from all facial key points.

[0042] In one embodiment, in response to the general face model including a plurality of facial key points, the reference face model obtained based on the general face model also includes corresponding facial key points.

[0043] Further, reference keypoints matching the salient keypoints are determined from among all facial keypoints.

[0044] In one implementation scenario, each facial key point in the reference face model includes corresponding position information, and a comparison key point matching the salient key point is determined from all facial key points based on the position information. For example, for a salient key point at the left eye corner of the image to be detected, the facial key point at the left eye corner of the reference face model is determined based on the key point information as the comparison key point.

[0045] In another implementation scenario, first feature information of the area around the salient key points in the image to be detected is obtained, and second feature information of the area around each facial key point in the reference face model is obtained. According to the similarity between the first feature information and the second feature information, a comparison key point matching the salient key point is determined. That is, for the first feature information corresponding to any salient key point, the facial key point corresponding to the second feature information having the highest similarity to the first feature information is used as the comparison key point.

[0046] B: Based on the salient key points and their matching control key points, determine the mapping relationship between the reference face model and the facial area in the image to be detected.

[0047] In one embodiment, a mapping relationship between a reference face model and a facial region in an image to be detected is determined based on multiple groups of salient key points and their matched control key points; so that through the mapping relationship, any position in the reference face model can correspond to a position in the facial region in the image to be detected.

[0048] In a specific application scenario, the above mapping relationship is the deformation coefficient of the reference face model and the facial area in the image to be detected. By multiplying the coordinates of a pixel point at any position in the reference face model by the deformation coefficient, the pixel point can be mapped to the image to be detected.

[0049] C: Based on the mapping relationship, the facial key points in the reference face model are mapped to the image to be detected.

[0050] In one embodiment, after determining the mapping relationship between the facial region in the image to be detected and the reference face model, all facial key points in the reference face model are mapped to the image to be detected to obtain multiple reference key points of the facial region in the image to be detected. This method does not require precise analysis of the facial region features in the image to be detected, and can preliminarily determine the position information of each key point, consumes less computing power resources, and facilitates the subsequent division of the area to be detected.

[0051] S102: Based on the reference key points, the image to be detected is divided into a plurality of areas to be detected.

[0052] In one embodiment, the implementation process of step S102 includes: in response to obtaining a plurality of reference key points in the image to be detected, dividing the image to be detected into a plurality of areas to be detected based on the distribution of the key points.

[0053] In an implementation scenario, according to the distribution of all reference key points in the image to be detected, all reference key points are divided into multiple key point sets, each of which corresponds to a facial area, such as a left eye area, a right eye area, a nose area, and a lip area. For each key point set, a corresponding area to be detected is determined from the image to be detected, and the area to be detected includes all reference key points in the corresponding key point set. By dividing the image to be detected into multiple areas to be detected according to the key point set, and each area to be detected corresponds to a facial area, the coordinates of the target key points in each facial area can be predicted later, thereby improving the accuracy of the position of the target key points.

[0054] In a specific application scenario, for each key point set, the enclosed area of ​​all reference key points therein is determined. Since the accuracy of the obtained reference key points is low, the precise target key points may be located around the reference key points. Therefore, the enclosed area of ​​the reference key points is appropriately expanded to obtain the area to be detected. The specific area of ​​the enclosed area can be set according to the actual situation.

[0055] In addition, it should be noted that the specific shape of the area to be detected can be related to the shape of the above-mentioned enclosed area; or, the shape of the area to be detected can also be circular, rectangular or irregular.

[0056] In another implementation scenario, in response to the region identifier corresponding to each reference key point in the image to be detected, the image to be detected is divided into multiple regions to be detected according to the region identifier, wherein the region identifiers of the reference key points contained in each region to be detected are consistent.

[0057] In a specific application scenario, for the reference key points of the left eye region, the reference key points are assigned a left eye region identifier. Thus, after all the reference key points in the image to be detected are determined, the region to be detected is determined based on the reference key points including the left eye region identifier, and the region to be detected is the left eye region of the face in the image to be detected.

[0058] S103: predicting the coordinates of multiple key points in each area to be detected in sequence to obtain multiple target key points in each area to be detected.

[0059] In one embodiment, the specific implementation process of step S103 includes: for each area to be detected, inputting it into a trained detection model, so that the detection model uses an attention mechanism to predict the key point coordinates of each area to be detected to obtain multiple target key points.

[0060] The key point detection method proposed in the present application first determines the significant key points of the facial area in the image to be detected, and then efficiently determines the reference key points in the image to be detected based on the positions of the significant key points in the image to be detected. Therefore, the image to be detected can be divided into multiple areas to be detected based on the distribution of the reference key points in the image, so that there is no need to predict the key point coordinates of the complete image to be detected later, that is, the key point coordinates of the area to be detected containing a single feature information are predicted each time, which saves the computing resources required for the prediction and improves the efficiency of obtaining the target key points; at the same time, it also avoids the influence of the difference in feature information between different areas to be detected on the prediction results, and improves the accuracy of the obtained target key points.

[0061] In one embodiment, see Figure 3 and Figure 4 , Figure 3 is a flowchart of another implementation method corresponding to step S103, Figure 4 1 is a schematic diagram of an implementation method of the detection model of the present application. Specifically, multiple target key points are obtained by predicting the key point coordinates by the detection model 100, and the detection model 100 is composed of a feature extraction network 10 and a detection network 20, and is obtained after training based on multiple training images, each training image corresponds to a part of the facial area and contains multiple annotated key points, the annotated key points of some training images are obtained based on manual annotation, and the annotated key points of the remaining training images are obtained based on the key points corresponding to the significant key points in the training images. The specific implementation process includes:

[0062] S201: Input the area to be detected into the feature extraction network to obtain the regional image features output by the feature extraction network.

[0063] In one embodiment, the specific implementation process of step S201 includes: inputting the area to be detected into the feature extraction network 10 in the detection model, so that the feature extraction network 10 extracts features related to the facial information of the human face in the area to be detected, thereby obtaining regional image features under a preset dimension output by the feature extraction network 10. By obtaining the regional image features, a basis is provided for subsequent analysis.

[0064] S202: Input the regional image features into the detection network to obtain the target key points in the to-be-detected region outputted in sequence by the detection network using the attention mechanism.

[0065] In one embodiment, the implementation process of step S202 includes: inputting the content to be detected into the detection network 20 to obtain the detection result output by the detection network. The content to be detected at least includes the regional image features of the area to be detected, and the detection result includes all the output target key points in the area to be detected.

[0066] In one implementation scenario, in response to obtaining the regional image features output by the feature extraction network, the regional image features are used as content to be detected, and the content to be detected is input into the self-attention sub-network 21 in the detection network 20 to obtain the target key points analyzed and output by the self-attention sub-network 21 for the content to be detected, and the currently output target key points are used as the first detection result output by the detection network.

[0067] Further, the regional image features and the latest detection results are updated as the content to be detected, and the process returns to the step of inputting the content to be detected into the detection network to obtain the detection results output by the detection network, until the end condition is met, and all target key points of the area to be detected are determined based on the latest detection results. The end condition is related to the number or position of the target key points.

[0068] Specifically, the detection network makes predictions based on the regional image features and the detection results output in the previous round, that is, the position coordinates of the remaining target key points are predicted based on the position coordinates of the target key points that have been determined, so that the correlation between the target key points finally determined is stronger and the position accuracy is higher.

[0069] In a specific application scenario, judging whether the detection result output by the detection network satisfies the termination condition includes: detecting whether the number of times the detection result is obtained reaches the number threshold. In response to the number of times the detection result is obtained reaching the number threshold, it is determined that the termination condition is satisfied, and all target key points of the area to be detected are determined based on the latest detection result; in response to the number of times the detection result is obtained not reaching the number threshold, it is determined that the termination condition is not satisfied, and the regional image features and the latest detection result are updated as the content to be detected, and the detection result output by the detection network based on the content to be detected is obtained. Optionally, it is also possible to judge whether the number of target key points that have been obtained has reached the number threshold; if so, it is determined that the termination condition is satisfied.

[0070] In another specific application scenario, determining whether the detection result output by the detection network satisfies the termination condition may also include: determining whether the target key point in the latest detection result overlaps with the target key point in the historical detection result. If so, it is determined that the termination condition is satisfied; otherwise, the termination condition is not satisfied.

[0071] In another specific application scenario, determining whether the detection result output by the detection network satisfies the termination condition may also include: determining the target key points in the area to be detected based on all the detection results that have been obtained, and determining the density of the current target key points. Determine whether the density reaches the density threshold. If so, it is determined that the termination condition is met; otherwise, it is determined that the termination condition is not met. Among them, the density of the above-mentioned target key points is determined by the number of target key points in the same area to be detected, and the more the number of target key points in the same area to be detected, the higher the density.

[0072] In one embodiment, in response to the key point detection method proposed in this application, it is necessary to pre-train a detection model before predicting the target key points of the image to be detected. The training process of the detection model is described in detail below. Figure 4 See also Figure 5 , Figure 5 1 is a flow chart of an implementation method of a training method for a detection model of the present application. The training method comprises:

[0073] S301: Input the training image into the feature extraction network to obtain the training image features.

[0074] In one embodiment, the implementation process of step S301 includes: constructing a detection model 100 for target key point detection, and the specific model structure is as follows: Figure 4 shown.

[0075] Further, a plurality of training images are obtained, and the training images are input into the feature extraction network 10 in the detection model 100 to obtain training image features. Each training image corresponds to a partial facial area, for example, an image of the left eye area, an image of the nose area, or an image of the lip area, etc. In addition, the training image contains a plurality of annotated key points.

[0076] In one implementation scenario, the labeled key points in the training images are obtained based on manual annotation.

[0077] In another implementation scenario, in order to save the cost of manual annotation, the annotation key points of some training images are obtained based on manual annotation, and the annotation key points of the remaining training images are obtained based on the reference key points corresponding to the salient key points in the training images, that is, the reference key points are used as the annotation key points. The specific process of obtaining the reference key points based on the salient key points in the training images can refer to the above corresponding implementation method, which will not be elaborated in detail here.

[0078] S302: Input the training image features into the detection network to obtain multiple predicted key points in the training image that are sequentially output by the detection network using an attention mechanism.

[0079] In one embodiment, the implementation process of step S302 includes: inputting the training image features output by the feature extraction network 10 into the detection network 20 to obtain a plurality of predicted key points outputted sequentially by the detection network 20 using the attention mechanism. The specific process of the detection network 20 sequentially outputting a plurality of predicted key points can refer to the above step S202, which will not be elaborated in detail here.

[0080] S303: Based on the labeled key points and the predicted key points, determine the training loss, and use the training loss to adjust the parameters of the feature extraction network and the detection network until the convergence condition is met to obtain the trained detection model.

[0081] In one embodiment, the specific implementation process of step S303 includes: after obtaining the predicted key points, based on the coordinates of all annotated key points and all predicted key points in the training image, obtaining the training loss, and using the training loss to adjust the parameters of the feature extraction network and the detection network until the convergence adjustment is met to obtain the trained detection model.

[0082] In one implementation scenario, in response to obtaining multiple predicted key points output by the detection network, for each predicted key point, the position deviation between each predicted key point and the corresponding labeled key point is calculated, and the sum of the position deviations corresponding to all predicted key points is used as the training loss. The training loss is used to adjust the parameters in the feature extraction network and the detection network until the convergence adjustment is met to obtain the trained detection model.

[0083] In another implementation scenario, after obtaining the predicted key points corresponding to each training image, the detection model is first trained using the annotated key points obtained based on the reference key points corresponding to the salient key points and the corresponding predicted key points, so that the model has the ability to recognize key points. Then, the detection model is trained using the annotated key points obtained based on manual annotation and the corresponding predicted key points to further improve the detection accuracy of the detection model.

[0084] In a specific application scenario, for training images where the labeled key points are obtained based on the reference key points corresponding to the salient key points, the position deviation between each predicted key point and the corresponding labeled key point is calculated, and the sum of the position deviations corresponding to all predicted key points is used as the training loss, and the parameters in the detection model are preliminarily adjusted using the training loss. Then, for training images where the labeled key points are obtained based on manual annotation, the position deviation between each predicted key point and the corresponding labeled key point is calculated, and the sum of the position deviations corresponding to all predicted key points is used as the training loss, and the parameters in the feature extraction network and the detection network are adjusted using the training loss until convergence adjustment is met to obtain a trained detection model.

[0085] In another embodiment, see Figure 6 and Figure 7 , Figure 6 is a flow chart of another implementation method of the training method of the detection model of the present application, Figure 7 is a schematic diagram of another embodiment of the detection model of the present application. In response to the detection model further comprising an edge prediction network 30, the training method of the detection model further comprises:

[0086] S401: Input the training image into the feature extraction network to obtain the training image features.

[0087] In one embodiment, the specific implementation process of step S401 includes: inputting the training image into the feature extraction network in the detection model to obtain the training image features. The training image corresponds to a partial facial area and contains multiple annotated key points. The specific acquisition process can refer to the above step S301.

[0088] S402: Input the training image features into the detection network and the edge prediction network to obtain a plurality of predicted key points in the training image outputted in sequence by the detection network using the attention mechanism, and a predicted edge outputted by the edge prediction network using the thermal information corresponding to the training image features.

[0089] In one embodiment, the specific implementation process of step S402 includes: inputting the obtained training image features into the detection network 20 and the edge prediction network 30 respectively. So that the detection network 20 uses the attention mechanism to analyze the training image features and outputs multiple predicted key points in sequence. And, the edge prediction network 30 obtains the thermal information corresponding to the training image according to the training image features, and outputs the predicted edges corresponding to the key points in the training image according to the thermal information. Among them, the thermal information is used to represent the number and position of the key points in the corresponding training image, and the predicted edge is the edge of the shape surrounded by the key points contained in the training image.

[0090] In another embodiment, the specific implementation process of step S402 may further include: directly determining the predicted edge according to the predicted multiple key points. Specifically, the training image features are input into the detection network 20, so that the detection network 20 uses the attention mechanism to sequentially output the multiple predicted key points, performs shape fitting on the obtained multiple predicted key points, and uses the edge of the fitted shape as the above-mentioned predicted edge.

[0091] S403: Based on the labeled key points, predicted key points and predicted edges, the training loss is determined, and the parameters of the feature extraction network and the detection network are adjusted using the training loss until the convergence condition is met, thereby obtaining a trained detection model.

[0092] In one embodiment, the specific implementation process of step S403 includes: obtaining position loss based on the coordinates of all annotated key points and all predicted key points in the training image, and obtaining shape loss based on the shape information and predicted edges fitted from all annotated key points in the training image.

[0093] In one implementation scenario, in response to obtaining multiple predicted key points output by the detection network, for each predicted key point, the position deviation between each predicted key point and the corresponding labeled key point is calculated, and the sum of the position deviations corresponding to all predicted key points is used as the position loss.

[0094] And, in response to obtaining the predicted edge output by the edge prediction network, the shape deviation between the shape information fitted by all the annotated key points in the training image and the shape information of the predicted edge is calculated, and the sum of the shape deviations corresponding to all the predicted edges is used as the shape loss.

[0095] Furthermore, based on the position loss and the shape loss, a training loss is obtained, and the training loss is used to adjust the parameters of the feature extraction network and the detection network.

[0096] In one implementation scenario, the sum of the position loss and the shape loss is used as the training loss, and the training loss is used to adjust the parameters in the feature extraction network and the detection network until the convergence adjustment is met to obtain a trained detection model.

[0097] In another implementation scenario, a first weight is set for the position loss, and a second weight is set for the shape loss. A first product of the position loss and the first weight is obtained, and a second product of the shape loss and the second weight is obtained. The sum of the first product and the second product is used as the training loss. The training loss is used to adjust the parameters of the feature extraction network and the detection network in the detection model until convergence adjustment is satisfied, thereby obtaining a trained detection model.

[0098] In addition to obtaining multiple predicted key points, the above scheme also uses the edge prediction network 30 to predict predicted edges, so as to adjust at least some parameters in the detection model according to the position information of the predicted key points and the shape information of the predicted edges. As a result, when the detection model obtained after training predicts the coordinates of key points in the image to be detected, it can predict multiple target key points according to the association information between key points in the same facial area, that is, the trained monitoring model also has the ability to constrain the shape formed by multiple key points. The accuracy of the obtained target key point position is greatly improved.

[0099] See also Figure 8 , Figure 8The key point detection system proposed in the present application includes: a processing module 40 , a division module 50 and a detection module 60 .

[0100] Specifically, in response to obtaining the image to be detected, the processing module 40 is used to obtain salient key points from the facial region in the image to be detected, and determine multiple reference key points in the image to be detected based on the salient key points, wherein the positions of the multiple salient key points are predefined.

[0101] In one embodiment, the processing module 40 obtains salient key points from a facial area in an image to be detected, and determines a plurality of reference key points in the image to be detected based on the salient key points, including: determining the facial area in the image to be detected, and obtaining all visible salient key points from the facial area based on the positions of all predefined salient key points; adjusting a universal face model based on the obtained salient key points to obtain a reference face model; wherein the universal face model is obtained based on a plurality of sample face models; and mapping the facial key points in the reference face model to the image to be detected to obtain a plurality of reference key points in the image to be detected.

[0102] In an implementation scenario, please continue to refer to Figure 8 The key point detection system proposed in the present application also includes a mapping module 41 coupled to the processing module 40, and the mapping module 41 is used to obtain all pre-set facial key points in the reference face model, and determine the control key points that match the significant key points from all facial key points; based on the significant key points and their matching control key points, determine the mapping relationship between the reference face model and the facial area in the image to be detected; based on the mapping relationship, map the facial key points in the reference face model to the image to be detected.

[0103] The division module 50 is used to divide the image to be detected into a plurality of areas to be detected based on the reference key points.

[0104] The detection module 60 is used to predict the coordinates of multiple key points in each area to be detected in sequence to obtain multiple target key points in each area to be detected.

[0105] In one embodiment, the detection module 60 predicts the coordinates of multiple key points in each area to be detected in sequence to obtain multiple target key points in each area to be detected, including: inputting the area to be detected into a feature extraction network to obtain regional image features output by the feature extraction network; inputting the regional image features into the detection network to obtain multiple target key points in the area to be detected outputted in sequence by the detection network using an attention mechanism; wherein the feature extraction network and the detection network constitute a detection model, and the detection model is obtained after training based on multiple training images, each training image corresponds to a partial facial area and contains multiple annotated key points, the annotated key points of some training images are obtained based on manual annotation, and the annotated key points of the remaining training images are obtained based on reference key points corresponding to significant key points in the training images.

[0106] In one implementation scenario, the detection module 60 inputs the regional image features into the detection network, and obtains multiple target key points in the area to be detected that are output in sequence by the detection network using the attention mechanism, including: inputting the content to be detected into the detection network, and obtaining the detection result output by the detection network; wherein the content to be detected includes at least the regional image features of the area to be detected, and the detection result includes all the output target key points in the area to be detected; updating the regional image features and the latest detection result to the content to be detected, returning to the step of inputting the content to be detected into the detection network, and obtaining the detection result output by the detection network, until the end condition is met, and all the target key points of the area to be detected are determined based on the latest detection result; wherein the end condition is related to the number or position of the target key points.

[0107] In one embodiment, please continue to refer to Figure 8 , the key point detection system proposed in the present application also includes a training module 61 coupled to the detection module 60. The training module 61 is used to train the detection model. When the detection model corresponds to an edge prediction network, the process of training the detection model by the training module 61 includes: inputting the training image into the feature extraction network to obtain the training image features; inputting the training image features into the detection network and the edge prediction network to obtain multiple predicted key points in the training image output in sequence by the detection network using the attention mechanism, and the predicted edges output by the edge prediction network using the thermal information corresponding to the training image features; based on the labeled key points, predicted key points and predicted edges, determining the training loss, using the training loss to adjust the parameters of the feature extraction network and the detection network until the convergence conditions are met, and obtaining the trained detection model.

[0108] In one implementation scenario, the training module 61 determines the training loss based on the labeled key points, predicted key points and predicted edges, including: obtaining the position loss based on the coordinates of all labeled key points and all predicted key points in the training image, and obtaining the shape loss based on the shape information and predicted edges fitted from all labeled key points in the training image; obtaining the training loss based on the position loss and the shape loss.

[0109] See also Fig. 9 , Fig. 9 : It is a structural diagram of an embodiment of an electronic device of the present application. The electronic device includes: a memory 70 and a processor 80 coupled to each other. The memory 70 stores program data, and the processor 80 is used to call the program data to implement the key point detection method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which is not limited here. In addition, the processor 80 can also be called a CPU (Center Processing Unit). The processor 80 may be an integrated circuit chip with signal processing capabilities. The processor 80 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 80 can be implemented by an integrated circuit chip.

[0110] See also Fig.10 , Fig.10 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 90 stores program data 91 that can be run by a processor. When the program data 91 is executed by the processor, the key point detection method mentioned in any of the above embodiments is implemented.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0115] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A key point detection method, characterized in that: include: In response to obtaining an image to be detected, obtaining salient key points from a facial region in the image to be detected, and determining a plurality of reference key points in the image to be detected based on the salient key points; wherein positions of the plurality of salient key points are predefined; Based on the reference key points, the image to be detected is divided into a plurality of areas to be detected; Predicting the coordinates of multiple key points in each of the to-be-detected areas in turn to obtain multiple target key points in each of the to-be-detected areas; The method of obtaining significant key points from the facial region in the image to be detected and determining a plurality of reference key points in the image to be detected based on the significant key points comprises: determining the facial region in the image to be detected and obtaining all visible significant key points from the facial region based on the positions of all predefined significant key points; adjusting a universal face model based on the obtained significant key points to obtain a reference face model; wherein the universal face model is obtained based on a plurality of sample face models; and mapping the facial key points in the reference face model to the image to be detected to obtain a plurality of reference key points in the image to be detected. The coordinates of multiple key points in each of the areas to be detected are predicted in sequence to obtain multiple target key points in each of the areas to be detected, including: inputting the area to be detected into a feature extraction network to obtain regional image features output by the feature extraction network; inputting the regional image features into a detection network to obtain multiple target key points in the area to be detected outputted in sequence by the detection network using an attention mechanism.

2. The method according to claim 1, characterized in that Mapping the facial key points in the reference face model to the image to be detected to obtain a plurality of reference key points in the image to be detected includes: Acquire all preset facial key points in the reference face model, and determine comparison key points that match the salient key points from all the facial key points; Determining a mapping relationship between the reference face model and the facial region in the image to be detected based on the salient key points and the matched control key points; Based on the mapping relationship, the facial key points in the reference face model are mapped to the image to be detected.

3. The method according to claim 1, characterized in that The feature extraction network and the detection network constitute a detection model, and the detection model is obtained after training based on multiple training images. Each of the training images corresponds to a partial facial area and contains multiple annotated key points. The annotated key points of some of the training images are obtained based on manual annotation, and the annotated key points of the remaining training images are obtained based on reference key points corresponding to the salient key points in the training images.

4. The method according to claim 3, characterized in that The inputting the regional image features into the detection network to obtain a plurality of target key points in the to-be-detected region outputted in sequence by the detection network using an attention mechanism comprises: Inputting the content to be detected into the detection network to obtain the detection result output by the detection network; wherein the content to be detected at least includes the regional image features of the area to be detected, and the detection result includes all the output target key points in the area to be detected; The image features of the area and the latest detection result are updated as the content to be detected, and the step of inputting the content to be detected into the detection network to obtain the detection result output by the detection network is returned until the end condition is met, and all the target key points of the area to be detected are determined based on the latest detection result; wherein the end condition is related to the number or position of the target key points.

5. The method according to claim 3, characterized in that: The detection model corresponds to an edge prediction network, and the training process of the detection model includes: Inputting the training image into the feature extraction network to obtain training image features; Inputting the training image features into the detection network and the edge prediction network, obtaining a plurality of predicted key points in the training image sequentially output by the detection network using an attention mechanism, and a predicted edge output by the edge prediction network using thermal information corresponding to the training image features; Based on the labeled key points, the predicted key points and the predicted edges, a training loss is determined, and the parameters of the feature extraction network and the detection network are adjusted using the training loss until a convergence condition is met, thereby obtaining the trained detection model.

6. The method according to claim 5, characterized in that The determining of the training loss based on the labeled key points, the predicted key points and the predicted edges includes: Based on the coordinates of all the annotated key points and all the predicted key points in the training image, a position loss is obtained; based on the shape information fitted by all the annotated key points in the training image and the predicted edges, a shape loss is obtained; The training loss is obtained based on the position loss and the shape loss.

7. A key point detection system, characterized in that: include: A processing module, in response to obtaining an image to be detected, the processing module is used to obtain significant key points from a facial area in the image to be detected, and determine a plurality of reference key points in the image to be detected based on the significant key points; wherein positions of the plurality of significant key points are predefined; A division module, used for dividing the image to be detected into a plurality of areas to be detected based on the reference key points; A detection module, used for predicting the coordinates of multiple key points in each of the to-be-detected areas in turn, to obtain multiple target key points in each of the to-be-detected areas; The method of obtaining significant key points from the facial region in the image to be detected and determining a plurality of reference key points in the image to be detected based on the significant key points comprises: determining the facial region in the image to be detected and obtaining all visible significant key points from the facial region based on the positions of all predefined significant key points; adjusting a universal face model based on the obtained significant key points to obtain a reference face model; wherein the universal face model is obtained based on a plurality of sample face models; and mapping the facial key points in the reference face model to the image to be detected to obtain a plurality of reference key points in the image to be detected. The coordinates of multiple key points in each of the areas to be detected are predicted in sequence to obtain multiple target key points in each of the areas to be detected, including: inputting the area to be detected into a feature extraction network to obtain regional image features output by the feature extraction network; inputting the regional image features into a detection network to obtain multiple target key points in the area to be detected outputted in sequence by the detection network using an attention mechanism.

8. An electronic device, characterized in that: include: A memory and a processor coupled to each other, the memory storing program data, and the processor calling the program data to execute the key point detection method according to any one of claims 1-6.

9. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the key point detection method according to any one of claims 1 to 6 is implemented.

Images