Method, device, apparatus and computer-readable medium for generating facial key point information

Abstract

The embodiments of the present disclosure disclose a method, apparatus, device, and computer-readable medium for generating facial key point information. A specific implementation of the method includes: obtaining a target facial image and a pre-labeled facial key point coordinate set; inputting the target facial image and the pre-labeled facial key point coordinate set into a pre-trained densely connected convolutional neural network to obtain a feature map set; obtaining a first output result set based on the feature map set and the pre-trained first convolutional neural network; obtaining a second output result set based on the feature map set and the pre-trained second convolutional neural network; generating a facial key point information set, wherein the facial key point information is used to represent the position information of the facial key points in the target facial image and the connection information between the facial key points in the target facial image. This implementation achieves effective detection of facial key points in dense crowds, providing a basis for facial expression analysis.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method, apparatus, device, and computer-readable medium for generating facial key point information. Background Art

[0002] Currently, facial landmark detection can be used to locate key facial regions, including eyebrows, eyes, nose, mouth, and facial contours. The facial landmark information obtained through facial landmark detection is often used in applications such as facial expression estimation. The typical approach is to first perform face detection. Then, facial landmark information is generated based on the face detection results.

[0003] However, when using the above method to generate facial key point information, the following technical problems often occur:

[0004] First, the method relies heavily on face detection. Once face detection fails, facial key point information cannot be generated.

[0005] Second, in dense crowds, the generation of facial key point information may be missed or the facial key point information of different people may be mistakenly generated as the facial key point information of one person. Summary of the Invention

[0006] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] Some embodiments of the present disclosure propose methods, devices, equipment, and computer-readable media for generating facial key point information to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a method, apparatus, device and computer-readable medium for generating facial key point information, the method comprising: obtaining a target facial image and a set of facial key point coordinates pre-annotated in the target facial image; inputting the target facial image and the pre-annotated facial key point coordinates into a pre-trained densely connected convolutional neural network to obtain a feature map set; obtaining a first output result set based on the feature map set and the pre-trained first convolutional neural network; obtaining a second output result set based on the feature map set and the pre-trained second convolutional neural network; generating a facial key point information set based on the first output result set and the second output result set, wherein the facial key point information is used to represent the position information of the facial key points in the target facial image and the connection information between the facial key points in the target facial image.

[0009] In a second aspect, some embodiments of the present disclosure provide a device for generating facial key point information, the device comprising: an acquisition unit, configured to acquire a target facial image and a set of facial key point coordinates pre-annotated in the above target facial image; a first generation unit, configured to input the above target facial image and the above pre-annotated facial key point coordinate set into a pre-trained densely connected convolutional neural network to obtain a feature map set; a second generation unit, configured to obtain a first output result set based on the above feature map set and the pre-trained first convolutional neural network; a third generation unit, configured to obtain a second output result set based on the above feature map set and the pre-trained second convolutional neural network; a fourth generation unit, configured to generate a facial key point information set based on the above first output result set and the above second output result set, wherein the above facial key point information is used to characterize the position information of the facial key points in the above target facial image and the connection information between the facial key points in the above target facial image.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0012] The aforementioned embodiments of the present disclosure have the following beneficial effects: The facial key point information generated by the facial key point information generation methods of some embodiments of the present disclosure eliminates reliance on face detection methods. Specifically, errors in facial key point information generation occur due to a heavy reliance on face detection methods. Once face detection fails, facial key point information generation cannot be performed. Therefore, in some embodiments of the present disclosure, the neural network in the facial key point information generation methods includes not only a densely connected convolutional neural network, but also a first convolutional neural network and a second convolutional neural network. First, the densely connected convolutional neural network performs feature extraction on the target facial image and a set of pre-annotated facial key point coordinates to generate a feature map. Densely connected convolutional neural networks can significantly alleviate the vanishing gradient problem. The feature map set is then input into the first convolutional neural network and the second convolutional neural network, respectively. The first convolutional neural network then further extracts features from the feature map set to generate a first output result set. The first output result obtained by the first convolutional neural network includes predicted facial key point coordinates and the confidence level of the predicted results. Next, the second convolutional neural network predicts facial key point association information on the feature map set to obtain a second output result set. This second output result set includes predictions of facial pixels and association information of facial key point coordinates. Finally, by further analyzing the first and second output result sets, the facial key point coordinates corresponding to each face in the facial image are connected to obtain a facial key point information set. This facial key point information set can represent the positional information of facial key points in the target facial image and the connection information between facial key points in the target facial image. Due to the participation of the first and second convolutional neural networks, the facial key point information generation method is changed to first predicting facial key points using the first convolutional neural network to obtain a first output result set, and then predicting association information of facial key point coordinates using the second convolutional neural network to obtain a second output result set. The first and second output result sets are then mapped to the corresponding faces to obtain a facial key point information set. This makes the generation of facial key point information no longer dependent on face detection methods. This improves the possibility of generating facial key point information. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1is a schematic diagram of an application scenario of the method for generating facial key point information according to some embodiments of the present disclosure;

[0015] Figure 2 is a flowchart of some embodiments of the method for generating facial key point information according to the present disclosure;

[0016] Figure 3 is a schematic structural diagram of some embodiments of the apparatus for generating facial key point information according to the present disclosure;

[0017] Figure 4 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 It is a schematic diagram of an application scenario of the method for generating facial key point information in some embodiments of the present disclosure.

[0025] exist Figure 1In the application scenario, first, the computing device 101 can obtain a target facial image 105 and a pre-annotated facial key point coordinate set 106 in the target facial image. Then, the target facial image 105 and the pre-annotated facial key point coordinate set 106 are input into a pre-trained densely connected convolutional neural network 102 to obtain a feature map set 107. Afterwards, based on the feature map set 107 and the pre-trained first convolutional neural network 103, a first output result set 108 is obtained. Then, based on the feature map set 107 and the pre-trained second convolutional neural network 104, a second output result set 109 is obtained. Finally, based on the first output result set 108 and the second output result set 109, a facial key point information set 110 is generated, wherein the facial key point information is used to represent the position information of the facial key points in the target facial image and the connection information between the facial key points in the target facial image.

[0026] It is understandable that the method for generating facial key point information can be executed by a terminal device, or it can also be executed by a computing device 101. The execution subject of the above method can also include a device formed by integrating the above terminal device and the above computing device 101 through a network, or it can also be executed by various software programs. Among them, the terminal device can be various electronic devices with information processing capabilities, including but not limited to smart phones, tablet computers, e-book readers, laptop computers and desktop computers. The execution subject can also be embodied as a computing device 101, software, etc. When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0027] It should be understood that Figure 1 The number of computing devices in the embodiment is merely illustrative. Any number of computing devices may be provided according to implementation requirements.

[0028] Continue to refer Figure 2 , shows a process 200 of some embodiments of the method for generating facial key point information according to the present disclosure. The method for generating facial key point information includes the following steps:

[0029] Step 201: Obtain a target face image and a set of facial key point coordinates pre-annotated in the target face image.

[0030] In some embodiments, the execution subject of the method for generating facial key point information (eg Figure 1The computing device 101 shown can obtain a target facial image and a set of pre-annotated facial key point coordinates in the target facial image via a wired or wireless connection. The target facial image can be an image containing at least one face, such as a photograph of a face. The set of facial key point coordinates can be a set of facial key point coordinates in the target facial image obtained using a data annotation method (e.g., a polygon annotation method).

[0031] As an example, the target face image may be a facial expression image, and the facial key point coordinate set may be [(10, 12), (14, 12)].

[0032] Step 202: Input the target face image and the pre-labeled facial key point coordinate set into a pre-trained densely connected convolutional neural network to obtain a feature map set.

[0033] In some embodiments, the execution entity may input the target facial image and the facial key point coordinate set into a pre-trained densely connected convolutional neural network (DenseNet convolutional neural network) to obtain a feature map set. The pre-trained densely connected convolutional neural network may be a densely connected convolutional neural network obtained by inputting a facial image set and a set of annotation files related to the facial image set into a convolutional neural network for model training. The annotation file may be a file used to characterize facial features in a facial image. The facial features may include facial position coordinates, facial key point coordinates, and the like.

[0034] As an example, the feature map set may be a set of feature vectors containing a series of features of the target facial image.

[0035] In some optional implementations of some embodiments, the densely connected convolutional neural network includes a first subnetwork, a first transition layer, a second subnetwork, a second transition layer, a third subnetwork, a third transition layer, and a fourth subnetwork; and the target face image and the pre-labeled facial key point coordinate set are input into the pre-trained densely connected convolutional neural network to obtain a feature map set. The following steps are included:

[0036] In the first step, the target face image and the pre-labeled face key point coordinate set are input into the first sub-network to obtain a first sub-feature map set.

[0037] As an example, the first sub-network may be a network comprising six 1×1 convolution kernels and six 3×3 convolution kernels. The first sub-feature map set may be a set of feature vectors that store a series of features of the target face image.

[0038] In the second step, the first sub-feature map set is input into the first transition layer to obtain a second sub-feature map set.

[0039] As an example, the first transition layer may be a network comprising a 1×1 convolution kernel and a 2×2 average pooling layer, wherein the stride of the average pooling layer may be 2. The second sub-feature map set may be a set of feature vectors that store a series of features of the target facial image.

[0040] In the third step, the second sub-feature map set is input into the second sub-network to obtain a third sub-feature map set.

[0041] As an example, the second sub-network may be a network comprising 12 1×1 convolution kernels and 6 3×3 convolution kernels. The third sub-feature map set may be a set of feature vectors that store a series of features of the target face image.

[0042] In the fourth step, the first sub-feature map set and the third sub-feature map set are input into the second transition layer to obtain a fourth sub-feature map set.

[0043] As an example, the second transition layer may be a network comprising a 1×1 convolution kernel and a 2×2 average pooling layer, wherein the stride of the average pooling layer may be 2. The fourth sub-feature map set may be a set of feature vectors that store a series of features of the target facial image.

[0044] In the fifth step, the fourth sub-feature map set is input into the third sub-network to obtain the fifth sub-feature map set.

[0045] As an example, the third sub-network may be a network comprising 32 1×1 convolution kernels and 6 3×3 convolution kernels. The fifth sub-feature map set may be a set of feature vectors that store a series of features of the target face image.

[0046] In the sixth step, the first sub-feature map set, the third sub-feature map set, and the fifth sub-feature map set are input into the third transition layer to obtain a sixth sub-feature map set.

[0047] As an example, the third transition layer may be a network comprising a 1×1 convolution kernel and a 2×2 average pooling layer, wherein the stride of the average pooling layer may be 2. The sixth sub-feature map set may be a set of feature vectors that store a series of features of the target facial image.

[0048] In the seventh step, the sixth sub-feature map set is input into the fourth sub-network to obtain the above-mentioned feature map set.

[0049] As an example, the fourth sub-network may be a network including 32 1×1 convolution kernels and 6 3×3 convolution kernels.

[0050] Step 203: Obtain a first output result set based on the feature map set and the pre-trained first convolutional neural network.

[0051] In some embodiments, the execution entity may obtain a first output result set based on the feature map set and a pre-trained first convolutional neural network. The first output result set may include a prediction result set of facial key point coordinates and a confidence set of the prediction results. The prediction result set of facial key point coordinates may be a set of facial key point coordinates predicted by the first convolutional neural network training. The confidence set may be a set of similarities between the prediction results in the prediction result set of facial key point coordinates and the corresponding facial key point coordinates in the pre-labeled facial key point coordinate set. The pre-trained first convolutional neural network may be a convolutional neural network obtained by inputting the feature map set of a facial image into the first convolutional neural network for confidence prediction training.

[0052] As an example, the first convolutional neural network may be a network comprising three 3×3 convolutional layers and two 1×1 convolutional layers. The first output may be a prediction result of the key point information, for example, [(10.5, 11.9), 0.95]. The prediction result may be [(10.5, 11.9)]. The confidence level of the prediction result may be a probability value of the predicted facial key point coordinates corresponding to the set of facial key point coordinates, for example, 0.95.

[0053] Step 204: Obtain a second output result set based on the feature map set and the pre-trained second convolutional neural network.

[0054] In some embodiments, the execution entity may obtain a second output result set based on the feature map set and a pre-trained second convolutional neural network, wherein the second output result includes association information between each pair of facial key point coordinates in the facial key point coordinate set. The association information may be information about the affinity between each pair of facial key point coordinates. The pre-trained second convolutional neural network may be a convolutional neural network obtained by inputting the feature map set of a facial image into the second convolutional neural network for facial key point affinity prediction training.

[0055] As an example, the second convolutional neural network can be a network consisting of three 3×3 convolutional layers and two 1×1 convolutional layers. The second output can be a predicted affinity for every two facial landmarks in the target facial image. For example, [(10, 12), (14, 12), 0.95] indicates that the affinity for the facial landmarks (10, 12) and (14, 12) is 0.95.

[0056] In some optional implementations of some embodiments, the feature map set is input into the second convolutional neural network to obtain a feature information set, wherein the feature information set may be a set of pixel coordinates of each feature map in the feature map set.

[0057] As an example, the above feature information set may be [(1, 2), (3, 4), (5, 6)].

[0058] In some optional implementations of some embodiments, the above-mentioned feature information set, the above-mentioned pre-labeled facial key point coordinate set and the above-mentioned feature map set are combined to obtain a target feature map set.

[0059] As an example, the above target feature map set can be [(1, 2), (3, 4), (5, 6), (10, 12), (14, 12)].

[0060] In some optional implementations of some embodiments, the feature information includes pixel coordinates; and the processing of each target feature map in the target feature map set to generate a second output result may include the following steps:

[0061] In the first step, in response to the pixel coordinates included in the target feature map satisfying the first preset condition, a sub-output result is obtained according to the following formula:

[0062]

[0063] Where P represents the coordinates of the key points of the face. C represents the pixel coordinates. m represents the face. i represents the face number. j represents the pixel number. k and q represent the numbers of the key points of the face, and k and q are not equal. i Represents the i-th face in the above target face image. Represents the coordinates of the kth facial key point in the i-th face in the above target face image. Represents the coordinates of the qth facial key point in the i-th face in the above target face image. represents the pixel coordinates of the jth pixel in the i-th face in the target face image. The horizontal axis value of . express The horizontal axis value of . express The horizontal axis value of . express The horizontal axis value of . Represents the sub-output result of the j-th pixel in the i-th face in the above target face image. A vector representing the coordinates of the kth facial key point in the i-th face in the target facial image and the pixel coordinates of the j-th pixel point in the i-th face in the target facial image. A vector representing the coordinates of the kth facial key point in the i-th face in the target facial image and the coordinates of the qth facial key point in the i-th face in the target facial image. Represents a vector Model. Represents a vector The first preset condition may be that the pixel coordinates are located on the target face image.

[0064] As an example, the above-mentioned face serial number can be 1. The above-mentioned pixel serial number can be 1. The serial numbers of the above-mentioned facial key points can be 1 and 2 respectively. The coordinates of the first facial key point in the first face in the above-mentioned target facial image can be (10, 12). The coordinates of the second facial key point in the first face in the above-mentioned target facial image can be (14, 12). The pixel coordinates of the first pixel in the first face in the above-mentioned target facial image can be (11, 13). The vector of the coordinates of the first facial key point in the first face in the above-mentioned target facial image and the pixel coordinates of the first pixel in the first face in the above-mentioned target facial image can be (1, 1). The vector of the coordinates of the first facial key point in the first face in the above-mentioned target facial image and the coordinates of the second facial key point in the first face in the above-mentioned target facial image can be (2, 0). The sub-output result of the first pixel in the first face in the above-mentioned target facial image can be 1. (The calculation process is as follows):

[0065]

[0066] In the second step, in response to the above sub-output result meeting the second preset condition, the above sub-output result is input into the following formula to obtain the second output result:

[0067]

[0068] Where U represents the second output result. m represents a face. c represents a pixel coordinate. i represents a face number. j represents a pixel number. represents the sub-output result of the jth pixel in the i-th face in the target face image. n represents the total number of faces in the target face image. S represents the total number of pixels in the target face image. N represents the total number of sub-output results in the above sub-output result set that meet the second preset condition. The above second preset condition can be that the above sub-output result belongs to (0, 1].

[0069] As an example, the total number of faces in the target facial image may be 1. The total number of pixels in the target facial image may be 3. The total number of sub-output results in the sub-output result set that meet the second preset condition may be 3. The sub-output result of the first pixel in the first face in the target facial image may be 1. The sub-output result of the second pixel in the first face in the target facial image may be 0.7. The sub-output result of the third pixel in the first face in the target facial image may be 0.4. The second output result may be 0.7. (The calculation process is as follows):

[0070]

[0071] Step 205: Generate a facial key point information set based on the first output result set and the second output result set.

[0072] In some embodiments, the execution entity may generate a set of facial key point information based on the first output result set and the second output result set, wherein the facial key point information is used to represent the position information of the facial key points in the target facial image and the connection information between the facial key points in the target facial image.

[0073] As an example, the first output result may be [(10.5, 11.9), 0.9], [(13.5, 11.9), 0.85], and the second output result may be [(10, 12), (14, 12), 0.9].

[0074] In some optional implementations of some embodiments, each first output result in the first output result set is combined with the corresponding second output result in the second output result set to generate a third output result, thereby obtaining a third output result set.

[0075] As an example, the first output result can be [(10.5, 11.9), 0.9], [(13.5, 11.9), 0.85]. Where (10.5, 11.9) and (13.5, 11.9) represent the predicted facial key point coordinates, respectively. 0.9 and 0.85 represent the confidence levels of the predicted facial key point coordinates and the facial key point coordinate sets corresponding thereto, respectively. The second output result can be [(10, 12), (14, 12), 0.9]. Where the facial key point coordinates corresponding to the predicted facial key point coordinates (10.5, 11.9) are (10, 12), and the confidence level is 0.9. The facial key point coordinates corresponding to the predicted facial key point coordinates (13.5, 11.9) are (14, 12), and the confidence level is 0.85. 0.9 indicates the affinity between the facial key point coordinates (10, 12) and (14, 12). The third output result can be [(10.5, 11.9), (13.5, 11.9), 0.9].

[0076] In some optional implementations of some embodiments, the predicted results in each of the third output results in the third output result set are connected to generate facial key point information, thereby obtaining a facial key point set. The connection processing may be connecting the predicted results in the third output results.

[0077] As an example, the third output result may be [(10.5, 11.9), (13.5, 11.9), 0.9]. The predicted results in the third output result may be (10.5, 11.9), (13.5, 11.9).

[0078] The above formula, as an inventive feature of an embodiment of the present disclosure, addresses the second technical issue mentioned in the background technology: "In dense crowds, facial key point information may be omitted or incorrectly generated as the facial key point information of different people." Factors that often contribute to omissions or errors in the generation of facial key point information are as follows: Facial key point information generation typically involves performing face detection. Then, facial key point information is generated based on the face detection results. In this case, successful face detection is required for facial key point information generation. Unsatisfactory face detection can lead to omissions or errors in the generation of facial key point information. Addressing these factors significantly improves the success rate of facial key point information generation. To achieve this, the above formula introduces formulas related to facial key point information generation. First, in response to pixel coordinates included in the target feature map satisfying a first preset condition, the pixel coordinates are evaluated to obtain a sub-output result. The sub-output result can be used to determine the affinity of the pixel coordinates in the target feature map. Then, in response to the above-mentioned sub-output results satisfying the second preset condition, a sum operation is performed on the sub-output results satisfying the second preset condition to obtain a sum value. The above-mentioned sum value includes the affinity between the pixel point and the facial key points in the above-mentioned target feature image. Afterwards, an average operation is performed on the above-mentioned sum value to obtain a second output result. The second output result includes the affinity between the facial key points in the above-mentioned target feature image. Finally, according to the affinity between the facial key points and the predicted results of the facial key points, the corresponding faces are corresponded to obtain a set of facial key point information. Thereby, the problem that the generation of facial key point information may be omitted or the facial key point information of different people may be mistakenly generated as the facial key point information of one person is solved.

[0079] The aforementioned embodiments of the present disclosure have the following beneficial effects: The facial key point information generated by the facial key point information generation methods of some embodiments of the present disclosure eliminates reliance on face detection methods. Specifically, errors in facial key point information generation occur due to a heavy reliance on face detection methods. Once face detection fails, facial key point information generation cannot be performed. Therefore, in some embodiments of the present disclosure, the neural network in the facial key point information generation methods includes not only a densely connected convolutional neural network, but also a first convolutional neural network and a second convolutional neural network. First, the densely connected convolutional neural network performs feature extraction on the target facial image and a set of pre-annotated facial key point coordinates to generate a feature map. Densely connected convolutional neural networks can significantly alleviate the vanishing gradient problem. The feature map set is then input into the first convolutional neural network and the second convolutional neural network, respectively. The first convolutional neural network then further extracts features from the feature map set to generate a first output result set. The first output result obtained by the first convolutional neural network includes predicted facial key point coordinates and the confidence level of the predicted results. Next, the second convolutional neural network predicts facial key point association information on the feature map set to obtain a second output result set. This second output result set includes predictions of facial pixels and association information of facial key point coordinates. Finally, by further analyzing the first and second output result sets, the facial key point coordinates corresponding to each face in the facial image are connected to obtain a facial key point information set. This facial key point information set can represent the positional information of facial key points in the target facial image and the connection information between facial key points in the target facial image. Due to the participation of the first and second convolutional neural networks, the facial key point information generation method is changed to first predicting facial key points using the first convolutional neural network to obtain a first output result set, and then predicting association information of facial key point coordinates using the second convolutional neural network to obtain a second output result set. The first and second output result sets are then mapped to the corresponding faces to obtain a facial key point information set. This makes the generation of facial key point information no longer dependent on face detection methods. This improves the possibility of generating facial key point information.

[0080] Further references Figure 3 As an implementation of the above methods in the above figures, the present disclosure provides some embodiments of a device for generating facial key point information. These device embodiments are similar to Figure 2 Corresponding to the above method embodiments, the device can be specifically applied to various electronic devices.

[0081] like Figure 3As shown, the key point information generation 300 of some embodiments includes: an acquisition unit 301 , a first generation unit 302 , a second generation unit 303 , a third generation unit 304 and a fourth generation unit 305 . Among them, the acquisition unit 301 is configured to obtain the target facial image and the pre-annotated facial key point coordinate set in the above target facial image; the first generation unit 302 is configured to input the above target facial image and the above pre-annotated facial key point coordinate set into a pre-trained densely connected convolutional neural network to obtain a feature map set; the second generation unit 303 is configured to obtain a first output result set based on the above feature map set and the pre-trained first convolutional neural network; the third generation unit 304 is configured to obtain a second output result set based on the above feature map set and the pre-trained second convolutional neural network; the fourth generation unit 305 is configured to generate a facial key point information set based on the above first output result set and the above second output result set, wherein the above facial key point information is used to represent the position information of the facial key points in the above target facial image and the connection information between the facial key points in the above target facial image.

[0082] It is understood that the units described in the device 300 are similar to those in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 300 and the units included therein, and will not be repeated here.

[0083] Reference below Figure 4 , which shows an electronic device (eg, Figure 1 Schematic diagram of the structure of the computing device 101)400. Figure 4 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0084] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0085] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0086] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0087] It should be noted that in some embodiments of the present disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0088] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0089] The computer-readable medium may be included in the device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a target facial image and a set of facial key point coordinates pre-labeled in the target facial image; inputs the target facial image and the pre-labeled facial key point coordinates into a pre-trained densely connected convolutional neural network to obtain a feature map set; obtains a first output result set based on the feature map set and the pre-trained first convolutional neural network; obtains a second output result set based on the feature map set and the pre-trained second convolutional neural network; generates a facial key point information set based on the first output result set and the second output result set, wherein the facial key point information is used to represent the position information of the facial key points in the target facial image and the connection information between the facial key points in the target facial image.

[0090] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] The units described in some embodiments of the present disclosure may be implemented in software or in hardware. The described units may also be provided in a processor. For example, they may be described as follows: a processor includes an acquisition unit, a first generation unit, a second generation unit, a third generation unit, and a fourth generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves. For example, the acquisition unit may also be described as "a unit for acquiring a target facial image and a set of facial key point coordinates pre-annotated in the target facial image."

[0093] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0094] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure to form a technical method.

Claims

1. A method for generating facial key point information, comprising: Obtaining a target facial image and a set of facial key point coordinates pre-annotated in the target facial image, wherein the target facial image includes at least two facial images; Inputting the target face image and the pre-annotated facial key point coordinate set into a pre-trained densely connected convolutional neural network to obtain a feature map set; Obtaining a first output result set based on the feature map set and a pre-trained first convolutional neural network, wherein the first output result set includes a prediction result set of facial key point coordinates and a confidence set of the prediction results, the prediction result set of facial key point coordinates being a set of facial key point coordinates predicted by the first convolutional neural network training, and the confidence set being a set of similarities between the prediction results in the prediction result set of facial key point coordinates and the corresponding facial key point coordinates in the pre-labeled facial key point coordinate set; Obtaining a second output result set based on the feature map set and a pre-trained second convolutional neural network; Generate a facial key point information set based on the first output result set and the second output result set, wherein the facial key point information is used to represent position information of facial key points in the target facial image and connection information between facial key points in the target facial image; The second output result set is obtained based on the feature map set and the pre-trained second convolutional neural network, including: Inputting the feature map set into the second convolutional neural network to obtain a feature information set; Combining the feature information set, the pre-labeled facial key point coordinate set, and the feature map set to obtain a target feature map set; Processing each target feature map in the target feature map set to generate a second output result, obtaining a second output result set, wherein the feature information includes pixel coordinates, including: In response to the pixel coordinates included in the target feature map satisfying the first preset condition, a sub-output result is obtained according to the following formula: , in, Represents a human face, Indicates the face sequence number, Indicates the pixel number, 、 represents the sequence number of the facial key points, and and Not equal, Indicates the first A personal face, Indicates the first The first in the face Coordinates of key points of individual faces, Indicates the first The first in the face Coordinates of key points of individual faces, Indicates the first The first in the face The pixel coordinates of the pixel point, express The horizontal axis value of express The horizontal axis value of express The horizontal axis value of Indicates the first The first in the face The sub-output result of pixels, Indicates the first The first in the face The coordinates of the key points of the individual face are consistent with the first The first in the face The vector of pixel coordinates of pixels, Indicates the first The first in the face The coordinates of the key points of the individual face are consistent with the first The first in the face The vector of coordinates of key points of a person's face, Represents a vector The model, Represents a vector The model; In response to the sub-output result satisfying the second preset condition, the sub-output result is input into the following formula to obtain a second output result: ， in, represents the second output result, represents the total number of faces in the target face image, represents the total number of pixels in the target face image, Indicates the total number of sub-output results in the sub-output result set that meet the second preset condition.

2. The method according to claim 1, wherein The densely connected convolutional neural network includes a first sub-network, a first transition layer, a second sub-network, a second transition layer, a third sub-network, a third transition layer and a fourth sub-network; and The target face image and the pre-labeled face key point coordinate set are input into a pre-trained densely connected convolutional neural network to obtain a feature map set, including: Inputting the target face image and the pre-labeled face key point coordinate set into the first sub-network to obtain a first sub-feature map set; Inputting the first sub-feature map set into the first transition layer to obtain a second sub-feature map set; Inputting the second sub-feature map set into the second sub-network to obtain a third sub-feature map set; Inputting the first sub-feature map set and the third sub-feature map set into the second transition layer to obtain a fourth sub-feature map set; Inputting the fourth sub-feature map set into the third sub-network to obtain a fifth sub-feature map set; Inputting the first sub-feature map set, the third sub-feature map set, and the fifth sub-feature map set into the third transition layer to obtain a sixth sub-feature map set; The sixth sub-feature map set is input into the fourth sub-network to obtain the feature map set.

3. The method according to claim 1, wherein The step of obtaining a first output result set based on the feature map set and a pre-trained first convolutional neural network includes: Inputting the feature map set into the first convolutional neural network to obtain a set of prediction results of facial key point coordinates; Performing confidence generation processing on each prediction result in the prediction result set and the facial key point coordinates corresponding to the prediction result in the pre-labeled facial key point coordinate set to generate a confidence, thereby obtaining a confidence set; Determining each prediction result in the prediction result set and the confidence level corresponding to the prediction result in the confidence level set as a target output result, to obtain a target output result set; Target output results that meet a first preset condition are screened out from the target output result set as first output results to obtain a first output result set.

4. The method according to claim 1, wherein The generating a facial key point information set based on the first output result set and the second output result set includes: Combining each first output result in the first output result set with the corresponding second output result in the second output result set to generate a third output result, thereby obtaining a third output result set; The prediction results in each third output result in the third output result set are connected to generate facial key point information to obtain a facial key point set.

5. A device for generating facial key point information, comprising: an acquiring unit configured to acquire a target face image and a set of facial key point coordinates pre-annotated in the target face image, wherein the target face image includes at least two face images; A first generating unit is configured to input the target face image and the pre-annotated face key point coordinate set into a pre-trained densely connected convolutional neural network to obtain a feature map set; a second generating unit, configured to obtain a first output result set based on the feature map set and a pre-trained first convolutional neural network, wherein the first output result set includes a prediction result set of facial key point coordinates and a confidence set of the prediction results, the prediction result set of facial key point coordinates being a set of facial key point coordinates predicted by the first convolutional neural network training, and the confidence set being a set of similarities between the prediction results in the prediction result set of facial key point coordinates and the corresponding facial key point coordinates in the pre-labeled facial key point coordinate set; A third generating unit is configured to obtain a second output result set based on the feature map set and a pre-trained second convolutional neural network; a fourth generating unit, configured to generate a facial key point information set based on the first output result set and the second output result set, wherein the facial key point information is used to represent position information of facial key points in the target facial image and connection information between facial key points in the target facial image; Wherein, the third generating unit is further configured to: Inputting the feature map set into the second convolutional neural network to obtain a feature information set; Combining the feature information set, the pre-labeled facial key point coordinate set, and the feature map set to obtain a target feature map set; Processing each target feature map in the target feature map set to generate a second output result, obtaining a second output result set, wherein the feature information includes pixel coordinates, including: In response to the pixel coordinates included in the target feature map satisfying the first preset condition, a sub-output result is obtained according to the following formula: , in, Represents a face, Indicates the face sequence number, Indicates the pixel number, 、 represents the sequence number of the facial key points, and and Not equal, Indicates the first A face, Indicates the first The first in the face Coordinates of key points of individual faces, Indicates the first The first in the face Coordinates of key points of individual faces, Indicates the first The first in the face The pixel coordinates of the pixel point, express The horizontal axis value of express The horizontal axis value of express The horizontal axis value of Indicates the first The first in the face The sub-output result of pixels, Indicates the first The first in the face The coordinates of the key points of the individual face are consistent with the first The first in the face The vector of pixel coordinates of pixels, Indicates the first The first in the face The coordinates of the key points of the individual face are consistent with the first The first in the face The vector of coordinates of key points of a person's face, Represents a vector The model, Represents a vector The model; In response to the sub-output result satisfying the second preset condition, the sub-output result is input into the following formula to obtain a second output result: ， in, represents the second output result, represents the total number of faces in the target face image, represents the total number of pixels in the target face image, Indicates the total number of sub-output results in the sub-output result set that meet the second preset condition.

6. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer-readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images