Face key point detection method, storage medium and intelligent device
By using collaborative training of backbone network and multi-task detection head in face key point detection, combined with multi-modal image and data enhancement, the problem of poor detection accuracy in on-board environment is solved, and the accuracy and robustness of detection are improved .
Patent Information
- Application Number
- CN202411459825.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-01-03
AI Technical Summary
Face key point detection is affected by factors such as light, occlusion, and posture in an on-board environment. The detection accuracy is poor and other features cannot be effectively integrated and utilized.
The detection model including backbone network, key point position detection head and auxiliary information detection head is adopted. Through collaborative training and multi-task loss function optimization, combined with visible light and infrared image modes, data enhancement technology is used to improve the robustness of the model.
It improves the accuracy and stability of face key point detection, enhances the robustness and generalization capabilities of the model, and adapts to complex vehicle-mounted environments.
Smart Images
Figure CN119251894A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of facial key point detection, and specifically provides a facial key point detection method, a storage medium and an intelligent device. Background Art
[0002] Facial key point detection refers to the location of key areas of the face, including eyebrows, eyes, nose, mouth, facial contours, etc. Facial key point detection is easily affected by various factors such as lighting, occlusion, posture, etc., especially in special environments such as in-cabin environments. In-cabin lighting, facial posture, occlusions, and seat positions can all affect the accuracy of facial key point detection. Conventional facial key point detection solutions usually only focus on the key point locations and fail to effectively integrate and utilize other features. Therefore, it is impossible to accurately and comprehensively solve the problem of facial key point detection in a vehicle environment.
[0003] Accordingly, the art needs a new facial key point detection solution to solve the above problems. Summary of the invention
[0004] The present application aims to solve the above-mentioned technical problem, namely, to solve the problem that facial key point detection fails to effectively integrate and utilize other features resulting in poor detection accuracy.
[0005] In a first aspect, the present application provides a method for detecting facial key points, comprising: obtaining a facial image; inputting the facial image into a trained detection model to obtain a facial key point detection result, wherein the detection model comprises a backbone network for extracting image features, a key point position detection head connected to the backbone network, and an auxiliary information detection head connected to the backbone network.
[0006] In a technical solution of the above-mentioned facial key point detection method, the method further includes: training the detection model, wherein the training includes collaborative training of the backbone network, the key point position detection head and the auxiliary information detection head, and the loss function used is: Loss 总 =λ1×Loss1+λ2×Loss2, where Loss1 is the loss function used by the key point position detection head, and λ1 is its corresponding weight; Loss2 is the loss function used by the auxiliary information detection head, and λ2 is its corresponding weight. 总 is the total loss function of the detection model.
[0007] In a technical solution of the above-mentioned facial key point detection method, the Loss1 is a Smooth L1 loss function.
[0008] In a technical solution of the above-mentioned face key point detection method, the Loss1 is composed of the Smooth L1 loss function and the data enhancement loss function Loss transform Composition: Loss1 = w1 × Smooth L1 + w2 × Loss transform , wherein the Loss transform Represents the difference between the first prediction result and the second prediction result, wherein the first prediction result is the key point position prediction result A obtained by inputting the original face image sample into the detection model after data enhancement and then inversely processing the prediction result A with data enhancement; the second prediction result is the key point position prediction result B obtained by inputting the original face image sample into the detection model, and wherein w1 and w2 are weights.
[0009] In a technical solution of the above-mentioned facial key point detection method, the auxiliary information detection head includes: a key point visibility detection head and / or a head posture detection head.
[0010] In a technical solution of the above-mentioned facial key point detection method, when the auxiliary information detection head is a key point visibility detection head, the Loss2 is a cross entropy loss function.
[0011] In a technical solution of the above-mentioned facial key point detection method, when the auxiliary information detection head is a head posture detection head, the Loss2 is a Smooth L1 loss function or Loss2=w3×Smooth L1 loss function+w4×Ranking loss function, where w3 and w4 are weights.
[0012] In a technical solution of the above-mentioned facial key point detection method, when the auxiliary information detection head is composed of a key point visibility detection head and a head posture detection head, the Loss2 = w5 × cross entropy loss function + w6 × Smooth L1 loss function; or the Loss2 = w5 × cross entropy loss function + w3 × Smooth L1 loss function + w4 × Ranking loss function, where w3-w6 are weights.
[0013] In a technical solution of the above-mentioned facial key point detection method, the facial image training samples include facial images in two modalities: visible light images and infrared images.
[0014] In a technical solution of the above-mentioned facial key point detection method, the facial image training samples include a first type of training samples and a second type of training samples, wherein the first type of training samples include a first number of facial key points, and the second type of training samples include a second number of facial key points.
[0015] In a second aspect, the present application provides a facial key point detection device, comprising: an acquisition module for acquiring a facial image; a detection module for inputting the facial image into a trained detection model to obtain a facial key point detection result, wherein the detection model comprises a backbone network for extracting image features, a key point position detection head connected to the backbone network, and an auxiliary information detection head connected to the backbone network.
[0016] In a technical solution of the above-mentioned facial key point detection device, the device further includes: a training module for training the detection model, wherein the training includes collaborative training of the backbone network, the key point position detection head and the auxiliary information detection head, and the loss function used is: Loss 总 =λ1×Loss1+λ2×Loss2, where Loss1 is the loss function used by the key point position detection head, and λ1 is its corresponding weight; Loss2 is the loss function used by the auxiliary information detection head, and λ2 is its corresponding weight. 总 is the total loss function of the detection model.
[0017] In a technical solution of the above-mentioned facial key point detection device, the Loss1 is a Smooth L1 loss function.
[0018] In a technical solution of the above-mentioned face key point detection device, the Loss1 is composed of the Smooth L1 loss function and the data enhancement loss function Loss transform Composition: Loss1 = w1 × Smooth L1 + w2 × Loss transform , wherein the Loss transform Represents the difference between the first prediction result and the second prediction result, wherein the first prediction result is the key point position prediction result A obtained by inputting the original face image sample into the detection model after data enhancement and then inversely processing the prediction result A with data enhancement; the second prediction result is the key point position prediction result B obtained by inputting the original face image sample into the detection model, and wherein w1 and w2 are weights.
[0019] In a technical solution of the above-mentioned facial key point detection device, the auxiliary information detection head includes: a key point visibility detection head and / or a head posture detection head.
[0020] In a technical solution of the above-mentioned facial key point detection device, when the auxiliary information detection head is a key point visibility detection head, the Loss2 is a cross entropy loss function.
[0021] In a technical solution of the above-mentioned facial key point detection device, when the auxiliary information detection head is a head posture detection head, the Loss2 is a Smooth L1 loss function or Loss2=w3×Smooth L1 loss function+w4×Ranking loss function, where w3 and w4 are weights.
[0022] In a technical solution of the above-mentioned facial key point detection device, when the auxiliary information detection head is composed of a key point visibility detection head and a head posture detection head, the Loss2 = w5 × cross entropy loss function + w6 × Smooth L1 loss function; or the Loss2 = w5 × cross entropy loss function + w3 × Smooth L1 loss function + w4 × Ranking loss function, where w3-w6 are weights.
[0023] In a technical solution of the above-mentioned facial key point detection device, the facial image training samples include facial images in two modalities: visible light images and infrared images.
[0024] In a technical solution of the above-mentioned facial key point detection device, the facial image training samples include a first type of training samples and a second type of training samples, wherein the first type of training samples include a first number of facial key points, and the second type of training samples include a second number of facial key points.
[0025] Solution 1. A method for detecting key points of a face, characterized in that the method comprises:
[0026] Get face image;
[0027] Input the face image into the trained detection model to obtain the face key point detection result,
[0028] The detection model includes a backbone network for extracting image features, a key point position detection head connected to the backbone network, and an auxiliary information detection head connected to the backbone network.
[0029] Solution 2. The method for detecting facial key points according to Solution 1, characterized in that the method further comprises:
[0030] The detection model is trained, wherein the training includes collaborative training of the backbone network, the key point position detection head, and the auxiliary information detection head, and the loss function used is:
[0031] Loss 总=λ1×Loss1+λ2×Loss2, where Loss1 is the loss function used by the key point position detection head, and λ1 is its corresponding weight; Loss2 is the loss function used by the auxiliary information detection head, and λ2 is its corresponding weight. 总 is the total loss function of the detection model.
[0032] Solution 3. The facial key point detection method according to Solution 2 is characterized in that the Loss1 is a SmoothL1 loss function.
[0033] Solution 4. The face key point detection method according to Solution 2 is characterized in that the Loss1 is composed of a SmoothL1 loss function and a data enhancement loss function Loss transform composition:
[0034] Loss1=w1×Smooth L1+w2×Loss transform ,
[0035] Among them, the Loss transform Represents the difference between the first prediction result and the second prediction result, wherein the first prediction result is the key point position prediction result A obtained by inputting the original face image sample into the detection model after data enhancement and then inversely processing the prediction result A with data enhancement; the second prediction result is the key point position prediction result B obtained by inputting the original face image sample into the detection model, and wherein w1 and w2 are weights.
[0036] Solution 5. The facial key point detection method according to Solution 2 is characterized in that the auxiliary information detection head includes: a key point visibility detection head and / or a head posture detection head.
[0037] Solution 6. The method for detecting key points of a face according to Solution 5 is characterized in that:
[0038] When the auxiliary information detection head is a key point visibility detection head, the Loss2 is a cross entropy loss function.
[0039] Solution 7. The method for detecting key points of a face according to Solution 5 is characterized in that:
[0040] When the auxiliary information detection head is a head posture detection head, the Loss2 is a Smooth L1 loss function or Loss2=w3×Smooth L1 loss function+w4×Ranking loss function, where w3 and w4 are weights.
[0041] Solution 8. The method for detecting key points of a face according to Solution 5 is characterized in that:
[0042] When the auxiliary information detection head consists of a key point visibility detection head and a head posture detection head, the Loss2=w5×cross entropy loss function+w6×Smooth L1 loss function; or the Loss2=w5×cross entropy loss function+w3×Smooth L1 loss function+w4×Ranking loss function, where w3-w6 are weights.
[0043] Solution 9. A facial key point detection method according to any one of Solutions 2-8, characterized in that the facial image training samples include facial images in two modalities: visible light images and infrared images.
[0044] Scheme 10. A facial key point detection method according to any one of Schemes 2-8, characterized in that the facial image training samples include a first category of training samples and a second category of training samples, wherein the first category of training samples includes a first number of facial key points, and the second category of training samples includes a second number of facial key points.
[0045] Solution 11. A facial key point detection method according to any one of Solutions 2-8, characterized in that the training is divided into multiple batches, and training samples in different batches are enhanced using different data enhancement methods.
[0046] Solution 12. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are suitable for being loaded and run by a processor to execute the facial key point detection method described in any one of Solutions 1 to 11.
[0047] Solution 13. A smart device, characterized in that the smart device comprises:
[0048] at least one processor;
[0049] and, a memory communicatively coupled to the at least one processor;
[0050] The memory stores a computer program, and when the computer program is executed by the at least one processor, the facial key point detection method described in any one of Schemes 1 to 11 is implemented.
[0051] The above one or more technical solutions of this application have at least one or more of the following Beneficial effects:
[0052] In the technical solution for implementing the present application, the detection model includes a backbone network for extracting image features, a key point position detection head connected to the backbone network, and an auxiliary information detection head connected to the backbone network. The trained detection model is used to detect facial key points, which solves the problem that facial key point detection fails to be effectively integrated and the detection accuracy is poor due to the use of other features.
[0053] In the technical solution of the present application, the purpose of balancing the degree of optimization between different detection tasks can be achieved by setting the weight of the detection head (network head). Since the optimization of some tasks helps to improve other tasks to a certain extent, a positive training environment is created to improve the overall performance of multiple tasks. In addition, setting corresponding weights for each network head loss function achieves the purpose of reducing interference between multiple tasks while making full use of the correlation between tasks, thereby improving the overall prediction performance and stability of the model.
[0054] In the technical solution for implementing the present application, multiple network heads share the common features extracted from face images by the backbone network (basic network), and use different loss functions to measure the error between the predicted results of each task and the actual results. The weighted sum of multiple loss functions is used as the loss function of the overall detection model, thereby achieving the purpose of sharing information and collaborating between multiple tasks, and further improving the overall performance of the model.
[0055] In the technical solution for implementing the present application, the detection model is trained using facial images in two modalities, namely visible light and infrared images, thereby achieving the purpose of improving the versatility of the model and providing an effective solution to the problem of multi-modal facial images in facial key point detection in a vehicle environment.
[0056] In the technical solution for implementing the present application, by setting the sampling ratio of facial images of two modalities and / or two numbers of key points, the purpose of improving the model performance by adjusting the training data (facial image training samples) is achieved, thereby achieving the effect of further improving the accuracy of facial key point detection.
[0057] In implementing the technical solution of the present application, by performing data enhancement processing on facial images, the effect of expanding training data and improving the robustness of the detection model is achieved, thereby achieving the purpose of improving the reliability and accuracy of facial key point detection.
[0058] In implementing the technical solution of the present application, the prediction results of face data on two different paths before and after data enhancement are compared, which is equivalent to introducing an additional self-supervisory signal to guide the training of the detection model, thereby achieving the effect of enhancing the robustness and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0060] Figure 1 It is a schematic flow chart of the main steps of a method for detecting key points of a face according to an embodiment of the present application;
[0061] Figure 2 is a schematic diagram of a detection model constructed by a basic network and multiple network heads according to an embodiment of the present application;
[0062] Figure 3 It is a schematic diagram of the overall process of a method for detecting key points of a face according to an embodiment of the present application;
[0063] Figure 4 It is a schematic diagram of the main structural block diagram of a facial key point detection device according to an embodiment of the present application;
[0064] Figure 5 The figure is a schematic diagram of the connection relationship between the processor and the memory of a smart device according to an embodiment of the present application. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.
[0066] It should be noted that, in the description of the present application, the terms "first", "second" etc. are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0067] The terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, an indirect connection through an intermediate medium, or the internal connection of two components, a wireless connection, or a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, if the meaning of "and / or" appears in this application, it includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0068] Definition of terms:
[0069] An automated driving system (ADS) refers to a system that will continuously perform all dynamic driving tasks (DDT) within its own operational domain design (ODD). That is, the machine system is allowed to fully assume the task of automatic vehicle control only under the specified appropriate driving scenario conditions, that is, the system is activated when the vehicle meets the ODD conditions, and the system replaces the human driver as the driving subject of the vehicle. Among them, the dynamic driving task DDT refers to the continuous lateral (left and right steering) and longitudinal motion control (acceleration, deceleration, constant speed) of the vehicle, as well as the detection and response to targets and events in the vehicle's driving environment. The design operating range ODD refers to the conditions under which the automated driving system can operate safely, and the conditions set may include geographic location, road type, speed range, weather, time, national and local traffic laws and regulations, etc.
[0070] The real in-cabin environment poses a severe challenge to facial key point detection technology, because various factors such as in-cabin lighting, facial posture, occlusions, seat position, facial expression, age and gender will affect the accuracy of key point detection. In addition, there are multimodal face images in the vehicle environment, such as visible light (RGB) images and infrared (IR) images. Conventional facial key point detection methods have the following problems: the training data is relatively simple, and there are fewer training samples for difficult scenes; the data enhancement methods are relatively simple, and various methods are not fully utilized for data expansion; the training strategy is relatively simple, and the auxiliary relationship between each task is not fully utilized. Therefore, conventional facial key point detection methods are difficult to cope with the complex and changeable vehicle environment and cannot fully solve the special problems of the vehicle environment.
[0071] In the embodiment of the present application, a method for detecting key points of a face is provided. Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of a method for detecting key points of a face according to an embodiment of the present application. Figure 1 As shown, the embodiment of the present application mainly includes the following steps S101 and S102:
[0072] Step S101, obtaining a face image.
[0073] In this embodiment, the face image is a collected image that needs to perform face key point detection, such as face data in a vehicle cabin.
[0074] Step S102, inputting the face image into the trained detection model to obtain the face key point detection result, wherein the detection model includes a backbone network for extracting image features, a key point position detection head connected to the backbone network, and an auxiliary information detection head connected to the backbone network.
[0075] It is understandable that facial key point detection is mainly used to detect the key point positions of important parts such as eyes, nose, mouth, etc. from facial images, but these key points are affected by various factors such as head posture, lighting, expression, etc., and it is difficult to accurately complete facial key point detection by only considering the key point positions. Therefore, the detection model in this embodiment uses auxiliary information (auxiliary features) in addition to the facial key point positions. Auxiliary features represent other features that can assist the facial key point detection task and are closely related to the key point positions, such as key point visibility, head posture (indicating the relative position of the head shape and facial key points), facial expressions, etc. It should be noted that this embodiment does not impose specific restrictions on the selection of auxiliary features.
[0076] Building multiple detection heads based on the positions of key points of the face and auxiliary features is also called a network head. Among them, the network head can be a linear layer in a neural network. Each network head can handle a detection (prediction) task. For example, a network head built based on the positions of key points of the face can handle the task of detecting the positions of key points of the face, that is, predicting the positions of key points; a network head built based on the visibility of key points can handle the task of detecting the visibility of key points, that is, predicting the visibility of key points. It can be understood that due to different detection tasks, the number of parameters of each linear layer is also different. Exemplarily, assuming that key point position detection is performed, the corresponding linear layer requires 106*2 parameters to represent the positions of 106 key points; key point visibility requires 106 parameters to indicate whether the key point is visible, assuming that 0 represents invisible and 1 represents visible; head posture requires 3 parameters to represent the pitch angle, yaw angle and roll angle respectively.
[0077] In this embodiment, the backbone network (Backbone) is also called the basic network, which is used to extract common features from face data (original face images), such as face shape, color, texture, etc. The backbone network can use a convolutional neural network, such as SlimNet, ResNet (Residual Network), VGG (Visual Geometry Group) and other deep convolutional neural networks.
[0078] In one embodiment, see the attached Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a detection model constructed by a basic network and multiple network heads according to an embodiment of the present application. Figure 2 As shown in the figure, the input is the original face image. The basic network extracts common features from the original face image and passes them to multiple network heads (network head 1...network head n) respectively. Each network head further learns complex features based on the corresponding detection task and common features, and finally outputs the prediction results of the corresponding detection task (output 1...output n).
[0079] In one implementation, multiple linear layers are connected after the fully connected layer of the backbone network SlimNet as detection heads, and the linear layer is, for example, GEMM (General Matrix Multiplication).
[0080] In one embodiment, a neck network may be included between the base network and the multiple network heads, so that the general features (feature maps) can be better matched with the network heads by adjusting the size and number of channels. Exemplarily, the neck network may use a convolutional neural network and a pooling neural network.
[0081] In this embodiment, the facial key point detection result is obtained by inputting the acquired facial image into the above-mentioned trained detection model, thereby solving the problem that the existing facial key point detection fails to effectively integrate and utilize other features, resulting in poor detection accuracy.
[0082] In one implementation of the present application, training the detection model includes collaboratively training the backbone network, the key point position detection head, and the auxiliary information detection head, and the loss function used is: Loss 总 =λ1×Loss1+λ2×Loss2, where Loss1 is the loss function used by the key point position detection head, and λ1 is its corresponding weight; Loss2 is the loss function used by the auxiliary information detection head, and λ2 is its corresponding weight. Loss is the total loss function of the detection model.
[0083] In this embodiment, since the position of the key points of the face is a continuous value, the loss function Loss1 used by the key point position detection head can select a regression loss function, such as the Smooth L1 loss function. Using the smooth L1 loss to optimize the position of the key points of the face can achieve the effect of stabilizing the gradient value of the detection model, smoothing the training, and not being easily affected by outliers.
[0084] For the key point visibility detection head, since the key point visibility (visible / invisible) is a discrete value, the loss function Loss2 used by the auxiliary information detection head can select a classification loss function such as a cross entropy loss function.
[0085] For the head posture detection head, the loss function Loss2 used by the auxiliary information detection head can be the Smooth L1 loss function. Considering that there is a certain distance relationship between various head postures, Loss2 can also be a combination of regression loss function and ranking loss function. For example: Loss2 = w3 × Smooth L1 loss function + w4 × Ranking loss function, w3 and w4 are the weights corresponding to the loss function. The ranking loss function can be used to measure the distance difference between various head postures, so as to improve the prediction ability of the model.
[0086] In one embodiment, when the auxiliary information detection head is composed of a key point visibility detection head and a head posture detection head, the loss function used by the auxiliary information detection head is Loss2 = w5 × cross entropy loss function + w6 × Smooth L1 loss function, or Loss2 = w5 × cross entropy loss function + w3 × Smooth L1 loss function + w4 × Ranking loss function, where w3-w6 are weights.
[0087] This implementation achieves the purpose of collaborative training of multiple detection tasks through the combination of multiple detection head loss functions, and uses the trained detection model for face key point detection, solving the problem of poor detection accuracy caused by the failure to effectively integrate face key point detection and the use of other features. In addition, multiple network heads share the common features extracted from face images by the basic network, use different loss functions to measure the error between the predicted results and the actual results of each task, and use the weighted sum of the loss functions of multiple network heads as the loss function of the overall detection model, achieving the purpose of sharing information and cooperating between multiple tasks, thereby achieving the effect of improving the overall performance of the model.
[0088] It should be noted that the loss function corresponding to any network head can be a classification loss function, a regression loss function, a ranking loss function, a mixed loss function, etc., which needs to be selected according to the detection task processed by the network head. The loss functions corresponding to multiple network heads can be used as the loss function of the entire detection model to guide model training by summing, weighted summing, etc. In this embodiment, corresponding weights are set for the loss functions of multiple network heads, and the optimization degree between different detection tasks is balanced by weights, thereby achieving the purpose of improving the flexibility of detection model training.
[0089] In one implementation, when setting corresponding weights for multiple network head loss functions, multiple groups may be pre-set, and then a group of weights with the best detection model performance is used as the weights of the multiple network head loss functions.
[0090] In one implementation, the loss function of the detection model as a whole is used to guide model training, and the parameters of the detection model as a whole (including the basic network and multiple network heads) can be adjusted by back propagation and other methods.
[0091] In one embodiment, in order to improve the accuracy of facial key point detection, in-cabin facial data in various scenarios can be collected, such as facial images with various head postures, facial images under various in-cabin lighting conditions, facial images under various facial expressions, facial images of people of various ages and genders in the cabin, etc., so that the facial training data can cover various scenarios, thereby achieving the purpose of improving the robustness of the detection model.
[0092] In one implementation, the collected various types of face data may be cleaned, labeled, expanded, enhanced, and the like to improve the quality of face training data, thereby achieving the effect of improving the accuracy of the detection model.
[0093] In one embodiment, data enhancement processing includes, but is not limited to: external expansion and black filling, random offset, random occlusion, random flip, random rotation, random blur, color jitter, etc. Among them, external expansion and black filling is to fill the face at the edge of the image with black blocks. In particular, in order to solve the problem that the color of the occluder is similar to that of the face, the occlusion of random color blocks is also added, and the random color is set within a fixed range that deviates from the color of the face. By performing data enhancement processing on the training samples, not only can the sample data of the face image be expanded, but also the purpose of simulating and responding to various actual scenes can be achieved.
[0094] In a technical solution of the above-mentioned face key point detection method, Loss1 is composed of Smooth L1 loss function and data enhancement loss function Loss transform Composition: Loss1 = w1 × Smooth L1 + w2 × Loss transform , where Loss transform It represents the difference between the first prediction result and the second prediction result, wherein the first prediction result is the key point position prediction result A obtained by inputting the original face image sample into the detection model after data enhancement and inverse processing the prediction result A; the second prediction result is the key point position prediction result B obtained by inputting the original face image sample into the detection model, and wherein w1 and w2 are weights.
[0095] In this embodiment, the difference between the first prediction result and the second prediction result can be calculated using Smooth L1 or mean-square error (MSE). transform Two examples are as follows:
[0096] Example 1: Assume that the size of the original image x1 is 128*128 (px). After a series of specific data enhancement processing, such as randomly cropping x1 to 100*100, rotating it 30 degrees clockwise, and enlarging it to 128*128, image x2 is obtained. Image x2 is input into the detection model to obtain the corresponding key point position prediction result A. Image x1 is input into the detection model to obtain the corresponding key point position prediction result B, which is the second prediction result. Perform the inverse processing of data enhancement on A to obtain the first prediction result, that is, A is regarded as the prediction point on image x2, and then the image x2 is successively reduced to 100*100, rotated 30 degrees counterclockwise, and restored to 128*128 to obtain C. Then calculate the Smooth L1 between C and B as the Loss transform .
[0097] Example 2: Assuming that the size of the original image x3 is 128*128, randomly crop x3 to 110*110, rotate 5 degrees clockwise, flip horizontally, and enlarge it to 128*128 to obtain image x4. Input image x4 into the detection model to obtain the corresponding key point position prediction result A, and input image x3 into the detection model to obtain the corresponding key point position prediction result B. Perform the inverse data enhancement on A, that is, regard A as the prediction point on image x4, and follow the x4 image to shrink to 110*110, flip horizontally, rotate 5 degrees counterclockwise, and restore to 128*128 to obtain C. Then calculate the MSE between C and B as the Loss transform .
[0098] In this embodiment, the introduction of transform loss has the following beneficial effects:
[0099] Enhanced self-supervision signal: Comparing the prediction results on two different paths before and after data enhancement is equivalent to introducing an additional self-supervision signal. The detection model can be forced to learn the changes in the key point positions under different data enhancement conditions, thereby enhancing the robustness and generalization ability of the model.
[0100] Improving the stability of key point prediction: using Loss transform When training the detection model, the model needs to maintain a certain degree of consistency in the prediction of key points. This can make the model predict key points more stably and avoid inconsistent or overly extreme predictions.
[0101] Refined key point prediction: The prediction results of face data on two different paths before and after data augmentation are compared, which makes the model pay more attention to the subtle changes and fine features of key points.
[0102] Improve the practical value and robustness of the model: By considering the prediction results under different paths, the model can better adapt to changes in the input image during the prediction process, thereby improving the robustness of the model in practical applications.
[0103] In a technical solution of the above-mentioned facial key point detection method, the facial image training samples include facial images in two modalities: visible light images and infrared images.
[0104] In this embodiment, in order to solve the problem of multi-modal facial images in the vehicle environment, this embodiment uses facial images of at least two modes to train the detection model. Specifically, the two modes include but are not limited to: visible light images and infrared images. Among them, RGB images and visible light images can be obtained by visible light imaging sensors and infrared imaging sensors, respectively. Since visible light is significantly reduced in a dark environment, the visible light image will lose useful information. Therefore, an infrared image can be collected in a dark environment. This embodiment uses facial images of at least two modes mentioned above to train the detection model, thereby achieving the purpose of improving the versatility of the detection model and providing an effective solution to the problem of multi-modal facial images in the detection of facial key points in a vehicle environment.
[0105] In one implementation, the (face image) training samples are obtained by preprocessing (such as cleaning and labeling) the collected face images, such as cleaning and removing noise images and irrelevant images in the collected face images, and labeling the face images accordingly.
[0106] In one embodiment, when the auxiliary features select key point visibility and head posture, it is necessary to mark the key point visibility and head posture for the face image. Specifically, marking the key point visibility means marking each key point in the face image as visible / invisible, and marking the head posture means marking the head posture angle for the face image. The head (face) posture angle can be obtained according to the key point position using the solvePnP algorithm.
[0107] In one implementation, the training samples may include both facial images that have not been processed with data enhancement and facial images that have been processed with data enhancement.
[0108] In a technical solution of the above-mentioned facial key point detection method, the facial image training samples include a first type of training samples and a second type of training samples, wherein the first type of training samples include a first number of facial key points, and the second type of training samples include a second number of facial key points.
[0109] In this embodiment, the first number and the second number are, for example, 106 points and 8 points. It should be noted that this embodiment does not impose a specific restriction on the number of facial key points in the training sample, and it can also be 5 points, 68 points, 81 points, etc.
[0110] In a technical solution of the above-mentioned facial key point detection method, the training is divided into multiple batches, and the training samples of different batches are enhanced by different data enhancement methods.
[0111] In this embodiment, different batches are processed using a series of different data enhancement methods. For example, 1000 training samples are divided into 10 batches (batch 1-batch 10), and 100 (face image) samples are used for training each time. Different images in different batches can be processed using a series of different data enhancement methods.
[0112] In one implementation, sampling ratios may be set for face images with different numbers of points and different modalities in each training batch of the detection model.
[0113] In this embodiment, different points and sampling ratios of facial images of different modalities may be set for different training batches according to specific requirements.
[0114] In one implementation, the sampling ratios may be set in multiple groups and adjusted based on the performance of the detection model after training.
[0115] It should be noted that the multi-task collaborative training of this application utilizes the interactive relationship between the position of facial key points, key point visibility and head posture prediction to improve the learning effect and stability of the model. This method is particularly suitable for processing face images. While solving the prediction of facial key points, key point visibility and head posture prediction, it can significantly improve the overall stability of the model.
[0116] This application uses the auxiliary relationship between tasks to implement multi-task collaborative training, which is mainly achieved through two parts: model architecture and training strategy. In the model architecture, a basic network is used to extract common features from the original face image for use by all tasks. Based on this design, key information and features can be shared and transmitted between different tasks, thereby realizing auxiliary relationships and collaborative learning between tasks. In addition, by establishing an independent network head for each task, predictions for specific tasks are performed. The purpose of ensuring that the network can effectively learn specific tasks while utilizing the auxiliary relationship between tasks is achieved.
[0117] In terms of training strategy, we first assign weights to the loss functions of each task based on the importance and actual effect of each task, and then add up the losses of all tasks to get the final objective function (the loss function of the detection model). During the training process, we optimize according to this objective function, achieving the goal of ensuring the quality of a single task while reflecting the synergy between tasks.
[0118] Based on the above model design and training strategy, we can effectively utilize the auxiliary relationship between tasks and realize multi-task collaborative training, thereby improving the overall prediction performance and stability of the model. Collaborative training makes it so that multiple tasks are no longer isolated, but collaborate with each other and share information, which in turn improves the performance of the entire model and achieves the effect of improving the accuracy of facial key point detection.
[0119] Specifically, multi-task collaborative (joint) training has the following beneficial effects:
[0120] Information sharing: Through multi-task joint training, information and feature sharing between multiple tasks can be achieved. Compared with independently trained single-task models, multi-task models can better explore and utilize the correlation between different tasks, thereby enhancing the generalization ability of the model and improving the stability of the model.
[0121] Regularization effect: The process of multiple tasks sharing common features is equivalent to introducing an implicit regularization effect. Specifically, since the model needs to adapt to multiple tasks at the same time, the model will not be overly dependent on the features of a specific task during training. This implicit regularization effect can prevent the detection model from overfitting, improve the stability of the model, and thus improve the reliability of facial key point detection.
[0122] Weight adjustment: During multi-task training, the importance and degree of optimization of each task can be balanced by adjusting the loss weight of the task. This weight adjustment can prevent the model from being overly dependent on a specific task, thereby improving the robustness and stability of the detection model.
[0123] Optionally, Figure 3 FIG. 1 is a schematic diagram of the overall process of a method for detecting key points of a face according to an embodiment of the present application. Figure 3 As shown, the method mainly includes the following steps S301 to S307:
[0124] Step S301, labeling face data. Optionally, the face data in the cabin is fully labeled and three types of labels are marked (such as the position of key points of the face, the visibility of key points, and the head posture).
[0125] Step S302: sampling and data preprocessing according to proportion. Optionally, the sampling proportion of each type of picture is set according to actual needs.
[0126] Step S303, data enhancement of occlusion, angle, color, etc. Optionally, data enhancement processing is performed on various types of face data according to settings.
[0127] Step S304, feature extraction. Optionally, the enhanced face data is sent to a detection model (basic network) to extract common features and perform feature learning.
[0128] Step S305, key point positions, key point visibility and head posture are obtained through three network heads. Optionally, the common features are passed to the key point positions, key point visibility and head posture network heads.
[0129] Step S306, add transform loss (Loss transform ). In this embodiment, by introducing transform loss and comparing the key point position prediction results of the original face image and the face image after data enhancement, the purpose of improving the accuracy and robustness of key point position prediction in scenes such as occlusion, illumination and large posture is achieved; the problem that image occlusion, illumination change and rotation angle affect key point detection is solved; the purpose of capturing facial detail features is achieved; and the key point detection (prediction) stability of the model in processing various image transformations is ensured, thereby improving the accuracy of key point position prediction.
[0130] Step S307, multi-task collaborative training. Through multi-task collaborative training of key point positions, visible attributes and head posture, the auxiliary relationship between tasks is utilized to achieve the effect of improving the stability, accuracy and robustness of the detection model. For details of the above steps, please refer to other embodiments of this application, which will not be repeated here.
[0131] This embodiment achieves the purpose of better coping with complex and changeable vehicle environments based on the comprehensive prediction of the location, visibility and head posture of facial key points. Through data annotation, data enhancement and unique training strategies, the accuracy and stability of facial key point detection are effectively improved.
[0132] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present application.
[0133] Furthermore, the present application also provides a facial key point detection device, such as Figure 4 As shown, it includes: an acquisition module 401, used to acquire a face image; a detection module 402, used to input the face image into a trained detection model to obtain a face key point detection result, wherein the detection model includes a backbone network for extracting image features, a key point position detection head connected to the backbone network, and an auxiliary information detection head connected to the backbone network.
[0134] In a technical solution of the above-mentioned face key point detection device, the device also includes: a training module for training the detection model, wherein the training includes collaborative training of the backbone network, the key point position detection head and the auxiliary information detection head, and the loss function used is: Loss 总=λ1×Loss1+λ2×Loss2, where Loss1 is the loss function used by the key point position detection head, and λ1 is its corresponding weight; Loss2 is the loss function used by the auxiliary information detection head, and λ2 is its corresponding weight. 总 is the total loss function of the detection model.
[0135] In a technical solution of the above-mentioned facial key point detection device, Loss1 is a Smooth L1 loss function.
[0136] In a technical solution of the above-mentioned face key point detection device, Loss1 is composed of Smooth L1 loss function and data enhancement loss function Loss transform Composition: Loss1 = w1 × Smooth L1 + w2 × Loss transform , where Loss transform It represents the difference between the first prediction result and the second prediction result, wherein the first prediction result is the key point position prediction result A obtained by inputting the original face image sample into the detection model after data enhancement and inverse processing the prediction result A; the second prediction result is the key point position prediction result B obtained by inputting the original face image sample into the detection model, and wherein w1 and w2 are weights.
[0137] In a technical solution of the above-mentioned facial key point detection device, the auxiliary information detection head includes: a key point visibility detection head and / or a head posture detection head.
[0138] In a technical solution of the above-mentioned facial key point detection device, when the auxiliary information detection head is a key point visibility detection head, Loss2 is a cross entropy loss function.
[0139] In a technical solution of the above-mentioned facial key point detection device, when the auxiliary information detection head is a head posture detection head, Loss2 is a Smooth L1 loss function or Loss2=w3×Smooth L1 loss function+w4×Ranking loss function, where w3 and w4 are weights.
[0140] In a technical solution of the above-mentioned facial key point detection device, when the auxiliary information detection head is composed of a key point visibility detection head and a head posture detection head, Loss2 = w5 × cross entropy loss function + w6 × Smooth L1 loss function; or Loss2 = w5 × cross entropy loss function + w3 × Smooth L1 loss function + w4 × Ranking loss function, where w3-w6 are weights.
[0141] In a technical solution of the above-mentioned facial key point detection device, the facial image training samples include facial images in two modalities: visible light images and infrared images.
[0142] In a technical solution of the above-mentioned facial key point detection device, the facial image training samples include a first type of training samples and a second type of training samples, wherein the first type of training samples include a first number of facial key points, and the second type of training samples include a second number of facial key points.
[0143] In a technical solution of the above-mentioned facial key point detection device, training is divided into multiple batches, and training samples in different batches are enhanced using different data enhancement methods.
[0144] It is understood by those skilled in the art that all or part of the processes in the method for implementing any of the above embodiments of the present application may also be completed by hardware related to computer program instructions, and the computer program may be stored in a computer-readable storage medium, and the computer program, when executed by a processor, may implement the steps of each of the above method embodiments. Among them, the computer program includes computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. Computer-readable storage media may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc., which can carry computer program code.
[0145] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium can be configured to store a program for executing the face key point detection method in the above method embodiment, and the program can be loaded and run by the processor to implement the above face key point detection method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.
[0146] Furthermore, the present application also provides an intelligent device, which may include at least one processor; and a memory connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method for detecting key points of a face described in any of the above embodiments is implemented. The intelligent device of the present application may include a driving device, a smart car, a robot, and other devices. Figure 5 , Figure 5FIG. 1 is a schematic diagram of the connection relationship between the processor and the memory of a smart device according to an embodiment of the present application. Figure 5 As shown, Figure 5 The memory and the processor are exemplarily shown to be communicatively connected via a bus.
[0147] In some embodiments of the present application, the smart device further includes at least one sensor, which is used to sense information. The sensor is communicatively connected to any type of processor mentioned in the present application. Optionally, the smart device further includes an autonomous driving system, which is used to guide the smart device to drive itself or assist in driving. The processor communicates with the sensor and / or the autonomous driving system to complete the facial key point detection method described in any of the above embodiments.
[0148] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present application, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.
[0149] It is understood by those skilled in the art that each module in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present application, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present application.
[0150] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.
[0151] The user personal information processed by this application will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service. It may involve the user's biometric information, account information, device information, driving information, vehicle information or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.
[0152] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.
[0153] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.
Claims
1. A method for detecting key points of a face, characterized in that: The method comprises: Get face image; The face image is input into a trained detection model to obtain a face key point detection result, wherein the detection model includes a backbone network for extracting image features, a key point position detection head connected to the backbone network, and an auxiliary information detection head connected to the backbone network.
2. The method for detecting facial key points according to claim 1, characterized in that: The method further comprises: The detection model is trained, wherein the training includes collaborative training of the backbone network, the key point position detection head, and the auxiliary information detection head, and the loss function used is: Loss 总 =λ1×Loss1+λ2×Loss2, where Loss1 is the loss function used by the key point position detection head, and λ1 is its corresponding weight; Loss2 is the loss function used by the auxiliary information detection head, and λ2 is its corresponding weight. 总 is the total loss function of the detection model.
3. The method for detecting facial key points according to claim 2, wherein: The Loss1 is the Smooth L1 loss function.
4. The method for detecting facial key points according to claim 2, wherein: The Loss1 consists of the Smooth L1 loss function and the data enhancement loss function Loss transform composition: Loss1=w1×Smooth L1+w2×Loss transform , Among them, the Loss transform Represents the difference between the first prediction result and the second prediction result, wherein the first prediction result is the key point position prediction result A obtained by inputting the original face image sample into the detection model after data enhancement and then inversely processing the prediction result A with data enhancement; the second prediction result is the key point position prediction result B obtained by inputting the original face image sample into the detection model, and wherein w1 and w2 are weights.
5. The method for detecting facial key points according to claim 2, characterized in that: The auxiliary information detection head includes: a key point visibility detection head and / or a head posture detection head.
6. The method for detecting key points of a face according to claim 5, characterized in that: When the auxiliary information detection head is a key point visibility detection head, the Loss2 is a cross entropy loss function.
7. The method for detecting facial key points according to claim 5, characterized in that: When the auxiliary information detection head is a head posture detection head, the Loss2 is a Smooth L1 loss function or Loss2=w3×Smooth L1 loss function+w4×Ranking loss function, where w3 and w4 are weights.
8. The method for detecting facial key points according to claim 5, characterized in that: When the auxiliary information detection head consists of a key point visibility detection head and a head posture detection head, the Loss2=w5×cross entropy loss function+w6×Smooth L1 loss function; or the Loss2=w5×cross entropy loss function+w3×Smooth L1 loss function+w4×Ranking loss function, where w3-w6 are weights.
9. The method for detecting facial key points according to any one of claims 2 to 8, characterized in that: The face image training samples include face images in two modalities: visible light image and infrared image.
10. The method for detecting facial key points according to any one of claims 2 to 8, characterized in that: The face image training samples include a first type of training samples and a second type of training samples, wherein the first type of training samples include a first number of face key points, and the second type of training samples include a second number of face key points.