Key point detection model training method, device, electronic device and storage medium

Abstract

The present invention discloses a key point detection model training method, device, electronic device, and storage medium, which can be applied to the field of mapping. The method includes: obtaining a target facial detection area in a training sample image, performing a first-scale enhancement process on the target facial detection area to obtain a first facial enhancement image; determining a facial reference area corresponding to the target facial detection area based on key point annotation information corresponding to the target facial detection area in the training sample image, performing a second-scale enhancement process on the facial reference area to obtain a second facial enhancement image; training a preset neural network model for key point detection based on the first and second facial enhancement images to obtain a key point detection model; and using the key point detection model to detect facial key points in facial images. The present invention improves the robustness of the trained key point detection model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a key point detection model training method, device, electronic equipment and storage medium. Background Art

[0002] Facial key point detection is a key step in the field of facial recognition and analysis (such as expression analysis, posture judgment, quality analysis, 3D facial reconstruction, 3D animation and other application scenarios). In some application scenarios of facial key point detection (such as facial recognition systems), the robustness of facial key point detection affects the overall accuracy of subsequent facial recognition.

[0003] In related technologies, in order to improve the robustness of facial key point detection, one way is to design the key point detection model structure and use a multi-layer cascade network for model training; another way is to use multi-task training in key point detection model training, and combine some tasks related to facial key point detection (such as facial posture estimation tasks, facial detection tasks, attribute estimation tasks, etc.) for joint training.

[0004] The first method mentioned above requires repeated training and verification of the specially designed key point detection network, which not only prolongs the training cycle, but also the obtained network cannot adapt to other data sources, resulting in a waste of design resources. In addition, the multiple cascade networks also have the problem that the model parameters are too large to meet the time-consuming requirements in actual scenarios; the second method mentioned above requires different information to be labeled for different tasks for the same training data, which greatly increases the cost of collecting training data. In addition, since the convergence speed during the training process of each task is inconsistent, it is necessary to adjust the weights of the loss constraints of different tasks. In the process of this weight adjustment, different hyperparameters need to be designed for different tasks, which leads to the inability to achieve stable improvement in model training. Summary of the Invention

[0005] In order to solve the problems of the prior art, the embodiments of the present invention provide a key point detection model training method, device, electronic device and storage medium. The technical solution is as follows:

[0006] In one aspect, a key point detection model training method is provided, the method comprising:

[0007] Obtaining a target face detection region in a training sample image, and performing a first scale enhancement process on the target face detection region to obtain a first face enhanced image;

[0008] Determining a facial reference region corresponding to the target facial detection region based on key point annotation information corresponding to the target facial detection region in the training sample image, and performing a second scale enhancement process on the facial reference region to obtain a second facial enhanced image;

[0009] Based on the first facial enhancement picture and the second facial enhancement picture, a preset neural network model is trained for key point detection to obtain a key point detection model; the key point detection model is used to detect facial key points in facial pictures.

[0010] In another aspect, a key point detection model training device is provided, the device comprising:

[0011] A first scale enhancement module is configured to obtain a target facial detection region in a training sample image, perform a first scale enhancement process on the target facial detection region, and obtain a first facial enhanced image;

[0012] A second scale enhancement module is configured to determine a facial reference region corresponding to the target facial detection region based on key point annotation information corresponding to the target facial detection region in the training sample image, and perform a second scale enhancement process on the facial reference region to obtain a second facial enhanced image;

[0013] A model training module is used to perform key point detection training on a preset neural network model based on the first facial enhancement picture and the second facial enhancement picture to obtain a key point detection model; the key point detection model is used to detect facial key points in facial pictures.

[0014] In an exemplary embodiment, the first scale enhancement module includes:

[0015] A first random selection module, configured to randomly select a first scale enhancement value from a first scale enhancement range;

[0016] a first scale enhancement processing submodule, configured to perform a first scale enhancement processing on the target face detection area based on the first scale enhancement value;

[0017] The second scale enhancement module includes:

[0018] A second random selection module, configured to randomly select a second scale enhancement value from a second scale enhancement range;

[0019] The second scale enhancement processing submodule is configured to perform a second scale enhancement processing on the facial reference area based on the second scale enhancement value; wherein the second scale enhancement range is determined based on the first scale enhancement range.

[0020] In an exemplary embodiment, the apparatus further includes a scale enhancement range determination module, wherein the scale enhancement range determination module is configured to:

[0021] Obtaining a preset scale enhancement limit; the preset scale enhancement limit includes a preset scale enhancement upper limit and a preset scale enhancement lower limit;

[0022] Performing scale enhancement processing on the target face detection area based on the preset scale enhancement limit to obtain an enhanced face detection area;

[0023] Determining a sample facial region based on the key point annotation information of the target facial detection region;

[0024] determining a target scale enhancement limit based on the enhanced face detection region, the sample face region, and a preset overlap ratio; wherein the target scale enhancement limit includes a target scale enhancement upper limit and a target scale enhancement lower limit that ensure that the overlap ratio between the enhanced face detection region and the enhanced sample face region reaches the preset overlap ratio; and the enhanced sample face region is a region obtained by performing scale enhancement processing on the sample face region based on the target scale enhancement limit;

[0025] The range indicated by the preset scale enhancement limit is used as the first scale enhancement range, and the range indicated by the target scale enhancement limit is used as the second scale enhancement range.

[0026] In an exemplary embodiment, the second scale enhancement module further includes:

[0027] an affine transformation module, configured to perform an affine transformation of the coordinates of the key point annotation information corresponding to the target facial detection area based on the first facial enhancement image, to obtain target key point annotation information; the target key point annotation information includes coordinate information of each key point in the first facial enhancement image;

[0028] The facial reference area determination module is used to determine the facial reference area corresponding to the target facial detection area based on the target key point annotation information.

[0029] In an exemplary embodiment, the facial reference area determination module includes:

[0030] a coordinate information determining module, configured to determine maximum coordinate information and minimum coordinate information based on the coordinate information of each key point in the first facial enhancement image; the maximum coordinate information includes a maximum horizontal coordinate and a maximum vertical coordinate, and the minimum coordinate information includes a minimum horizontal coordinate and a minimum vertical coordinate;

[0031] The square area determination module is used to determine a target square area based on the maximum coordinate information and the minimum coordinate information; the target square area serves as a facial reference area corresponding to the target facial detection area.

[0032] In an exemplary embodiment, the model training module includes:

[0033] A key point prediction module is configured to input the first enhanced face image and the second enhanced face image as target training samples into a preset neural network model for key point detection to obtain predicted key point information;

[0034] a loss value determination module, configured to determine a loss value based on a difference between the predicted key point information and the target key point annotation information using a preset regression loss function;

[0035] An iterative training module is used to adjust the model parameters of the preset neural network model based on the loss value and continue iterative training until a preset training end condition is met to obtain the key point detection model.

[0036] In an exemplary embodiment, the first scale enhancement module further includes:

[0037] A training sample image acquisition module is used to acquire training sample images from a training sample image set; the training sample images include key point annotation information of the face;

[0038] A face detection module is used to perform face detection on the training sample image to obtain candidate face detection areas;

[0039] The face detection region screening module is configured to use the candidate face detection region as the target face detection region when the candidate face detection region contains all key points corresponding to the corresponding key point annotation information.

[0040] In an exemplary embodiment, the face detection module is specifically configured to input the training sample image into a face detection model for face detection to obtain output face detection frame information; the face detection frame information indicates the candidate face detection area.

[0041] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the key point detection model training method of any of the above aspects.

[0042] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement a key point detection model training method as described in any of the above aspects.

[0043] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the keypoint detection model training method of any of the above aspects.

[0044] The embodiment of the present invention obtains a first facial enhancement image by performing scale enhancement processing on the target facial detection area, and determines the facial reference area corresponding to the target facial detection area based on the key point annotation information corresponding to the target facial detection area in the training sample image, and then performs scale enhancement processing on the facial reference area to obtain a second facial enhancement image. The first facial enhancement image and the second facial enhancement image are combined to train a preset neural network model to obtain a key point detection model. In the above technical solution, there is no need to design a special network structure for the key point detection model. By introducing key point annotation information to estimate the facial area to expand the acquired facial detection area, the facial area estimated by the key point annotation information does not change with the facial detection results, thereby improving the robustness of the trained key point detection model. In addition, since training is only performed for the key point detection task, not only the cost of training data collection is reduced, but also there is no need to introduce weight hyperparameters between different tasks in model training, which can achieve stable improvement of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 1 is a flow chart of a key point detection model training method provided by an embodiment of the present invention;

[0047] Figure 2 This is an example of the process of obtaining the target face detection area provided by an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of another key point detection model training method provided by an embodiment of the present invention;

[0049] Figure 4 This is a structural block diagram of a key point detection model training device provided by an embodiment of the present invention;

[0050] Figure 5 This is a hardware structure block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0054] When using a keypoint detection model for facial keypoint detection, the input to the keypoint detection model must be a facial image. That is, facial detection must be performed on the original input image, and the detected facial region is used as the input to the keypoint detection model. Therefore, the accuracy of the keypoint detection model depends on the accuracy of facial detection. Furthermore, because the keypoint detection model requires accurate regression of keypoint locations, it is sensitive to the scale of the facial image (i.e., image size). When the scale of the test face is inconsistent with the scale of the training image, the regression deviation of the keypoint detection model is large. This requires that the keypoint detection model must be compatible with the facial detection algorithm. The scale stability of the facial detection region directly affects the output accuracy of the keypoint detection model. However, in practical applications, the scale of the facial region output by different facial detection algorithms fluctuates. This requires the keypoint detection model to be highly robust to ensure good keypoint detection accuracy even when the scale of the input facial region fluctuates.

[0055] In the related art, one method for improving the robustness of the key point detection model requires designing the key point detection model as a special multi-layer cascade network structure, which not only prolongs the model training cycle and the trained network cannot adapt to other data sources, resulting in a waste of design resources, but also the multiple cascade networks have model parameters that are too large to meet the time-consuming requirements in actual scenarios; another method requires different information to be labeled for different training tasks for the same training data, which greatly increases the cost of collecting training data, and the need to design different hyperparameters for different tasks makes it impossible to achieve stable improvement in model training.

[0056] In response to the above-mentioned problems existing in related technologies in improving the robustness of key point detection models, an embodiment of the present invention provides a key point detection model training method. The training process only involves key point detection tasks. The facial area is estimated by using the key point annotation information corresponding to the facial detection area in the key point detection task, and then the estimated facial area is scale-enhanced. The preset neural network model is then trained for key point detection based on the estimated facial area after scale enhancement and the scale-enhanced facial detection area. Since the facial area obtained based on the key point annotation information does not change with the facial detection algorithm, the stability of the facial area in the training data is ensured, thereby improving the robustness of the key point detection model.

[0057] It can be seen that the training method of the embodiment of the present invention improves the robustness of the key point detection model. Since it only trains for the key point detection task, compared with the related technology that adopts multi-task joint training, the embodiment of the present invention only needs to label the facial key points of the training data, which greatly reduces the cost of training data collection, and there is no need to introduce weight hyperparameters between different tasks in model training, which can achieve stable improvement of model training; compared with the related technology that adopts a multi-level special network structure design for the key point detection model, the embodiment of the present invention can be trained based on a general neural network model, which shortens the model training cycle, improves the adaptability of the model and can meet the time-consuming requirements in actual scenarios.

[0058] It should be noted that the key point detection model training method of the embodiment of the present invention can be applied to a key point detection model training device, and the key point detection model training device can be configured in an electronic device, which can be a terminal or a server.

[0059] Terminals include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, in-vehicle terminals, aircraft, etc. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0060] In one exemplary embodiment, each electronic device can be a node device in a blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, enabling information sharing among multiple node devices. Multiple node devices in a blockchain system can be configured with the same blockchain, which consists of multiple blocks, with adjacent blocks having an associated relationship. This allows any tampering of data in any block to be detected by the next block, thereby preventing tampering of the blockchain data and ensuring the security and reliability of the blockchain data.

[0061] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0062] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0063] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0064] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0065] The following is a detailed introduction to the key point detection model training method of an embodiment of the present invention.

[0066] See also Figure 1 , which shows a flow chart of a key point detection model training method provided by an embodiment of the present invention. It should be noted that this specification provides method operation steps as described in the embodiment or flow chart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many steps, and does not represent the only execution order. When the system or product is actually executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 1 As shown, the method may include:

[0067] S101 , obtaining a target face detection region in a training sample image, performing a first scale enhancement process on the target face detection region, and obtaining a first face enhanced image.

[0068] The training sample image can be any image in the training sample image set, and the training sample image contains a face, which can be a human face or an animal face. Keypoint annotation information is associated with each face in the training sample image. The keypoint annotation information indicates the location of keypoints in the corresponding face in the training sample image, such as the horizontal and vertical coordinates of each keypoint.

[0069] The target facial detection region in a training sample image is determined by performing facial detection on the training sample image using a facial detection algorithm. For example, the training sample image can be input into a facial detection model for facial detection, resulting in facial detection frame information output by the model. This facial detection frame information indicates the facial detection region in the training sample image. For example, a face detection model can be used to input a training sample image into the model for facial detection, resulting in a facial detection frame output by the model.

[0070] Considering that in actual applications, the accuracy of the face detection algorithm cannot guarantee that all face detections are accurate, and some facial regions are too small to have a positive effect on the subsequent training of the key point detection model, in an exemplary embodiment, the above step S101 may include the following steps when obtaining the target face detection region in the training sample image:

[0071] Obtaining a training sample image from a training sample image set; the training sample image includes key point annotation information of the face;

[0072] Performing face detection on the training sample image to obtain candidate face detection areas;

[0073] When the candidate face detection region includes all key points corresponding to the corresponding key point annotation information, the candidate face detection region is used as the target face detection region.

[0074] Specifically, after performing face detection on each training sample image to obtain candidate face detection areas, manually annotated key points are used to screen the candidate face detection areas. If all key points corresponding to the corresponding key point annotation information fall into the candidate face detection area, the candidate face detection area is retained as the target face detection area; conversely, if all key points corresponding to the corresponding key point annotation information do not all fall into the candidate face detection area, the candidate face detection area is not used as the target face detection area and is deleted.

[0075] Exemplarily, the key point annotation information may include the positions of the eye key points, nose key points, and mouth key points in the training sample image. The eye key points may include two eye corners and the center of the eye, the nose key points may be the center of the nose, and the mouth key points may include two mouth corners and the center of the mouth. Thus, the positions of the five key points may be annotated for each face in the training sample image. After obtaining the candidate facial detection region, based on the key point annotation information of the face corresponding to the candidate facial detection region, it is determined whether all five annotated key points on the face fall within the candidate facial detection region. If all do, the candidate facial detection region is retained as the target facial detection region. If not, the candidate facial detection region is not retained.

[0076] In the above implementation, by screening the candidate facial detection areas obtained by facial detection based on the corresponding key point annotation information, it is ensured that the training data used to train the key point detection model can play a positive role in the training process, which is conducive to improving the accuracy of the key point detection model.

[0077] In an exemplary embodiment, performing face detection on the training sample image to obtain the candidate face detection area may include: inputting the training sample image into a face detection model to perform face detection, and obtaining output face detection frame information.

[0078] The facial detection frame information indicates the candidate facial detection area. The facial detection model can be a neural network model deployed in a facial recognition system for implementing facial detection, such as a regional convolutional neural network model, a fast regional convolutional neural network model, and the like.

[0079] For example, the face detection frame information can be expressed as (x, y, w det , h det ), where (x, y) is the horizontal and vertical coordinates of the upper left corner of the detection box, w det is the width of the detection box, h det is the height of the detection box.

[0080] Take face detection as an example, Figure 2 The figure below shows an example process for obtaining the target face detection region. A training sample image set is input into the face detection model, which then performs face detection on each training sample image to obtain a corresponding face detection frame. The model then filters the face detection frames based on the key point annotations of each face detection frame to obtain the required face detection frames for training, which must contain all the annotated key points within the corresponding face detection frame.

[0081] The first scale enhancement process refers to scaling the target facial detection area to adjust the size of the area. Exemplarily, the adjusted first facial enhancement image is a square image.

[0082] In a specific implementation, the first scale enhancement processing can be performed based on the first scale enhancement range. Based on this, the above-mentioned first scale enhancement processing on the target facial detection area can include: randomly selecting a first scale enhancement value from the first scale enhancement range; and performing the first scale enhancement processing on the target facial detection area based on the first scale enhancement value.

[0083] For example, if (α, β) represents the first scale enhancement range, the first scale enhancement value γ can be expressed as γ=random(α, β), where α and β can be empirical values ​​in actual applications, such as α is 0.9 and β is 1.3; random() represents a random function. Then, when performing the first scale enhancement processing on the target face detection area, the maximum value of the width and height corresponding to the target face detection area can be first selected as the side length to be enhanced, that is, the side length to be enhanced rec det It can be expressed as rec det =max(w det , h det ), and then use the first scale enhancement value γ to treat the enhanced side length rec det Adjust the side length of the first face enhancement picture to obtain the side length of the first face enhancement picture Thus the first face-enhanced picture is based on the A square image with side length .

[0084] S103: Determine a facial reference region corresponding to the target facial detection region based on key point annotation information corresponding to the target facial detection region in the training sample image, and perform a second scale enhancement process on the facial reference region to obtain a second facial enhanced image.

[0085] Specifically, the facial reference region is an estimated facial region based on the key point annotation information corresponding to the target facial detection region in the training sample image. In a specific implementation, the facial reference region can be determined by the following steps:

[0086] Based on the first face enhancement image, performing an affine transformation on the coordinates of the key point annotation information corresponding to the target face detection area to obtain the target key point annotation information;

[0087] Based on the target key point labeling information, a facial reference area corresponding to the target facial detection area is determined.

[0088] The target keypoint annotation information includes the coordinate information of each keypoint in the first enhanced facial image. Specifically, an affine transformation of the coordinates can be used to determine the position, i.e., the coordinate information, of the corresponding keypoint in the training sample image in the first enhanced facial image. Affine transformation is essentially a superposition of linear transformation and translation transformation. The specific affine transformation process can be found in related descriptions in the prior art and will not be elaborated here.

[0089] In an exemplary embodiment, the above-mentioned determination of the facial reference area corresponding to the target facial detection area based on the target key point annotation information may include the following steps:

[0090] Determining maximum coordinate information and minimum coordinate information based on coordinate information of each key point in the first facial enhancement image;

[0091] A target square area is determined based on the maximum coordinate information and the minimum coordinate information; the target square area serves as a facial reference area corresponding to the target facial detection area.

[0092] The maximum coordinate information includes a maximum horizontal coordinate and a maximum vertical coordinate, and the minimum coordinate information includes a minimum horizontal coordinate and a minimum vertical coordinate.

[0093] In a specific implementation, the following formula can be used to determine the maximum coordinate information and the minimum coordinate information:

[0094]

[0095] Among them, xmin represents the minimum horizontal coordinate; xmax represents the maximum horizontal coordinate; ymin represents the minimum vertical coordinate; ymax represents the maximum vertical coordinate; gt x Represents the horizontal coordinate of the key point in the target key point annotation information; gt y Indicates the vertical coordinate of the key point in the target key point annotation information.

[0096] Then, the target square area can be obtained by the following formula:

[0097]

[0098] Among them, rec pts Indicates the side length of the target square area.

[0099] The second scale enhancement process refers to scaling the facial reference region to adjust the size of the region. Exemplarily, the scaled second facial enhancement image is a square image.

[0100] In a specific implementation, the second scale enhancement process can be performed based on a second scale enhancement range. It should be noted that the second scale enhancement range needs to be determined based on the aforementioned first scale enhancement range. Based on this, performing the second scale enhancement process on the facial reference region may include: randomly selecting a second scale enhancement value from the second scale enhancement range; and performing the second scale enhancement process on the facial reference region based on the second scale enhancement value.

[0101] For example, if (λ, μ) represents the second scale enhancement range, the second scale enhancement value θ can be expressed as θ=random(λ, μ), where random() represents a random function. Then, when performing the second scale enhancement process on the facial reference area, the second scale enhancement value θ is used to adjust the side length rec of the target square area. pts Adjust the side length of the second face enhancement picture to obtain the side length of the second face enhancement picture Thus the second face enhanced picture is based on the A square image with side length .

[0102] Based on this, in an exemplary embodiment, Figure 3 FIG. 1 is a schematic diagram of another key point detection model training method. After obtaining the target facial detection area in the training sample image, the method further includes a step of determining a scale enhancement range. That is, the method may also include:

[0103] Obtaining a preset scale enhancement limit; the preset scale enhancement limit includes a preset scale enhancement upper limit and a preset scale enhancement lower limit;

[0104] Performing scale enhancement processing on the target face detection area based on a preset scale enhancement limit to obtain an enhanced face detection area;

[0105] Determine the sample facial area based on the key point annotation information of the target facial detection area;

[0106] Determining a target scale enhancement limit based on the enhanced face detection region, the sample face region, and a preset overlap ratio; wherein the target scale enhancement limit includes a target scale enhancement upper limit and a target scale enhancement lower limit that ensure that the overlap ratio between the enhanced face detection region and the enhanced sample face region reaches the preset overlap ratio; and the enhanced sample face region is a region obtained by performing scale enhancement processing on the sample face region based on the target scale enhancement limit.

[0107] The range indicated by the preset scale enhancement limit is used as the first scale enhancement range, and the range indicated by the target scale enhancement limit is used as the second scale enhancement range.

[0108] The determination of the sample facial area may refer to the aforementioned method for determining the facial reference area, which will not be described in detail here.

[0109] The overlap rate is a concept used in object detection. It represents the overlap rate of the "predicted bounding box" and the "real bounding box", that is, the ratio of their intersection and union, which can also be called IOU (Intersection over Union). The ideal situation is complete overlap, that is, the overlap rate is 1.

[0110] The preset overlap ratio can be set according to actual needs. In the embodiment of the present invention, the preset overlap ratio can be set to 90% or above.

[0111] Specifically, assuming that the preset scale enhancement upper limit is α and the preset scale enhancement lower limit is β, the values ​​of α and β can be set based on actual experience, such as α = 0.9 and β = 1.3. Then, α and P are used to scale the target face detection area (i.e., scale enhancement), respectively, to obtain the enhanced face detection area under the two scale enhancement processes. Among them, the enhanced face detection area corresponding to α can be used to calculate the lower limit λ of the second scale enhancement range, i.e., the target scale enhancement lower limit; the enhanced face detection area corresponding to β can be used to calculate the upper limit μ of the second scale enhancement range, i.e., the target scale enhancement upper limit.

[0112] Taking the preset IOU of 90% as an example, assuming α = 0.9, then when calculating the lower limit λ of the second scale enhancement range, the target face detection area can be scaled by 0.9 to obtain the enhanced face detection area, and the sample face area can be scaled with λ as the unknown parameter to obtain the enhanced sample face area. Then, a solution equation is constructed based on the IOU between the enhanced face detection area and the enhanced sample face area (including the unknown parameter λ). The constraint of the equation is that the IOU is equal to 90% (that is, the preset IOU). Based on the equation, the value of the unknown parameter λ can be solved, and then the lower limit of the second scale enhancement range can be obtained.

[0113] Similarly, assuming β = 1.3, when calculating the upper limit μ of the second scale enhancement range, the target face detection area can be scaled by 1.3 to obtain the enhanced face detection area, and the sample face area can be scaled with μ as the unknown parameter to obtain the enhanced sample face area. Then, a solution equation is constructed based on the IOU between the enhanced face detection area and the enhanced sample face area (including the unknown parameter μ). The constraint of the equation is that the IOU is equal to 90% (that is, the preset IOU). Based on the equation, the value of the unknown parameter μ can be solved, and then the upper limit of the second scale enhancement range can be obtained.

[0114] Then, (0.9, 1.3) is used as the first scale enhancement range, and the calculated (λ, μ) is used as the second scale enhancement range, so that the second scale enhancement range corresponding to the first scale enhancement range can be obtained.

[0115] In order to improve the accuracy of the second scale enhancement range and thus improve the accuracy of the final model training, multiple target face detection areas can be centrally sampled based on the target face detection areas corresponding to the training sample image set. Then, the upper and lower limits of the second scale enhancement range are calculated for each target face detection area. Finally, the mean of the upper limit of the second scale enhancement range and the mean of the lower limit of the second scale enhancement range are calculated, and the calculated mean values ​​are used as the upper and lower limits of the target scale enhancement, thereby obtaining a more accurate second scale enhancement range.

[0116] S105: Perform key point detection training on a preset neural network model based on the first facial enhancement picture and the second facial enhancement picture to obtain a key point detection model.

[0117] The key point detection model is used to detect facial key points in facial images.

[0118] In an embodiment of the present invention, the target training sample set used to train the key point detection model includes a first facial enhancement picture, and the target training sample set is also expanded using a second facial enhancement picture, that is, the target training sample set includes the first facial enhancement picture and the second facial enhancement picture, and the target training samples in the expanded target training sample set are used for training when training the key point detection model.

[0119] Among them, the preset neural network model can be but is not limited to a convolutional neural network, and can include operations such as convolution calculation, nonlinear activation function (ReLU) calculation, and pooling calculation.

[0120] In a specific embodiment, performing key point detection training on a preset neural network model based on the first face-enhanced image and the second face-enhanced image to obtain a key point detection model may include the following steps:

[0121] The first enhanced face image and the second enhanced face image are respectively used as target training samples, and input into a preset neural network model for key point detection to obtain predicted key point information;

[0122] Determine a loss value based on the difference between the predicted key point information and the target key point annotation information using a preset regression loss function;

[0123] The model parameters of the preset neural network model are adjusted based on the loss value and the iterative training is continued until the preset training end condition is met to obtain the key point detection model.

[0124] Specifically, keypoint detection involves extracting facial keypoints from the target training sample to obtain predicted keypoint information, which includes the coordinates of each predicted keypoint. The preset regression loss function can be the commonly used L1 loss function (i.e., MAE, Mean Absolute Error), L2 loss function (i.e., MSE, Mean Square Error), or smoth_L1 loss function. Of course, other regression loss functions can also be used.

[0125] When adjusting the model parameters of a preset neural network model based on the loss value, you can use gradient descent methods, such as stochastic gradient descent, stochastic gradient descent with momentum, the Adagard method, and the Adam method. The Adagard method automatically changes the learning rate during training, setting a global learning rate. The actual learning rate is inversely proportional to the square root of the sum of the previous parameter norms. The Adam method uses the first-order and second-order moment estimates of the gradient to dynamically adjust the learning rate of each parameter. After bias correction, the learning rate after each iteration has a certain range, making the parameters relatively stable.

[0126] The preset training termination condition may be that the number of iterations reaches a preset iteration threshold, that the loss value is less than a preset loss threshold, or that the difference between the loss values ​​of two adjacent iterations is less than a preset difference value. When the preset training termination condition is met, the training of the preset neural network model is terminated, and the preset neural network model corresponding to the model parameters at the end of training is used as the key point detection model.

[0127] As can be seen from the above technical solutions of the embodiments of the present invention, the embodiments of the present invention perform joint scale enhancement using the facial detection region and the facial region estimated by the corresponding annotated key points, and train the key point detection model based on the jointly scale enhanced data. Since the facial region estimated based on the annotated key points does not change with the facial detection algorithm, the stability of the facial region in the training data is ensured, thereby improving the robustness of the key point detection model. When applied to a face recognition system, the key point detection model obtained based on the embodiments of the present invention is robust to upgrades of the face detection model. That is, upgrades to the face detection model do not affect the accuracy of the key point detection model, thereby ensuring independent upgrades between the various models in the face recognition system.

[0128] It is understandable that in order to further improve the accuracy and robustness of the trained key point detection model, other training methods can be combined when training the key point detection model based on the target training sample set. For example, knowledge distillation technology can be used for knowledge transfer training. For information about knowledge distillation technology, please refer to related technologies and will not go into details here.

[0129] Corresponding to the key point detection model training methods provided in the above-mentioned embodiments, an embodiment of the present invention also provides a key point detection model training device. Since the key point detection model training device provided in the embodiment of the present invention corresponds to the key point detection model training methods provided in the above-mentioned embodiments, the implementation method of the aforementioned key point detection model training method is also applicable to the key point detection model training device provided in this embodiment and will not be described in detail in this embodiment.

[0130] See also Figure 4 , which is a schematic diagram of the structure of a key point detection model training device provided by an embodiment of the present invention, the device has the function of implementing the key point detection model training method in the above method embodiment, and the function can be implemented by hardware or by hardware executing corresponding software. Figure 4 As shown, the key point detection model training device 400 may include:

[0131] A first scale enhancement module 410 is configured to obtain a target face detection region in a training sample image and perform a first scale enhancement process on the target face detection region to obtain a first face enhanced image;

[0132] A second scale enhancement module 420 is configured to determine a facial reference region corresponding to the target facial detection region based on key point annotation information corresponding to the target facial detection region in the training sample image, and perform a second scale enhancement process on the facial reference region to obtain a second facial enhanced image;

[0133] The model training module 430 is used to perform key point detection training on a preset neural network model based on the first facial enhancement picture and the second facial enhancement picture to obtain a key point detection model; the key point detection model is used to detect facial key points in facial pictures.

[0134] In an exemplary embodiment, the first scale enhancement module 410 includes:

[0135] A first random selection module, configured to randomly select a first scale enhancement value from a first scale enhancement range;

[0136] a first scale enhancement processing submodule, configured to perform a first scale enhancement processing on the target face detection area based on the first scale enhancement value;

[0137] The second scale enhancement module 420 includes:

[0138] A second random selection module, configured to randomly select a second scale enhancement value from a second scale enhancement range;

[0139] The second scale enhancement processing submodule is configured to perform a second scale enhancement processing on the facial reference area based on the second scale enhancement value; wherein the second scale enhancement range is determined based on the first scale enhancement range.

[0140] In an exemplary embodiment, the apparatus further includes a scale enhancement range determination module, wherein the scale enhancement range determination module is configured to:

[0141] Obtaining a preset scale enhancement limit; the preset scale enhancement limit includes a preset scale enhancement upper limit and a preset scale enhancement lower limit;

[0142] Performing scale enhancement processing on the target face detection area based on the preset scale enhancement limit to obtain an enhanced face detection area;

[0143] Determining a sample facial region based on the key point annotation information of the target facial detection region;

[0144] determining a target scale enhancement limit based on the enhanced face detection region, the sample face region, and a preset overlap ratio; wherein the target scale enhancement limit includes a target scale enhancement upper limit and a target scale enhancement lower limit that ensure that the overlap ratio between the enhanced face detection region and the enhanced sample face region reaches the preset overlap ratio; and the enhanced sample face region is a region obtained by performing scale enhancement processing on the sample face region based on the target scale enhancement limit;

[0145] The range indicated by the preset scale enhancement limit is used as the first scale enhancement range, and the range indicated by the target scale enhancement limit is used as the second scale enhancement range.

[0146] In an exemplary embodiment, the second scale enhancement module 420 further includes:

[0147] an affine transformation module, configured to perform an affine transformation of the coordinates of the key point annotation information corresponding to the target facial detection area based on the first facial enhancement image, to obtain target key point annotation information; the target key point annotation information includes coordinate information of each key point in the first facial enhancement image;

[0148] The facial reference area determination module is used to determine the facial reference area corresponding to the target facial detection area based on the target key point annotation information.

[0149] In an exemplary embodiment, the facial reference area determination module includes:

[0150] a coordinate information determining module, configured to determine maximum coordinate information and minimum coordinate information based on the coordinate information of each key point in the first facial enhancement image; the maximum coordinate information includes a maximum horizontal coordinate and a maximum vertical coordinate, and the minimum coordinate information includes a minimum horizontal coordinate and a minimum vertical coordinate;

[0151] The square area determination module is used to determine a target square area based on the maximum coordinate information and the minimum coordinate information; the target square area serves as a facial reference area corresponding to the target facial detection area.

[0152] In an exemplary embodiment, the model training module 430 includes:

[0153] A key point prediction module is configured to input the first enhanced face image and the second enhanced face image as target training samples into a preset neural network model for key point detection to obtain predicted key point information;

[0154] a loss value determination module, configured to determine a loss value based on a difference between the predicted key point information and the target key point annotation information using a preset regression loss function;

[0155] An iterative training module is used to adjust the model parameters of the preset neural network model based on the loss value and continue iterative training until a preset training end condition is met to obtain the key point detection model.

[0156] In an exemplary embodiment, the first scale enhancement module 410 further includes:

[0157] A training sample image acquisition module is used to acquire training sample images from a training sample image set; the training sample images include key point annotation information of the face;

[0158] A face detection module is used to perform face detection on the training sample image to obtain candidate face detection areas;

[0159] The face detection region screening module is configured to use the candidate face detection region as the target face detection region when the candidate face detection region contains all key points corresponding to the corresponding key point annotation information.

[0160] In an exemplary embodiment, the face detection module is specifically configured to input the training sample image into a face detection model for face detection to obtain output face detection frame information; the face detection frame information indicates the candidate face detection area.

[0161] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0162] An embodiment of the present invention provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any key point detection model training method provided in the above method embodiment.

[0163] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0164] The method embodiments provided in the embodiments of the present invention can be executed in a computer terminal, a server or a similar computing device, that is, the above electronic device can include a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 5 This is a hardware structure diagram of an electronic device for running a key point detection model training method provided by an embodiment of the present invention, such as Figure 5As shown, the server 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 510 (the processor 510 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 530 for storing data, and one or more storage media 520 (such as one or more mass storage devices) for storing application programs 523 or data 522. Among them, the memory 530 and the storage medium 520 can be temporary storage or permanent storage. The program stored in the storage medium 520 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the central processing unit 510 can be configured to communicate with the storage medium 520 to execute a series of instruction operations in the storage medium 520 on the server 500. The server 500 may also include one or more power supplies 560, one or more wired or wireless network interfaces 550, one or more input and output interfaces 540, and / or one or more operating systems 521, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0165] The input / output interface 540 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communication provider of the server 500. In one embodiment, the input / output interface 540 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the input / output interface 540 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0166] It can be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components than shown, or with Figure 5 Different configurations shown.

[0167] An embodiment of the present invention also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a key point detection model training method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any one of the key point detection model training methods provided in the above method embodiments.

[0168] Embodiments of the present invention further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the key point detection model training methods described above.

[0169] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0170] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0171] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0172] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0173] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A key point detection model training method, characterized in that: The method comprises: Get the target face detection area in the training sample image; Obtaining preset scale enhancement limits, the preset scale enhancement limits including a preset scale enhancement upper limit and a preset scale enhancement lower limit; performing scale enhancement processing on the target face detection area based on the preset scale enhancement limits to obtain an enhanced face detection area; and determining a sample face area based on key point annotation information corresponding to the target face detection area in the training sample image; Determining a target scale enhancement limit based on the enhanced face detection region, the sample face region, and a preset overlap ratio; wherein the target scale enhancement limit includes a target scale enhancement upper limit and a target scale enhancement lower limit for ensuring that the overlap ratio between the enhanced face detection region and the enhanced sample face region reaches the preset overlap ratio; the enhanced sample face region is a region obtained by performing scale enhancement processing on the sample face region based on the target scale enhancement limit; using a range indicated by the preset scale enhancement limit as a first scale enhancement range, and using a range indicated by the target scale enhancement limit as a second scale enhancement range; randomly selecting a first scale enhancement value from the first scale enhancement range; performing a first scale enhancement process on the target face detection area based on the first scale enhancement value to obtain a first face enhanced image; Determining a facial reference region corresponding to the target facial detection region based on key point annotation information corresponding to the target facial detection region in the training sample image, randomly selecting a second scale enhancement value from the second scale enhancement range; and performing a second scale enhancement process on the facial reference region based on the second scale enhancement value to obtain a second facial enhanced image; Based on the first facial enhancement picture and the second facial enhancement picture, a preset neural network model is trained for key point detection to obtain a key point detection model; the key point detection model is used to detect facial key points in facial pictures.

2. The method according to claim 1, characterized in that The determining, based on key point annotation information corresponding to the target facial detection area in the training sample image, a facial reference area corresponding to the target facial detection area includes: Based on the first face-enhanced image, performing an affine transformation on the key point annotation information corresponding to the target face detection area to obtain target key point annotation information; the target key point annotation information includes coordinate information of each key point in the first face-enhanced image; Based on the target key point labeling information, a facial reference area corresponding to the target facial detection area is determined.

3. The method according to claim 2, characterized in that The determining, based on the target key point labeling information, a facial reference area corresponding to the target facial detection area comprises: Determining maximum coordinate information and minimum coordinate information based on coordinate information of each key point in the first facial enhancement image; the maximum coordinate information includes a maximum horizontal coordinate and a maximum vertical coordinate, and the minimum coordinate information includes a minimum horizontal coordinate and a minimum vertical coordinate; A target square area is determined based on the maximum coordinate information and the minimum coordinate information; the target square area serves as a facial reference area corresponding to the target facial detection area.

4. The method according to claim 2, characterized in that The key point detection training of a preset neural network model based on the first face enhancement picture and the second face enhancement picture to obtain a key point detection model includes: The first enhanced face image and the second enhanced face image are respectively used as target training samples, and are input into a preset neural network model to perform key point detection to obtain predicted key point information; Determining a loss value based on the difference between the predicted key point information and the target key point annotation information using a preset regression loss function; The model parameters of the preset neural network model are adjusted based on the loss value and the iterative training is continued until the preset training end condition is met to obtain the key point detection model.

5. The method according to any one of claims 1 to 4, characterized in that The step of obtaining a target facial detection area in a training sample image includes: Obtaining a training sample image from a training sample image set; the training sample image includes key point annotation information of a face; Performing face detection on the training sample image to obtain candidate face detection areas; When the candidate face detection region includes all key points corresponding to the corresponding key point annotation information, the candidate face detection region is used as the target face detection region.

6. The method according to claim 5, characterized in that The performing face detection on the training sample image to obtain candidate face detection areas comprises: The training sample image is input into a face detection model for face detection to obtain output face detection frame information; the face detection frame information indicates the candidate face detection area.

7. A key point detection model training device, characterized in that: The device comprises: The first scale enhancement module is used to obtain the target face detection area in the training sample image; A scale enhancement range determination module is configured to obtain preset scale enhancement limits, the preset scale enhancement limits comprising a preset scale enhancement upper limit and a preset scale enhancement lower limit; perform scale enhancement processing on the target face detection region based on the preset scale enhancement limits to obtain an enhanced face detection region; determine a sample face region based on key point annotation information corresponding to the target face detection region in the training sample image; determine a target scale enhancement limit based on the enhanced face detection region, the sample face region, and a preset overlap ratio; wherein the target scale enhancement limit comprises a target scale enhancement upper limit and a target scale enhancement lower limit that ensure that the overlap ratio between the enhanced face detection region and the enhanced sample face region reaches the preset overlap ratio, and the enhanced sample face region is a region obtained by performing scale enhancement processing on the sample face region based on the target scale enhancement limit; and use the range indicated by the preset scale enhancement limit as a first scale enhancement range and the range indicated by the target scale enhancement limit as a second scale enhancement range; The first scale enhancement module is further configured to randomly select a first scale enhancement value from the first scale enhancement range; perform a first scale enhancement process on the target face detection area based on the first scale enhancement value to obtain a first face enhanced image; a second scale enhancement module, configured to determine a facial reference region corresponding to the target facial detection region based on key point annotation information corresponding to the target facial detection region in the training sample image, randomly select a second scale enhancement value from the second scale enhancement range, and perform a second scale enhancement process on the facial reference region based on the second scale enhancement value to obtain a second facial enhanced image; A model training module is used to perform key point detection training on a preset neural network model based on the first facial enhancement picture and the second facial enhancement picture to obtain a key point detection model; the key point detection model is used to detect facial key points in facial pictures.

8. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the key point detection model training method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the key point detection model training method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the key point detection model training method according to any one of claims 1 to 6.

Images