Key point prediction model training method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a key point prediction model training method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a sample key point in a sample image and first sample position information of the sample key point; extracting a feature map of the sample image through a to-be-trained key point prediction model; determining first prediction position information of the sample key point and first prediction offset information of a target pixel area where the sample key point is located in the feature map; determining a model loss value based on the first sample position information, the first prediction position information, and first sample offset information and first prediction offset information of the sample key point in the target pixel region; and performing model parameter updating on the to-be-trained key point prediction model based on the model loss value to obtain a trained key point prediction model. According to the invention, the precision and stability of the key point prediction model can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a key point prediction model training method, device, electronic device and computer-readable storage medium. Background Art

[0002] Key point detection is an important task in the field of computer vision and is widely used in scenarios such as face recognition, expression analysis, and image editing.

[0003] In the related art, the main methods of key point detection include methods based on heatmap to determine the key point position and methods based on regression method to determine the key point position. Since the method based on heatmap to determine the key point position is slow, the method based on regression method to determine the key point position is generally used for face recognition and expression analysis.

[0004] However, the method based on regression method to determine the location of key points has low accuracy and stability. Summary of the invention

[0005] The embodiments of the present application provide a key point prediction model training method, device, electronic device and computer-readable storage medium, which can improve the accuracy and stability of the key point prediction model.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] An embodiment of the present application provides a key point prediction model training method, the method comprising: obtaining sample key points in a sample image and first sample position information of the sample key points; extracting a feature map of the sample image through a key point prediction model to be trained; determining the first predicted position information of the sample key points, and first predicted offset information of a target pixel area where the sample key points are located in the feature map; determining a model loss value based on the first sample position information, the first predicted position information, the first sample offset information of the sample key points in the target pixel area, and the first predicted offset information; and updating model parameters of the key point prediction model to be trained based on the model loss value to obtain a trained key point prediction model.

[0008] An embodiment of the present application provides a key point prediction model training device, the device comprising: a sample acquisition module, used to acquire sample key points in a sample image and first sample position information of the sample key points; a feature map extraction module, used to extract the feature map of the sample image through the key point prediction model to be trained; a prediction module, used to determine the first prediction position information of the sample key points, and the first prediction offset information of the target pixel area where the sample key points are located in the feature map; a loss determination module, used to determine a model loss value based on the first sample position information, the first prediction position information, the first sample offset information of the sample key points in the target pixel area, and the first prediction offset information; a model training module, used to update the model parameters of the key point prediction model to be trained based on the model loss value to obtain a trained key point prediction model.

[0009] In the above scheme, the loss determination module is also used to determine the first loss value of the key point prediction model based on the first sample position information; determine the sample neighbor key point of the sample key point based on the first sample position information; determine the second loss value of the key point prediction model based on the first sample offset information, the first predicted offset information, the second sample offset information of the sample neighbor key point in the target pixel area, and the second predicted offset information of the sample neighbor key point in the target pixel area; determine the third loss value of the key point prediction model based on the first sample position information, the first predicted position information, the second sample position information of the sample neighbor key point, and the second predicted position information of the sample neighbor key point; determine the model loss value based on the first loss value, the second loss value, and the third loss value.

[0010] In the above scheme, the loss determination module is also used to determine the first label score of the sample key point located in the pixel area of ​​the feature map based on the first sample position information; perform feature mapping processing on the feature map to obtain a first prediction score of the sample key point located in the pixel area; perform feature map loss calculation on the first prediction score and the first label score to obtain the first loss value.

[0011] In the above scheme, the loss determination module is also used to determine that the first label score is a first preset score when it is determined that the sample key point is located within the pixel area based on the first sample position information; and to determine that the first label score is a second preset score when it is determined that the sample key point is located outside the pixel area based on the first sample position information.

[0012] In the above scheme, the loss determination module is also used to perform a first offset loss calculation on the first prediction offset information and the first sample offset information to obtain a fourth loss value; perform a second offset loss calculation on the second prediction offset information and the second sample offset information to obtain a fifth loss value; and perform loss value fusion on the fourth loss value and the fifth loss value to obtain the second loss value.

[0013] In the above scheme, the loss determination module is also used to determine the first predicted distance between the sample key point and the sample neighbor key point based on the first predicted position information and the second predicted position information; determine the first sample distance between the sample key point and the sample neighbor key point based on the first sample position information and the second sample position information; perform distance loss calculation on the first predicted distance and the first sample distance to obtain a sixth loss value; perform position loss calculation on the first sample position information and the first predicted position information to obtain a seventh loss value; perform loss value fusion on the sixth loss value and the seventh loss value to obtain the third loss value.

[0014] In the above scheme, the loss determination module is also used to determine the position information error between the first sample position information and the first predicted position information; when the position information error is less than the preset position error, the first parameter is determined based on the preset parameter and the position information error, and the product of the preset position error and the first parameter is determined as the seventh loss value of the sample key point.

[0015] In the above scheme, the loss determination module is also used to determine the position information error between the first sample position information and the first predicted position information; when the position information error is greater than or equal to a preset position error, a second parameter is determined based on a preset parameter and the position information error, and the product of the preset position error and the second parameter is determined; the difference between the preset position error and the product is determined as a first difference; and the second difference between the position error and the first difference is determined as the seventh loss value.

[0016] An embodiment of the present application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the key point prediction model training method provided in the embodiment of the present application.

[0017] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the key point prediction model training method provided in the embodiment of the present application when executed by a processor.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] In the process of training the key point prediction model, the actual first sample position information of the sample key point and the first sample offset information of the target pixel area of ​​the sample key point in the feature map, as well as the first predicted offset information of the sample key point predicted by the key point prediction model and the first predicted position information of the target pixel area of ​​the sample key point in the feature map are determined. A more accurate model loss value can be calculated using the first sample position information, the first predicted position information, the first sample offset information and the first predicted offset information. The key point prediction model can then be trained using the model loss value to improve the detection accuracy and stability of the key point prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the architecture of the key point prediction model training system provided in an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of the process of the key point prediction model training method provided in the embodiment of the present application. Figure 1 ;

[0023] Figure 4 It is a structural diagram of a key point prediction model provided in an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of the process of the key point prediction model training method provided in the embodiment of the present application. Figure 2 ;

[0025] Figure 6 This is a schematic diagram of the process of the key point prediction model training method provided in the embodiment of the present application. Figure 3 ;

[0026] Figure 7 This is a schematic diagram of the process of the key point prediction model training method provided in the embodiment of the present application. Figure 4 ;

[0027] Figure 8 This is a schematic diagram of the process of the key point prediction model training method provided in the embodiment of the present application. Figure 5 ;

[0028] Fig. 9 It is another optional flowchart of the key point prediction model training method provided in the embodiment of the present application;

[0029] Fig.10It is a structural diagram of a basic model of the key point prediction model proposed in the embodiment of the present application;

[0030] Fig.11 It is a schematic diagram of multiple feature graphs provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0035] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0037] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0038] 1) Key points: refers to points in an image that have an identification function, such as facial key points used to describe the key feature positions of a face. Facial key points include but are not limited to the following parts: eyes, nose, mouth, eyebrows, facial contours, etc.

[0039] 2) Key point prediction model: It is a machine learning model used to predict the location of key points in an image. The key point prediction model takes an image as input and outputs the specific coordinates of the key points.

[0040] 3) Sample images: refers to images with known key point annotations used in the training or testing process. Sample images are used to train and evaluate the performance of key point prediction models.

[0041] 4) Feature map: It is a multidimensional array extracted from the input image by the convolutional neural network (CNN) in the key point prediction model, which reflects the local features and structural information in the image.

[0042] 5) Pixel area: refers to the local area in the feature map, usually a window or grid of a fixed size.

[0043] 6) Sample key points: refers to the key points in the sample images used in the training process to guide the key point prediction model to learn the correct key point locations.

[0044] 7) Sample position information: refers to the specific coordinate position of the sample key point in the sample image.

[0045] 8) Prediction position information: refers to the coordinate position of the key point predicted by the key point prediction model in the sample image.

[0046] 9) Sample offset information: refers to the relative position information of the sample key point within the target pixel area where it is located, usually expressed as an offset relative to the upper left corner of the target pixel area.

[0047] 10) Prediction offset information: refers to the relative position information of the key point predicted by the key point prediction model within the target pixel area where it is located, also expressed as an offset relative to the upper left corner of the target pixel area.

[0048] The mainstream methods of facial key point detection mainly include heatmap method and regression method. The heatmap method represents the position of the key point as a probability map. The value of each pixel in the map represents the probability that the position belongs to a certain key point. The position of the key point is determined by finding the highest probability of each pixel. The regression method directly predicts the coordinates of the key point, regards the position of the key point as a continuous value, and directly regresses these coordinates through the key point prediction model. The key point detection accuracy of the heatmap method is high but the speed is slow. However, in practical applications, facial key point detection is generally applied to edge platforms. Edge platforms (refer to computing devices or systems located at the edge of the network) have limited capacity and therefore cannot support the heatmap method to predict key points. However, the accuracy and stability of predicting key points using regression methods are low.

[0049] Based on the problems existing in the related art, the embodiments of the present application provide a key point prediction model training method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy and stability of the key point prediction model. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a terminal or a server will be described.

[0050] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the key point prediction model training system provided in the embodiment of the present application. In order to support a key point prediction model training application, the key point prediction model training system 100 in the embodiment of the present application includes at least a terminal 400, a network 300 and a server 200. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0051] For example, taking the robot's lip movement speech recognition scenario as an example, after receiving the voice command of the target object (such as the user), the bionic humanoid robot can determine whether the target object is speaking by identifying the key points of the face. If the judgment result is that the target object is speaking, the human-computer interaction function is turned on and the voice command is responded; if the judgment result is that the target object is not speaking, the robot remains in the waiting state. Figure 1, the user can perform interactive operations on the client of the key point prediction model training application through the terminal 400, and the interactive operations can be, for example, inputting a sample image, clicking to start model training operations, etc. After receiving the interactive operation of the user, the client sends a key point prediction model training request to the server 200 through the network 300. After receiving the key point prediction model training request, the server 200 responds to the key point prediction model training request sent by the terminal, obtains the sample key points in the sample image and the first sample position information of the sample key points; the server 200 extracts the feature map of the sample image through the key point prediction model to be trained; the server 200 determines the first predicted position information of the sample key points, and the first predicted offset information of the target pixel area where the sample key points are located in the feature map; the server 200 determines the model loss value based on the first sample position information, the first predicted position information, the first sample offset information of the sample key points in the target pixel area, and the first predicted offset information; the server 200 updates the model parameters of the key point prediction model to be trained based on the model loss value to obtain the trained key point prediction model. After the key point prediction model is trained, the user can issue a voice command. After receiving the voice command, the robot's terminal 400 collects the user's facial image, encapsulates the facial image into a key point prediction request, and sends the key point prediction request to the server 200 through the network 300. In response to the key point prediction request, the server 200 processes the facial image based on the key point prediction model to obtain the key points of the face; the server 200 determines the user's speech state judgment result based on the key points of the face. The server 200 can send the speech state judgment result to the terminal 400. When the speech state judgment result is that the user is speaking, the terminal 400 wakes up and responds to the voice command; if the speech state judgment result is that the object is not speaking, the terminal 400 remains in the waiting state to be awakened.

[0052] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0053] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0054] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0055] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0056] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0057] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0058] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0059] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0060] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0061] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0062] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The key point prediction model training device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a sample acquisition module 4551, a feature map extraction module 4552, a prediction module 4553, a loss determination module 4554 and a model training module 4555. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0063] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the key point prediction model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0064] The key point prediction model training method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0065] The following describes the key point prediction model training method provided by the embodiment of the present application. As mentioned above, the electronic device that implements the key point prediction model training method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0066] It should be noted that in the example of key point prediction model training below, the image is a face image in a face recognition scenario. Based on the understanding of the following text, those skilled in the art can apply the key point prediction model training method provided in the embodiment of the present application to other scenarios, such as posture estimation scenarios, medical image analysis, autonomous driving scenarios, gesture recognition scenarios, etc.

[0067] Figure 3 This is a schematic diagram of the process of the key point prediction model training method provided in the embodiment of the present application. Figure 1 , will combine Figure 3 The steps shown are explained as Figure 3 As shown, the execution subject of the key point prediction model training method is a server as an example for explanation, and the method includes the following steps 101 to 105:

[0068] In step 101, sample key points in a sample image and first sample position information of the sample key points are obtained.

[0069] Here, the sample image is an image annotated with one or more known sample key points. The sample key points are points used to describe the positions of key features in the sample image. The first sample position information refers to the coordinates of the sample key points in the sample image. Exemplarily, the sample image is a face image with an image size of 112×112 pixels, that is, the width and height are both 112, and there are 98 sample key points in the sample image. Each sample key point is marked with a circle in the sample image and is accompanied by coordinates. The first sample position information of the sample key point can be the coordinates (10, 10). The 98 sample key points include but are not limited to the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth of the face.

[0070] In step 102, a feature map of the sample image is extracted by using the key point prediction model to be trained.

[0071] Here, the number of sample images can be one or more. The key point prediction model to be trained is a machine learning model for predicting the position of key points in an image. The embodiment of the present application does not limit the specific model structure of the key point prediction model. The sample image is input into the key point prediction model to be trained, and the convolution layer in the key point prediction model convolves the sample image to obtain a feature map.

[0072] Figure 4 is a schematic diagram of the structure of the key point prediction model provided in the embodiment of the present application. Figure 4, the key point prediction model can include multiple convolutional layers, a global maximum pooling layer (Group Max Pooling, GMP), a feature fully connected layer (fea_fc) and a result fully connected layer (res_fc). The sample image is a 112×112×3 face image, 112×112 is the width and height, and 3 is the number of channels. The sample image is subjected to multiple convolution operations to obtain an initial feature map of 7×7×32, and then the initial feature map is convolved to obtain a feature map of 7×7×98.

[0073] In step 103, first predicted position information of the sample key point and first predicted offset information of the target pixel region where the sample key point is located in the feature map are determined.

[0074] Here, the key point prediction model can be used to output the first predicted position information of each sample key point in the sample image. The first predicted position information is the coordinates of the sample key point on the sample image predicted by the key point prediction model. The first predicted position information and the first sample position information of the sample key point may be the same or different. Exemplarily, the sample image is a face image with an image size of 112×112 pixels and 98 sample key points. The first sample position information of the first sample key point A is the coordinates (10, 10), and the first predicted position information is (6, 8).

[0075] The feature map includes multiple pixel regions, and the multiple pixel regions are divided based on the width and height of the feature map. For example, if the size of the feature map is 7×7×98, the feature map can be divided into 7 rows and 7 columns, that is, 49 pixel regions. 98 is the number of channels, and each channel is used to predict a sample key point. For each sample key point, the target pixel region where the sample key point is located can also be determined based on the first sample position information of the sample key point. Since the original sample image size is 112×112, each pixel region in the feature map includes 16 pixels. Assuming that the coordinates of the upper left corner vertex of the sample image are (0, 0), the coordinates of the upper left corner vertex of the pixel region of the 1st row and 1st column in the feature map are (0, 0), and the coordinates of the lower right corner vertex are (16, 16). If the first sample position information of the sample key point A is (10, 10), the target pixel region of the sample key point A is the pixel region of the 1st row and 1st column.

[0076] The first prediction offset information may be the relative offset between the first prediction position information of the sample key point and the upper left corner vertex of the target pixel area. The first prediction offset information includes a relative prediction offset in a first direction and a relative prediction offset in a second direction. The first direction may be the x-axis direction in the coordinate system, and the second direction may be the y-axis direction in the coordinate system. Exemplarily, assuming that the first prediction offset information of the sample key point A in the target pixel area includes a relative prediction offset of 0.4 in the x-axis direction and a relative prediction offset of 0.5 in the y-axis direction, since there are 16 pixels in a pixel area, 16×0.4=6.4, the sample key point A and the upper left corner vertex of the target pixel area are offset to the right by 6 coordinate points, 16×0.5=8, the sample key point A and the upper left corner vertex of the target pixel area are offset downward by 8 coordinate points, and the first prediction position information of the sample key point A may be (6, 8).

[0077] See also Figure 4 , the key point prediction model can also obtain a 7×7×196 offset feature map by convolving the 7×7×32 initial feature map. 196 channels are used to predict the first predicted offset information of 98 sample key points in the first direction and the second direction.

[0078] In step 104, a model loss value is determined based on the first sample position information, the first predicted position information, the first sample offset information of the sample key point in the target pixel area, and the first predicted offset information.

[0079] Here, the first sample offset information may be the relative offset between the first sample position information of the sample key point and the upper left corner vertex of the target pixel area. The first sample offset information includes a relative sample offset in a first direction and a relative sample offset in a second direction. The method for determining the first sample offset information may refer to the method for determining the first prediction offset information in step 103, which will not be described in detail here. The model loss value is a quantitative indicator for measuring the difference between the prediction result of the key point prediction model and the true label of the sample key point. Based on the first sample position information and the first prediction position information of multiple sample key points, and the first sample offset information and the first prediction offset information of multiple sample key points in the target pixel area, multiple loss values ​​may be determined, and the multiple loss values ​​may be fused to obtain the model loss value.

[0080] In some embodiments, see Figure 5 , step 104 can be implemented by following steps 1041 to 1045, which are described in detail below.

[0081] In step 1041, a first loss value of the key point prediction model is determined based on the first sample position information.

[0082] Here, the first loss value is a quantitative indicator for measuring the prediction accuracy of the key point prediction model by comparing the first prediction score of the sample key point predicted by the key point prediction model in each pixel area of ​​the feature map with the first label score of the sample key point. Among them, the first prediction score is the probability that the sample key point predicted by the key point prediction model is located in the pixel area. The first label score includes two values ​​0 and 1. When the first label score is 0, the true coordinates of the sample key point are not located in the pixel area, and when the first label score is 1, the true coordinates of the sample key point are located in the pixel area. Therefore, the first loss value is a quantitative indicator for measuring the prediction accuracy of the key point prediction model by comparing the predicted probability of the sample key point predicted by the key point prediction model in each pixel area of ​​the feature map with the true label of the sample key point. The first label score of the sample key point can be determined based on the first sample position information of the sample key point. The first loss value is determined based on the first label score and the first prediction score of each sample key point located in each pixel area.

[0083] In some embodiments, see Figure 6 , step 1041 can be implemented by following steps 10411 to 10413, which are described in detail below.

[0084] In step 10411, based on the first sample position information, a first label score is determined when the sample key point is located in a pixel area in the feature map.

[0085] Here, the first label score is used to identify whether the sample key point is actually located in the pixel area. For each sample key point, the target pixel area where the sample key point is located can be determined based on the first sample position information of the sample key point. The first label score of the sample key point located in the target pixel area is set to "1", and the first label score of the sample key point located in the remaining multiple pixel areas in the feature map is set to "0".

[0086] In some embodiments, determining the first label score of the sample key point located in the pixel area of ​​the feature map based on the first sample position information in step 10411 can be achieved in the following manner: when it is determined that the sample key point is located in the pixel area according to the first sample position information, the first label score is determined to be a first preset score; or, when it is determined that the sample key point is located outside the pixel area according to the first sample position information, the first label score is determined to be a second preset score.

[0087] Exemplarily, the first preset score is a value of 1, and the second preset score is a value of 0. The first sample position information of the sample key point A is (10, 10). For the pixel area of ​​the 1st row and the 1st column in the feature map, the real coordinate (10, 10) of the sample key point A falls within the pixel area, and the sample key point A is determined to be located in the pixel area, and the first label score of the sample key point A in the pixel area is 1. For the pixel area of ​​the 1st row and the 2nd column, the real coordinate (10, 10) of the sample key point A does not fall within the pixel area, and the sample key point A is determined to be outside the pixel area, and the first label score of the sample key point A in the pixel area is 0.

[0088] In step 10412, feature mapping is performed on the feature map to obtain a first prediction score where the sample key point is located in the pixel area.

[0089] Exemplarily, the key point prediction model performs feature mapping processing on the 7×7×98 feature map, and can obtain the prediction probability of each key point in the 98 sample key points being located in each pixel area of ​​the 49 pixel areas, and determine the prediction probability as the first prediction score. The feature mapping processing can be a convolution processing. For the sample key point A, the first prediction score of the sample key point A located in the pixel area of ​​the 1st row and the 1st column is 0.95, and the first prediction score of the sample key point A located in the pixel area of ​​the 1st row and the 1st column is 0.03.

[0090] In step 10413, a feature map loss is calculated for the first prediction score and the first label score to obtain a first loss value.

[0091] Here, a loss function (such as cross entropy loss or mean square error) can be used to calculate the difference between the first prediction score and the first label score to obtain a first loss value. Exemplarily, for each sample key point and each pixel area, the score difference between the first label score and the first prediction score of the sample key point in the pixel area is determined, and the score difference is squared to obtain a square value. The first loss value can be determined based on the sum of the square values ​​of each sample key point in each pixel area and the size information of the feature map. The first loss value can satisfy the following formula (1).

[0092]

[0093] Among them, L s is the first loss value, i is the number of channels of the feature map, j is the height of the feature map, k is the width of the feature map, is the first label score of the i-th sample key point located in the pixel area of ​​the j-th row and the k-th column, The first prediction score of the i-th sample keypoint located in the pixel region of the j-th row and the k-th column.

[0094] The embodiment of the present application maps the actual positions of sample key points to pixel areas in a feature map and assigns first label scores to these pixel areas, which can clearly indicate which pixel areas contain real sample key points. By performing feature mapping processing on the feature map, a first prediction score is generated, which quantifies the possibility that each pixel area contains each sample key point. By calculating the difference between the first prediction score and the first label score, a first loss value is obtained, which can quantify the prediction error of the key point prediction model, help evaluate the prediction performance of the key point prediction model on the feature map, and guide subsequent parameter updates, thereby improving the overall prediction accuracy of the key point prediction model.

[0095] Continue to see Figure 5 , and the description continues with step 1041 above.

[0096] In step 1042, based on the first sample position information, the sample neighbor key points of the sample key point are determined.

[0097] Here, for each sample key point, the sample neighbor key point of the sample key point refers to the sample key point that is closest to the sample key point. The number of sample neighbor key points of the sample key point can be one or more. The sample distance between the sample key point and other sample key points can be determined based on the first sample position information of the sample key point and the first sample position information of other sample key points. At least one sample key point whose sample distance is less than or equal to the preset distance threshold is determined as the sample neighbor key point of the sample key point. Alternatively, multiple other sample key points can be sorted from small to large according to the sample distance, and the first N sample key points are selected to be determined as the sample neighbor key points of the sample key point, where N is a positive integer. Exemplarily, for the sample key point A among the 98 sample key points in the sample image, the sample distance between the sample key point A and the other 97 sample key points is calculated. The 10 sample key points with the smallest sample distance are selected from the 97 sample key points as the 10 sample neighbor key points of the sample key point A.

[0098] In step 1043, a second loss value of the key point prediction model is determined based on the first sample offset information, the first prediction offset information, the second sample offset information of the sample neighbor key point in the target pixel area, and the second prediction offset information of the sample neighbor key point in the target pixel area.

[0099] Here, the second loss value is an indicator to measure the prediction accuracy of the key point prediction model by quantifying the error of the key point prediction model in the key point offset prediction. Specifically, the second loss value combines the offset information of the sample key point and the sample neighbor key point of the sample key point in the feature map, so as to more comprehensively evaluate the prediction performance of the key point prediction model. The second sample offset information and the second predicted offset information of the sample neighbor key point in the target pixel area can refer to the first sample offset information and the first predicted offset information of the sample key point in the target pixel area in the other embodiments mentioned above, which will not be repeated here. Figure 4 The key point prediction model can also obtain a 7×7×1960 neighbor offset feature map by convolving the 7×7×32 initial feature map. Each of the 98 sample key points has 10 sample neighbor key points, and the 1960 channels are used to predict the second prediction offset information of the 980 sample neighbor key points in the first direction and the second direction.

[0100] In some embodiments, see Figure 7 , step 1043 can be implemented by following the steps 10431 to 10433, which are described in detail below.

[0101] In step 10431, a first offset loss calculation is performed on the first prediction offset information and the first sample offset information to obtain a fourth loss value.

[0102] Here, the fourth loss value is an indicator that measures the difference between the first predicted offset information of the sample key point predicted by the key point prediction model and the actual first sample offset information of the sample key point. The difference between the first predicted offset information and the first sample offset information can be calculated using a loss function (such as cross entropy loss or mean square error) to obtain the fourth loss value. Exemplarily, for each sample key point, the first offset difference between the first sample offset information and the first predicted offset information in the target pixel area of ​​the sample key point in the first direction, and the second offset difference between the first sample offset information and the first predicted offset information in the second direction are determined, and the fourth loss value is determined based on the sum of the first offset difference and the second offset difference. The fourth loss value may satisfy the following formula (2).

[0103]

[0104] Among them, L self-off is the fourth loss value, D=1 represents the first direction (x-axis direction), D=2 represents the second direction (y-axis direction), is the first prediction offset information in the x-axis or y-axis direction, is the first sample offset information in the x-axis or y-axis direction, It means that the first label score of the pixel area of ​​the jth row and jth column of the i-th sample key point is 1, that is, the pixel area of ​​the jth row and jth column is the target pixel area of ​​the i-th sample key point.

[0105] In step 10432, a second offset loss calculation is performed on the second prediction offset information and the second sample offset information to obtain a fifth loss value.

[0106] Here, the fifth loss value is an indicator that measures the difference between the second predicted offset information of the sample neighbor key point predicted by the key point prediction model and the actual second sample offset information of the sample neighbor key point. The difference between the second predicted offset information and the second sample offset information can be calculated using a loss function (such as cross entropy loss or mean square error) to obtain the fifth loss value. Exemplarily, for each sample key point, the third offset difference between the second sample offset information and the second predicted offset information in the first direction of each sample neighbor key point of the sample key point in the target pixel area, and the fourth offset difference between the second sample offset information and the second predicted offset information in the second direction are determined, and the fifth loss value is determined based on the sum of the third offset difference and the fourth offset difference. The fifth loss value may satisfy the following formula (3).

[0107]

[0108] Among them, L n-off is the fifth loss value, H is the first direction and the second direction of the 10 sample neighbor key nodes, is the second prediction offset information of the Hth sample neighbor key point of the i-th sample key point in the x-axis or y-axis direction, It is the second sample offset information of the Hth sample neighbor key point of the i-th sample key point in the x-axis or y-axis direction.

[0109] In step 10433, the fourth loss value and the fifth loss value are fused to obtain a second loss value.

[0110] Here, based on the preset first weight parameter corresponding to the fourth loss value and the second weight parameter corresponding to the fifth loss value, the fourth loss value and the fifth loss value can be weighted and summed to obtain the second loss value. The embodiment of the present application does not limit the specific values ​​of the first weight parameter and the second weight parameter, which can be set by oneself.

[0111] The embodiment of the present application calculates a fourth loss value reflecting the offset error of the sample key point and a fifth loss value reflecting the offset error of the sample neighbor key point, and fuses them to obtain a second loss value, thereby more comprehensively evaluating and optimizing the accuracy of the key point prediction model in the offset prediction of the key point and its neighbor key point, thereby improving the accuracy and stability of the trained key point prediction model.

[0112] Continue to see Figure 5 , and the description continues with step 1043 above.

[0113] In step 1044, a third loss value of the key point prediction model is determined based on the first sample position information, the first predicted position information, the second sample position information of the sample neighbor key point, and the second predicted position information of the sample neighbor key point.

[0114] Here, the third loss value is an indicator of the prediction distance error and position error between the comprehensive sample key point and the sample neighbor key point.

[0115] In some embodiments, see Figure 8 , step 1044 can be implemented by following steps 10441 to 10445, which are described in detail below.

[0116] In step 10441, a first predicted distance between a sample key point and a sample neighbor key point is determined based on the first predicted position information and the second predicted position information.

[0117] Here, for each sample key point, the first predicted distance between the sample key point and each sample neighbor key point can be calculated based on the first predicted position information of the sample key point and the second predicted position information of multiple sample neighbor key points of the sample key point. The embodiment of the present application does not limit the specific calculation formula of the first predicted distance. Since the first predicted position information and the second predicted position information are both coordinates, the method of calculating the distance using coordinates can be used.

[0118] In step 10442, based on the first sample position information and the second sample position information, a first sample distance between the sample key point and the sample neighbor key point is determined.

[0119] Here, for each sample key point, the first sample distance between the sample key point and each sample neighbor key point can be calculated based on the first sample position information of the sample key point and the second sample position information of multiple sample neighbor key points of the sample key point.

[0120] In step 10443, distance loss is calculated for the first predicted distance and the first sample distance to obtain a sixth loss value.

[0121] Here, the sixth loss value is an indicator for measuring the error between the distance between the sample key point predicted by the key point prediction model and the sample neighbor key point and the actual distance between the sample key point and the sample neighbor key point. The difference between the first predicted distance and the first sample distance can be calculated using a loss function (such as cross entropy loss or mean square error) to obtain the sixth loss value. Exemplarily, for each sample key point, the distance difference between the first sample distance of the sample key point and each sample neighbor key point and the first predicted distance is determined, and the sixth loss value is determined based on the sum of multiple distance differences. The sixth loss value can satisfy the following formula (4).

[0122]

[0123] Among them, L nb is the sixth loss value, P i is the i-th sample key point, P i_n is the nth sample neighbor key point of the i-th sample key point, Dist gt (P i -P i_n ) is the first sample distance between the i-th sample keypoint and the n-th sample neighbor keypoint of the i-th sample keypoint, Dist pred (P i -P i_n ) is the first predicted distance between the i-th sample keypoint and the n-th sample neighbor keypoint of the i-th sample keypoint.

[0124] In step 10444, position loss is calculated for the first sample position information and the first predicted position information to obtain a seventh loss value.

[0125] Here, the seventh loss value is an indicator for measuring the difference between the key point position predicted by the key point prediction model and the actual key point position. The loss function can be used to calculate the difference between the first sample position information and the first predicted position information to obtain the seventh loss value. The embodiment of the present application does not specifically limit the loss function for calculating the seventh loss value. For example, loss functions such as L1 loss and L2 loss can be used. Exemplarily, the regression loss function (wingloss) can be used to calculate the seventh loss value in the embodiment of the present application.

[0126] In the embodiment of the present application, the position loss calculation of the first sample position information and the first predicted position information in step 10444 to obtain the seventh loss value can be achieved in the following manner: First, determine the position information error between the first sample position information and the first predicted position information. Then, when the position information error is less than the preset position error, determine the first parameter based on the preset parameter and the position information error, and determine the product of the preset position error and the first parameter as the seventh loss value of the sample key point. Alternatively, when the position information error is greater than or equal to the preset position error, determine the second parameter based on the preset parameter and the position information error, and determine the product of the preset position error and the second parameter; determine the difference between the preset position error and the product as the first difference; and determine the second difference between the position error and the first difference as the seventh loss value.

[0127] It should be noted that the embodiments of the present application do not limit the values ​​of the preset position error and the preset parameter, and can be set based on actual conditions. For example, the preset position error is 10, and the preset parameter is 2. The calculation formulas for the first parameter and the second parameter may be the same or different. The position information error between the first sample position information and the first predicted position information is calculated separately in the first direction and the second direction. The difference between the first sample position information and the first predicted position information can be used as the position information error. That is, for sample key point A, the first sample position information of sample key point A is (10, 10), and the first predicted position information is (6, 8). Then, the position information error of sample key point A in the first direction is 10-6=4, and the position information error in the first direction is 10-8=2.

[0128] The seventh loss value may satisfy the following formula (5) and formula (6).

[0129]

[0130]

[0131] Among them, wing(x) is the seventh loss value, ω is the preset position error, |x| is the position information error, ∈ is the preset parameter, is the first parameter or the second parameter, and C is the first difference.

[0132] In step 10445, the sixth loss value and the seventh loss value are fused to obtain a third loss value.

[0133] Here, the sixth loss value and the seventh loss value are weighted and summed based on the preset weight parameter to obtain the third loss value.

[0134] The embodiment of the present application calculates the difference between the first prediction distance and the first sample distance, and combines the loss of the key point position to obtain a third loss value, which can more comprehensively evaluate the accuracy of the key point prediction model in predicting the positions of key points and their neighboring key points, thereby improving the overall performance of the key point prediction model.

[0135] Continue to see Figure 5 , and the description continues with step 1044 above.

[0136] In step 1045 , a model loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0137] Here, the sum of the first loss value, the second loss value and the third loss value may be determined as the model loss value. Alternatively, the first loss value, the second loss value and the third loss value may be weighted and summed based on a preset weight parameter to obtain the model loss value.

[0138] The embodiment of the present application obtains the model loss value by comprehensively considering the position difference and offset difference of the sample key points, the offset difference of the sample neighbor key points, and the distance difference between the sample key points and the sample neighbor key points. It can comprehensively evaluate and optimize the accuracy of the key point prediction model in key point prediction, thereby improving the overall performance.

[0139] Continue to see Figure 3 , and the description continues with step 104 above.

[0140] In step 105, based on the model loss value, the model parameters of the key point prediction model to be trained are updated to obtain the trained key point prediction model.

[0141] Here, the back propagation algorithm can be used to calculate the gradient of the model loss value with respect to the model parameters in the key point prediction model to be trained, and the model parameters can be updated using an optimization algorithm (such as a stochastic gradient descent method). The key point prediction model after the model parameters are updated is used to repeat steps 102-105 until the model loss value reaches the minimum value or reaches the preset number of training rounds, thereby obtaining a trained key point prediction model.

[0142] In the process of training the key point prediction model, the embodiment of the present application determines the actual first sample position information of the sample key point and the first sample offset information of the target pixel area of ​​the sample key point in the feature map, as well as the first predicted offset information of the sample key point predicted by the key point prediction model and the first predicted position information of the target pixel area of ​​the sample key point in the feature map. By using the first sample position information, the first predicted position information, the first sample offset information and the first predicted offset information, a more accurate model loss value can be calculated, and then the model loss value is used to train the key point prediction model, which can improve the detection accuracy and stability of the key point prediction model.

[0143] See also Figure 4 After training the key point prediction model, when using the key point prediction model to predict the key points of a face image, the key point prediction model performs convolution processing on the face image to obtain a first feature map of 7×7×98, a second feature map of 7×7×196, and a third feature map of 7×7×1980. The first feature map, the second feature map, and the third feature map are concatenated to obtain a target feature map, and then the target feature map is processed by pooling and mapping of the fully connected layer to obtain the coordinate values ​​of 98 key points.

[0144] Fig. 9 is another optional flow chart of the key point prediction model training method provided in the embodiment of the present application, such as Fig. 9 As shown, the method includes the following steps 201 to 208:

[0145] Step 201: The terminal receives an interactive operation from a user.

[0146] Here, the interactive operation can be clicking to input a sample image, clicking to start a model training operation, etc.

[0147] Step 202: The terminal generates a key point prediction model training request in response to the interactive operation.

[0148] Step 203: The terminal sends a key point prediction model training request to the server.

[0149] Step 204: The server obtains sample key points in the sample image and first sample position information of the sample key points in response to the key point prediction model training request sent by the terminal.

[0150] Here, the specific process of obtaining the sample key points in the sample image and the first sample position information of the sample key points can refer to step 101 in the above embodiment, and will not be repeated here.

[0151] Step 205: The server extracts a feature map of the sample image through the key point prediction model to be trained.

[0152] Here, the specific process of extracting the feature map of the sample image through the key point prediction model to be trained can refer to step 102 in the above embodiment, and will not be repeated.

[0153] Step 206: The server determines first predicted position information of the sample key point and first predicted offset information of the target pixel region where the sample key point is located in the feature map.

[0154] Here, the specific process of determining the first predicted position information of the sample key point and the first predicted offset information of the target pixel area where the sample key point is located in the feature map can refer to step 103 in the above embodiment, and will not be repeated.

[0155] Step 207: The server determines a model loss value based on the first sample position information, the first predicted position information, the first sample offset information of the sample key point in the target pixel area, and the first predicted offset information.

[0156] Here, the specific process of determining the model loss value based on the first sample position information, the first predicted position information, the first sample offset information of the sample key point in the target pixel area and the first predicted offset information can refer to step 104 in the above embodiment and will not be repeated.

[0157] In step 208 , the server updates the model parameters of the key point prediction model to be trained based on the model loss value to obtain a trained key point prediction model.

[0158] Here, based on the model loss value, the model parameters of the key point prediction model to be trained are updated. The specific process of obtaining the trained key point prediction model can refer to step 105 in the above embodiment, and will not be repeated.

[0159] In the embodiment of the present application, a server calculates a more accurate model loss value based on the first sample position information, the first predicted position information, the first sample offset information and the first predicted offset information, and then uses the model loss value to train the key point prediction model, which can improve the detection accuracy and stability of the key point prediction model.

[0160] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0161] The embodiment of the present application proposes a key point stable prediction method based on key point neighborhood constraints, which is a method for predicting key points based on regression, adding key point offset (self_offset) constraints and neighbor key point offset (neighborhood_offset) constraints to the last feature map of the key point prediction model to assist in more accurate generation of key point positions. In the regression method, regression loss (wingloss) can better capture the small errors of key points, so regression loss (wingloss) is used to train the key point prediction model. In addition, on the basis of regression loss (wingloss), the distance constraint of neighbor key points (neighborhood) is introduced to guide the key point prediction model to learn global capabilities.

[0162] Fig.10Schematic diagram of the structure of the basic model of the key point prediction model proposed in the embodiment of the present application. Fig.10 The key point prediction model consists of multiple convolutional layers (convolutional layer 301 and convolutional layer 305), a global maximum pooling layer 302 (Group Max Pooling, GMP), and two fully connected layers (feature fully connected layer 303, result fully connected layer 304). The global maximum pooling layer is used to reduce the spatial dimension of the feature map while retaining important features. The sample position information (ground-truth) refers to the coordinate data of the real face key point position. There are 98 key points in the face. Table 1 is the operation flow of the key point prediction model.

[0163] Table 1 Operational flow of key point prediction model

[0164]

[0165]

[0166] See Table 1, conv3×3 is a convolution operation, the bottleneck layer (Bottleneck) consists of multiple convolution layers, t represents the transposition magnification factor in the bottleneck, t=2 means that the number of channels is first enlarged to 64×2=128, and then reduced back to 64 at the output, Linear is the mapping operation in the fully connected layer, c represents the number of convolution kernel channels, n represents the number of times the operation is repeated, and s represents the side length. First, the input face image is 112×112×3 (width, height, number of channels), and then Fig.10 The first convolutional layer 301 in the image is processed to obtain a 56×56×64 feature map. Fig.10 After the stage 1 processing (stage 1 includes the Bottleneck operation in Table 1), a 28×28×64 feature map is obtained. After the stage 2 processing and convolution processing, a 7×7×32 feature map is obtained, and the 7×7×32 feature map is input into the global maximum pooling layer 302 to obtain a 32-bit feature vector, and finally 196 coordinate values ​​(corresponding to the first predicted position information in the above embodiment) are obtained after the fully connected layer.

[0167] See also Figure 4 During the training process, the 7×7×32 feature map is convolved again to obtain a 7×7×98 feature map (corresponding to the feature map in the above embodiment), a 7×7×196 offset feature map, and a 7×7×1960 neighbor offset feature map. Among them, the 7×7×98 feature map is responsible for the prediction of 98 key points, that is, each channel predicts one key point. Fig.11 It is a schematic diagram of multiple feature graphs provided in an embodiment of the present application. Fig.11(a) in the figure is a 7×7×98 feature map (score_map), including 7×7=49 pixel regions, which can calculate whether the key point falls in the pixel region. If the value (corresponding to the first prediction score in the above embodiment) is 1 or close to 1, the key point falls in the pixel region. The 7×7×196 offset feature map is used to predict the offset (offset) of each key point on the x-axis and y-axis (corresponding to the first prediction offset information in the above embodiment). Fig.11 (b) in the figure is an offset feature map (x-offset_map) of 7×7×196 in the x-axis direction. The offset of the key point in the x-axis is 0.4. Since the size of the offset feature map is 7×7 and the original face image is 112×112, there are 16 pixels in one pixel area in the feature map. Assuming the upper left corner vertex of the pixel area is (0, 0), 16×0.4=6.4, it is offset to the right by about 6 coordinate points. Fig.11 (c) in the figure is a 7×7×196 offset feature map (y-offset_map) in the y-axis direction, and the y-axis offset is calculated in the same way as above. The 7×7×1960 neighbor offset feature map is used to predict the offsets (offsets) of the 10 nearest neighbor key points of the key point on the x-axis and y-axis (corresponding to the second predicted offset information in the above embodiment). For any key point, the 10 points closest to the remaining 97 key points are selected as neighbor key points. The neighbor key points are determined by the distance calculated from the true coordinate values. Since neighbor key points are introduced in the prediction process, the predicted key points can be more stable, which can better reduce false detections in lip movement recognition and speaking scenarios.

[0168] The score map loss is w (corresponding to the first loss value in the above embodiment), and the offset feature map loss is L self-off (corresponding to the fourth loss value in the above embodiment), the neighborhood offset feature map (neighborhood-offset_map) loss is L n-off (corresponding to the fifth loss value in the above embodiment). s The above formula (1) can be satisfied. Offset feature map loss L self-off The above formula (2) can be satisfied, and the neighbor offset feature map loss L n-off The above formula (3) can be satisfied. Feature map loss L s The loss is calculated for the entire feature map, while the offset feature map loss L self-off and neighbor offset feature map loss L n-off Only in When , that is, the loss is calculated in the pixel area where the key point is actually located. In the formula, gt represents the groundtruth, pred represents the network prediction, and i represents the channel, corresponding to the index of the key point. s , offset feature map loss L self-off and neighbor offset feature map loss L n-off The weighted sum is performed to obtain the spatial guide loss (Spatial_guide_loss). The spatial guide loss satisfies the following formula (7).

[0169] L spatial_loss =L s +ω 1 L self-off +ω 2 L n-off Formula (7);

[0170] Among them, L spatial_loss is the spatial guidance loss, ω 1 and ω 2 All are hyperparameters (weight parameters).

[0171] In order to make the constraint loss of the feature map learn better, the distance constraint of the neighbor key points is further introduced as an auxiliary constraint. For each key point, its 10 nearest neighbor key points are recorded, and the distance from the key point to the 10 neighbor key points is introduced as an additional constraint. The distance satisfies the following formula (8).

[0172] Dist(P,N)=||P i -P N || Formula (8);

[0173] Among them, Dist(P,N) is the distance between the key point P and the neighbor key point N, P i is the i-th key point, P N is the Nth neighbor key point of the ith key point. The neighbor distance loss (corresponding to the sixth loss value in the above embodiment) can satisfy the above formula (4). The neighbor distance loss can help the overall loss value to be better reduced by guiding the rapid learning of the feature graph, so that the overall effect can be achieved more accurately and more stably. The total model loss value L total The following formula (9) can be satisfied.

[0174] L total =wingloss+L spatial_loss +L nb Formula (9);

[0175] Among them, L nbis the neighbor distance loss, wingloss is the regression loss (corresponding to the seventh loss value in the above embodiment). The regression loss can satisfy the above formulas (5) and (6).

[0176] The embodiment of the present application designs a stable and accurate key point prediction model optimization strategy for the edge platform, which improves the accuracy of the key point algorithm without increasing the complexity of the model, greatly improving the accuracy of the model's prediction of key points. In addition, the embodiment of the present application can also provide ideas for reference in other fields, and make the network learning more targeted through auxiliary information supervision, so as to have a higher accuracy rate and help the lightweight implementation of the model.

[0177] It is understandable that in the embodiments of the present application, related data such as facial images are involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0178] The following is a description of an exemplary structure of a key point prediction model training device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the key point prediction model training device 455 of the memory 450 may include: a sample acquisition module 4551, used to acquire sample key points in a sample image and first sample position information of the sample key points; a feature map extraction module 4552, used to extract the feature map of the sample image through the key point prediction model to be trained; a prediction module 4553, used to determine the first prediction position information of the sample key points, and the first prediction offset information of the target pixel area where the sample key points are located in the feature map; a loss determination module 4554, used to determine the model loss value based on the first sample position information, the first prediction position information, the first sample offset information of the sample key points in the target pixel area and the first prediction offset information; a model training module 4555, used to update the model parameters of the key point prediction model to be trained based on the model loss value to obtain the trained key point prediction model.

[0179] In some embodiments, the loss determination module 4554 is also used to determine a first loss value of the key point prediction model based on the first sample position information; determine a sample neighbor key point of the sample key point based on the first sample position information; determine a second loss value of the key point prediction model based on the first sample offset information, the first predicted offset information, the second sample offset information of the sample neighbor key point in the target pixel area, and the second predicted offset information of the sample neighbor key point in the target pixel area; determine a third loss value of the key point prediction model based on the first sample position information, the first predicted position information, the second sample position information of the sample neighbor key point, and the second predicted position information of the sample neighbor key point; determine a model loss value based on the first loss value, the second loss value, and the third loss value.

[0180] In some embodiments, the loss determination module 4554 is also used to determine a first label score when the sample key point is located in a pixel area in the feature map based on the first sample position information; perform feature mapping processing on the feature map to obtain a first prediction score when the sample key point is located in the pixel area; perform feature map loss calculation on the first prediction score and the first label score to obtain a first loss value.

[0181] In some embodiments, the loss determination module 4554 is also used to determine that the first label score is a first preset score when it is determined that the sample key point is located within the pixel area based on the first sample position information; and to determine that the first label score is a second preset score when it is determined that the sample key point is located outside the pixel area based on the first sample position information.

[0182] In some embodiments, the loss determination module 4554 is also used to perform a first offset loss calculation on the first prediction offset information and the first sample offset information to obtain a fourth loss value; perform a second offset loss calculation on the second prediction offset information and the second sample offset information to obtain a fifth loss value; and perform loss value fusion on the fourth loss value and the fifth loss value to obtain a second loss value.

[0183] In some embodiments, the loss determination module 4554 is also used to determine the first predicted distance between the sample key point and the sample neighbor key point based on the first predicted position information and the second predicted position information; determine the first sample distance between the sample key point and the sample neighbor key point based on the first sample position information and the second sample position information; perform distance loss calculation on the first predicted distance and the first sample distance to obtain a sixth loss value; perform position loss calculation on the first sample position information and the first predicted position information to obtain a seventh loss value; perform loss value fusion on the sixth loss value and the seventh loss value to obtain a third loss value.

[0184] In some embodiments, the loss determination module 4554 is also used to determine the position information error between the first sample position information and the first predicted position information; when the position information error is less than the preset position error, the first parameter is determined based on the preset parameter and the position information error, and the product of the preset position error and the first parameter is determined as the seventh loss value of the sample key point.

[0185] In some embodiments, the loss determination module 4554 is also used to determine the position information error between the first sample position information and the first predicted position information; when the position information error is greater than or equal to the preset position error, determine the second parameter based on the preset parameter and the position information error, and determine the product of the preset position error and the second parameter; determine the difference between the preset position error and the product as the first difference; and determine the second difference between the position error and the first difference as the seventh loss value.

[0186] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the key point prediction model training method described in the embodiment of the present application.

[0187] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the key point prediction model training method provided by the present application embodiment, for example, Figure 3 The key point prediction model training method is shown.

[0188] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0189] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0190] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0191] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0192] In summary, the embodiments of the present application have designed a stable and accurate key point prediction model optimization strategy for the edge platform, which enables the key point algorithm to improve accuracy without increasing the complexity of the model, greatly improving the accuracy of the model's prediction of key points. In addition, the embodiments of the present application can also provide ideas for reference in other fields, and make network learning more targeted through auxiliary information supervision, thereby having a higher accuracy rate and helping the lightweight implementation of the model.

[0193] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A key point prediction model training method, characterized in that: The method comprises: Acquire sample key points in a sample image and first sample position information of the sample key points; Extracting a feature map of the sample image by using a key point prediction model to be trained; Determine first predicted position information of the sample key point and first predicted offset information of a target pixel region where the sample key point is located in the feature map; Determine a model loss value based on the first sample position information, the first predicted position information, the first sample offset information of the sample key point in the target pixel area, and the first predicted offset information; Based on the model loss value, the model parameters of the key point prediction model to be trained are updated to obtain a trained key point prediction model.

2. The method according to claim 1, characterized in that The determining of the model loss value based on the first sample position information, the first predicted position information, the first sample offset information of the sample key point in the target pixel area, and the first predicted offset information includes: Determining a first loss value of the key point prediction model based on the first sample position information; Determine a sample neighbor key point of the sample key point based on the first sample position information; Determining a second loss value of the key point prediction model based on the first sample offset information, the first prediction offset information, the second sample offset information of the sample neighbor key point in the target pixel area, and the second prediction offset information of the sample neighbor key point in the target pixel area; Determining a third loss value of the key point prediction model based on the first sample position information, the first predicted position information, the second sample position information of the sample neighbor key point, and the second predicted position information of the sample neighbor key point; A model loss value is determined based on the first loss value, the second loss value, and the third loss value.

3. The method according to claim 2, characterized in that The determining, based on the first sample position information, a first loss value of the key point prediction model includes: Determine, based on the first sample position information, a first label score in which the sample key point is located within a pixel area in the feature map; Performing feature mapping processing on the feature map to obtain a first prediction score where the sample key point is located in the pixel area; A feature map loss calculation is performed on the first prediction score and the first label score to obtain the first loss value.

4. The method according to claim 3, characterized in that The step of determining, based on the first sample position information, a first label score where the sample key point is located in a pixel area in the feature map comprises: When it is determined according to the first sample position information that the sample key point is located in the pixel area, determining the first label score to be a first preset score; When it is determined that the sample key point is outside the pixel area according to the first sample position information, the first label score is determined to be a second preset score.

5. The method according to claim 2, characterized in that: The determining a second loss value of the key point prediction model based on the first sample offset information, the first prediction offset information, the second sample offset information of the sample neighbor key point in the target pixel area, and the second prediction offset information of the sample neighbor key point in the target pixel area includes: Performing a first offset loss calculation on the first prediction offset information and the first sample offset information to obtain a fourth loss value; Performing a second offset loss calculation on the second prediction offset information and the second sample offset information to obtain a fifth loss value; The fourth loss value and the fifth loss value are fused to obtain the second loss value.

6. The method according to claim 2, characterized in that The determining, based on the first sample position information, the first predicted position information, the second sample position information of the sample neighbor key point, and the second predicted position information of the sample neighbor key point, a third loss value of the key point prediction model comprises: Determine a first predicted distance between the sample key point and the sample neighbor key point based on the first predicted position information and the second predicted position information; Determine a first sample distance between the sample key point and the sample neighbor key point based on the first sample position information and the second sample position information; Performing distance loss calculation on the first predicted distance and the first sample distance to obtain a sixth loss value; Performing position loss calculation on the first sample position information and the first predicted position information to obtain a seventh loss value; The sixth loss value and the seventh loss value are fused to obtain the third loss value.

7. The method according to claim 6, characterized in that The performing position loss calculation based on the first sample position information and the first predicted position information to obtain a seventh loss value includes: Determining a position information error between the first sample position information and the first predicted position information; When the position information error is less than a preset position error, a first parameter is determined based on a preset parameter and the position information error, and the product of the preset position error and the first parameter is determined as the seventh loss value of the sample key point.

8. The method according to claim 6, characterized in that The performing position loss calculation based on the first sample position information and the first predicted position information to obtain a seventh loss value includes: Determine a position information error between the first sample position information and the first predicted position information; In a case where the position information error is greater than or equal to a preset position error, determining a second parameter based on a preset parameter and the position information error, and determining a product of the preset position error and the second parameter; Determine a difference between the preset position error and the product as a first difference; A second difference between the position error and the first difference is determined as the seventh loss value.

9. A key point prediction model training device, characterized in that: The device comprises: A sample acquisition module, used to acquire sample key points in a sample image and first sample position information of the sample key points; A feature map extraction module, used to extract a feature map of the sample image through a key point prediction model to be trained; A prediction module, used to determine first predicted position information of the sample key point and first predicted offset information of a target pixel area where the sample key point is located in the feature map; a loss determination module, configured to determine a model loss value based on the first sample position information, the first predicted position information, the first sample offset information of the sample key point in the target pixel area, and the first predicted offset information; The model training module is used to update the model parameters of the key point prediction model to be trained based on the model loss value to obtain the trained key point prediction model.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor is used to implement the key point prediction model training method described in any one of claims 1 to 8 when executing the computer executable instructions or computer program stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the key point prediction model training method described in any one of claims 1 to 8 is implemented.

Images