Model training method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a model training method and device, electronic equipment, a computer program product and a computer readable storage medium, and the method comprises the steps: carrying out the key point prediction of a sample object, and obtaining a first bounding box of the sample object and a first key point located in the first bounding box; based on the first bounding box, adjusting a second key point in a second bounding box of the sample object to obtain a third key point; determining a loss value based on the first key point and the third key point; and training the first model based on the loss value. According to the invention, the accuracy of key point prediction by the model can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a model training method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] In the field of computer vision, key point prediction technology is widely used in many fields such as posture estimation, motion analysis, virtual reality, human-computer interaction, and face recognition. For example, in face recognition, the model used for key point prediction needs to predict multiple key points of the face, such as eyes, nose, mouth, and cheeks, which can then be used for feature extraction for face verification or recognition. In other application scenarios, the model used for key point prediction may also need to predict specific marker points of human posture or objects.

[0003] In the related art, the key point prediction model usually directly uses linear regression to predict the key points. However, the linear regression method makes it difficult for the loss function to converge in the early stage of training, thus affecting the prediction accuracy of the model. Summary of the invention

[0004] The embodiments of the present application provide a model training method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of model prediction of key points.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application embodiment provides a model training method, the method comprising:

[0007] Perform key point prediction on the sample object to obtain a first bounding box of the sample object and a first key point located in the first bounding box; based on the first bounding box, adjust the second key point in the second bounding box of the sample object to obtain a third key point; determine a loss value based on the first key point and the third key point; and train the first model based on the loss value.

[0008] The present application embodiment provides a model training device, including:

[0009] A prediction module, configured to perform key point prediction on a sample object to obtain a first bounding box of the sample object and a first key point located in the first bounding box;

[0010] an adjustment module, configured to adjust the second key point in the second bounding box of the sample object based on the first bounding box to obtain a third key point;

[0011] A loss determination module, configured to determine a loss value based on the first key point and the third key point;

[0012] A training module is used to train the first model based on the loss value.

[0013] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0014] A memory for storing computer executable instructions or computer programs;

[0015] The processor is used to implement the model training method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the training method of the model provided in the embodiment of the present application when executed by a processor.

[0017] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the training method of the model provided in the embodiment of the present application is implemented.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] By applying the above embodiment, the key point prediction is performed on the sample object, and after obtaining the first bounding box and the first key point predicted for the sample object, the second bounding box of the sample object is adjusted to the first bounding box to obtain the third key point for which the first key point can be used as a position reference. The loss value of the model is determined based on the first key point and the third key point, and the first model is back-propagated using the loss value to achieve training of the first model. In the above manner, when training the first model, the linear regression method in the related art is no longer used to perform position regression on the predicted first key point. Instead, the third key point determined by the pre-marked second key point is used as the regression reference point of the first key point to achieve position regression of the first key point. This method reduces the situation in which the loss is difficult to converge due to gradient explosion or disappearance caused by linear regression in the early stage of the training of the first model, making the initial stage of the training of the first model more stable, thereby improving the accuracy of the first model in predicting key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a structural diagram of the training system architecture of the model provided in the embodiment of the present application;

[0021] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0022] Figure 3A It is a first flow chart of the training method of the model provided in the embodiment of the present application;

[0023] Figure 3B It is a second flow chart of the training method of the model provided in the embodiment of the present application;

[0024] Figure 3C is a flow chart of a method for determining a third key point provided in an embodiment of the present application;

[0025] Figure 3D It is a flowchart of a method for determining a loss value provided in an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a sample object provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a bounding box and key points of a sample object provided in an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of key point regression provided in an embodiment of the present application;

[0029] Figure 7 This is a third flow chart of the model training method provided in the embodiment of the present application.

[0030] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0035] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0037] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0038] 1) Bounding box refers to the smallest rectangular box that encloses the boundary of the sample object. The bounding box usually includes four vertices of the sample object, namely the upper left vertex, the upper right vertex, the lower right vertex, and the lower left vertex. These vertices define the position and size of the sample object in the image sample.

[0039] 2) The loss value is an indicator for evaluating the difference between the prediction result of the first model and the actual result. The smaller the loss value, the closer the prediction result of the first model is to the actual result, and the better the performance of the first model. The loss value of the first model is obtained by forward propagation and back propagation calculation of the first model using the real data set during the training process. During the training process, the optimizer adjusts the parameters of the first model according to the loss value to reduce the loss value and improve the performance of the first model until the training end condition is reached to obtain the second model. Here, the first model is used to predict the key points of sample objects in image samples.

[0040] In the field of computer vision, key point prediction technology is widely used in many fields such as posture estimation, motion analysis, virtual reality, human-computer interaction, and face recognition. A typical application example of key point prediction technology is in face recognition, where the model used for key point prediction needs to predict multiple key points of the face, such as eyes, nose, mouth, and cheeks, which can then be used for feature extraction for face verification or recognition. In other application scenarios, the model used for key point prediction may also need to predict specific marker points of human posture or objects.

[0041] Taking face recognition as an example, face recognition technology will be applied to daily life applications such as face unlocking, face access control, sign-in, and face payment, ID card comparison and other security authentication scenarios. In application scenarios related to face recognition, face prediction and face key point prediction are the basis of many applications. For example, when performing deep fake detection, it is usually necessary to predict the face of the input image, and then align the face according to some key points (five points) of the face. For example, in action liveness verification, face prediction is also required first, and then face key points (usually multiple key points) are predicted. Before performing key point prediction, it is often necessary to perform rotation correction on the face, and before performing rotation correction, the rotation angle of the face must be known, and the rotation angle of the face can often be roughly calculated based on the five key points of the face. In short, in application scenarios related to face authentication, face prediction and face key point prediction (usually five key points: left eye, right eye, nose, left lip corner, right lip corner) are basically inseparable.

[0042] In the related art, when models related to key point prediction are used to predict key points, linear regression is usually used directly to achieve this. However, the linear regression method usually has the following problems: linear regression is prone to fall into local minima, especially when the feature space is complex, which may affect the global optimal solution of model learning; in the initial training stage, since the gradient of the feature extraction network for key point prediction may be very small or very large, this will lead to gradient vanishing or explosion problems, making it difficult for the loss function to converge; linear regression assumes that there is a linear relationship between data, but the distribution of facial key points may not be linear. These problems with linear regression will lead to unsatisfactory training results, especially when the faces in the image are of different sizes, the key point prediction effect for small faces in the image is even worse. Therefore, in order to improve the prediction accuracy of the model used for key point prediction for key points, the present application proposes a model training method.

[0043] The embodiments of the present application provide a model training method, device, electronic device, computer-readable storage medium and computer program product, which can improve the accuracy of model prediction of key points.

[0044] See also Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the training system 100 of the model provided in the embodiment of the present application. To realize the training application of a model, the terminal 401 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. Among them, the first model for key point prediction is deployed in the server 200.

[0045] The terminal 401 is used to respond to the training instruction of the first model and send a training request of the first model to the server 200. The server 200 is used to predict key points of the sample object in response to the training request for the first model, obtain a first bounding box of the sample object and a first key point located in the first bounding box; based on the first bounding box, adjust the second key point in the second bounding box of the sample object to obtain a third key point; based on the first key point and the third key point, determine the loss value; and train the first model based on the loss value.

[0046] In some embodiments, the terminal 401 can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, a vehicle-mounted terminal, etc., and can also be implemented as a server. Below, an exemplary application when the device is implemented as a terminal will be described.

[0047] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 401 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0048] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present application, Figure 2 The electronic device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0049] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0050] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0051] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0052] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0053] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0054] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0055] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0056] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0057] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0058] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The training device 455 of the model stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a prediction module 4551, an adjustment module 4552, a loss determination module 4553, and a training module 4554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0059] In other embodiments, the training device of the model provided in the embodiments of the present application can be implemented in hardware. As an example, the training device of the model provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0060] Below, the training method of the model provided by the embodiment of the present application will be described in conjunction with the accompanying drawings. As mentioned above, the electronic device that implements the training method of the model of the embodiment of the present application can be a terminal 401, a server 200, or a combination of the two. The training method of the model of the embodiment of the present application is described by taking the execution subject as the server 200 as an example. The execution subject of each step will not be repeated below.

[0061] It should be noted that in the example of model training below, the sample object is a human face. Based on the understanding of the following text, those skilled in the art can apply the model training method provided in the embodiment of the present application to the processing of image sets including other types of objects, such as key point prediction for animal images.

[0062] See also Figure 3A , Figure 3A is a flow chart of the training method of the model provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0063] In step 101, key point prediction is performed on a sample object to obtain a first bounding box of the sample object and a first key point located in the first bounding box.

[0064] In an embodiment of the present application, a key point prediction is performed on a sample object through a first model to obtain a first bounding box of the sample object and a first key point located in the first bounding box. The first model is an initial prediction model constructed for key point prediction. By training the first model, a second model for key point prediction can be obtained. The input of the first model is an image sample, which includes at least one sample object to be predicted, and the sample object includes specific key points to be predicted. It can be understood that the key points corresponding to different sample objects are usually different. For example, when the sample object is a human face, the key points of the sample object usually include five key points: the left eye, the right eye, the nose, the left lip corner, and the right lip corner. For another example, when the sample object is an animal (such as a dog), the key points of the sample object usually include key points such as the left eye, the right eye, the nose, the mouth, and the tail. Next, the training method of the model provided in the embodiment of the present application will be described by taking the sample object as a human face as an example.

[0065] In actual implementation, the image sample is input into the first model, and the first model can predict key points of the sample object in the image sample, and predict a first bounding box of the sample object and a first key point located in the first bounding box.

[0066] In an embodiment of the present application, the bounding box of the sample object is the minimum rectangular box surrounding the sample object, wherein the first bounding box is the bounding box predicted by the first model for the sample object. When predicting the first bounding box, the first model also predicts the center point of the first bounding box, that is, the intersection position of the diagonal lines of the first bounding box, and records the first bounding box in the form of (x, y, w, h), where (x, y) represents the coordinates of the center point of the bounding box in the image sample, w represents the width of the bounding box, and h represents the height of the bounding box.

[0067] For example, see Figure 4The image sample includes a sample object, namely, a face object 410. The first model predicts key points for the face object 410, and obtains a first bounding box 4101 corresponding to the face object 410, and five first key points in the face object 410. The first key points include key point P1 (corresponding to the left eye), key point P2 (corresponding to the right eye), key point P3 (corresponding to the tip of the nose), key point P4 (corresponding to the left lip corner), and key point P5 (corresponding to the right lip corner).

[0068] In the embodiment of the present application, for the image sample used to train the first model, the sample object in the image sample is pre-labeled with a bounding box and key points, the labeled bounding box is recorded as the second bounding box, and the labeled key point is recorded as the second key point. The second bounding box labeled for the sample object is a standard bounding box at a standard size, and the second key point labeled for the sample object is a standard key point at a standard size. The position of the second key point can be understood as the position of a standard face at a standard size.

[0069] Among them, the Caucasian standard face refers to an artificial, idealized "typical" face image constructed in the field of computer vision and image processing to test and evaluate the performance of face recognition algorithms. This concept is based on an idealized assumption about facial features, that is, it is believed that there is a "standard" face morphology.

[0070] In some embodiments, see Figure 3B Before executing step 101, the sample objects may be labeled through steps 105 to 108.

[0071] In step 105, a third bounding box of the initially labeled sample object and a fourth key point located in the third bounding box are determined.

[0072] In the embodiment of the present application, since the input image size of the initial prediction model is a uniform size, before labeling the sample objects of each image sample in the image sample set, the image samples in the image sample set may be uniformly sized, so that the image samples are adjusted from the original size to the same size or resolution by scaling, so that the image samples meet the input image size of the first model. The image sample set includes multiple (greater than or equal to 2) image samples, and each image sample includes at least one sample object.

[0073] In actual implementation, after unifying the size of the image samples, for each image sample in the image sample set, the sample object in the image sample is initially labeled based on the current size of the image sample, and the third bounding box initially labeled for the sample object and the fourth key point located in the third bounding box are obtained. Here, the width and height of the third bounding box and the coordinates of the center point of the third bounding box are determined based on the unified size of the image samples, and the coordinates of the center point can be the pixel coordinates of the center point in the image sample. The position of the initially labeled fourth key point is the pixel position under the unified size of the image sample.

[0074] In another feasible manner, the bounding boxes and key points of the sample objects in the image samples may be initially annotated before the sizes of the image samples are unified. Then, the image samples may be scaled to the uniform size of the first model, and the scaled bounding boxes may be used as the third bounding boxes, and the scaled key points may be used as the fourth key points.

[0075] It should be noted that the scaling mentioned in the embodiments of the present application means that when the bounding box of the sample object is scaled from size A to size B, all key points in the bounding box will be scaled at the same scaling ratio as the bounding box to obtain the position of the key points in the scaled bounding box, that is, the positional relationship of the key points in the scaled bounding box is similar to the positional relationship in the original bounding box, except that the overall size is scaled.

[0076] In step 106, the third bounding box and the fourth key point are scaled based on a first length of a diagonal line of the third bounding box to obtain a second bounding box and a fifth key point located in the second bounding box.

[0077] In actual implementation, since different sample objects in image samples have different sizes and sample objects in different image samples also have different sizes, in order to reduce the interference caused by the different sizes of sample objects when the first model determines the key point position, the sizes of sample objects in the image samples can also be unified.

[0078] As an example, a first length for scaling the diagonal of the third bounding box of each sample object may be predetermined, and the third bounding boxes of all sample objects may be scaled with reference to the first length, so that the second length of the diagonal of the third bounding boxes of all sample objects is equal to the first length, and the second bounding box of the sample object is obtained, and the diagonal length of the second bounding box of the sample object is equal to the first length. Here, while scaling the third bounding box of the sample object, the fourth key point in the third bounding box is also scaled at the same proportion, and the fifth key point located in the second bounding box is obtained.

[0079] In some embodiments, the first length may be determined in the following manner.

[0080] If there are multiple sample objects, the second length of the diagonal of the third bounding box of each sample object is determined; and the second lengths of the third bounding boxes of the multiple sample objects are averaged to obtain the first length.

[0081] It should be noted that “plurality” refers to a number greater than or equal to two.

[0082] In actual implementation, the image sample set used to train the first model usually contains a large number of image samples, so the number of sample objects that need to be labeled is usually also multiple. If it is determined that the number of sample objects in the image sample set is N, and N is an integer greater than or equal to 2, the average of the second lengths of the third bounding boxes initially labeled by the N sample objects can be used as the first length.

[0083] As an example, for each sample object, the second length of the diagonal of the third bounding box initially annotated for the sample object is determined to obtain the second length d of the sample object. Then, N sample objects correspond to N second lengths, and the N second lengths are recorded as {d1, d2, d3…dn}. The N second lengths can be averaged (i.e., the mean) by the following formula to obtain the first length d′.

[0084]

[0085] As an example, after the first length is calculated, taking sample object 1 as an example, the scaling ratio is determined according to the ratio of the second length d1 of the third bounding box of sample object 1 to the first length d′, and based on the scaling ratio, the third bounding box of sample object 1 and the fourth key point in the third bounding box are scaled until the second length of the bounding box of sample object 1 is equal to the first length d′, thereby obtaining the second bounding box of sample object 1 and the fifth key point in the second bounding box. The same can be said for other sample objects, which will not be described in detail.

[0086] In step 107, based on the first position of the key point of the sample object in the second bounding box, the second position of the fifth key point in the second bounding box is adjusted to obtain a second key point.

[0087] In the embodiment of the present application, since the fifth key point is the corresponding position of the fourth key point after scaling with the third bounding box, and the fourth key point can be manually annotated or predicted by other prediction models, the position of the fourth key point may also have a certain deviation, especially when the face is used as the sample object, due to the changeable expression and the deviation of the display angle of the face on the image sample, the position of the annotated fourth key point will be biased. If the fourth key point is directly used to train the first model, the training result may not be very ideal.

[0088] Therefore, in order to reduce the impact of individual differences of sample objects on model training, taking human faces as sample objects, we can simulate and find the corresponding positions of the five key points corresponding to the class standard human face in the second bounding box as the first position for reference. Through the first position, the second position of the fifth key point of the sample object in the second bounding box is adjusted to obtain the second key point of the sample object, and the position of the second key point is the first position.

[0089] In some embodiments, the first position may be determined in the following manner.

[0090] If there are multiple sample objects, determine the horizontal distance and vertical distance between the fifth key point of each sample object and the center point of the second bounding box; average the multiple horizontal distances to obtain the average horizontal distance, and average the multiple vertical distances to obtain the average vertical distance; determine the first position based on the average horizontal distance and the average vertical distance.

[0091] In actual implementation, the position of the fourth key point initially marked for the sample object is the pixel coordinate of the key point in the image sample, and correspondingly, the position of the fifth key point to be determined is also the pixel coordinate position, but the positions of different sample objects in the image sample are different, and the first position to be determined is the key point position of the class standard human in the second bounding box of each sample object, so it is not reasonable to use the pixel coordinate position of the fifth key point to determine the first position. Here, the position of the fifth key point of the sample object can be converted from the pixel coordinate position to the offset position relative to the center point of the second bounding box, and then the first position is determined based on the offset position of the fifth key point.

[0092] As an example, Figure 5 A schematic diagram of a bounding box and key points of a sample object provided in an embodiment of the present application. Figure 5, the bounding box 510 is the second bounding box of the sample object, and there are five fifth key points of the sample object (key point P1, key point P2, key point P3, key point P4, key point P5) in the bounding box 510. The second bounding box is recorded in the form of (x, y, w, h), where (x, y) is the pixel coordinate position of the center point C of the second bounding box in the image sample. Taking the key point P1 as an example, if the pixel coordinate position of the key point P1 is (x1, y1), the horizontal distance dx of the key point P1 relative to the center point C of the second bounding box can be determined based on the difference between the coordinate x of the center point C of the second bounding box in the horizontal direction and the coordinate x1 of the key point P1 in the horizontal direction; the vertical distance dy of the key point P1 relative to the center point C of the second bounding box is determined based on the difference between the coordinate y of the center point C of the second bounding box in the vertical direction and the coordinate y1 of the key point P1 in the vertical direction, thereby obtaining the offset position (dx, dy) from the key point P1 to the center point of the second bounding box. The other key points can be proved in the same way and will not be repeated.

[0093] Since the positions of the facial features of a person are fixed, especially for people of the same race, the offsets of their facial features relative to the center are roughly similar. Therefore, in the embodiment of the present application, the first position of each key point can be determined by using the average offset of the same key point of several sample objects relative to the center point of the second bounding box.

[0094] As an example, if there are multiple sample objects, for example, N, where N is a positive integer greater than or equal to 2, and the N sample objects all have the same number of fifth key points, then for each sample object, the horizontal distance and vertical distance between the fifth key point corresponding to the same position (e.g., the left eye) and the center point of the second bounding box are determined, and N horizontal distances and N vertical distances between the fifth key point corresponding to the left eye position of the N sample objects and the center point of the second bounding box are obtained. The N horizontal distances are averaged to obtain the average horizontal distance; the N vertical distances are averaged to obtain the average vertical distance. The average horizontal distance and the average vertical distance are used as the first position of the fifth key point corresponding to the left eye position of the N sample objects. The first position of the fifth key point of each sample object at other positions can be verified in the same way and will not be repeated.

[0095] As an example, continue to see Figure 5 , the first position determined by the fifth key point (key point P1) corresponding to the left eye position of each sample object is recorded as (dx1, dy1), and the first position determined by the fifth key point (key point P2) corresponding to the right eye position is recorded as (dx2, dy2). The same is true for the others, and the first position corresponding to each fifth key point in the second bounding box is obtained.

[0096] In some embodiments, step 107 may be implemented in the following manner.

[0097] Based on the first position in the second bounding box and the second position of the fifth key point in the second bounding box, determine the adjustment direction and adjustment distance for the fifth key point; based on the adjustment direction and adjustment distance, adjust the second position of the fifth key point in the second bounding box to obtain the second key point in the second bounding box, and the second key point is located at the first position in the second bounding box.

[0098] As an example, for each fifth key point of each sample object, based on the offset position (dx, dy) between the fifth key point and the center point of the second bounding box in the second bounding box, and the first position determined for the fifth key point, the adjustment direction and adjustment distance of the fifth key point relative to the first position are determined. The adjustment direction is determined based on the orientation of the first position relative to the fifth key point; the adjustment distance includes the adjustment amount in the horizontal direction and the adjustment amount in the vertical direction when the fifth key point moves toward the first position.

[0099] As an example, if the offset position (dx, dy) of the fifth key point relative to the center point of the second bounding box is (-5, 4), and the offset position (dx1, dy1) of the first position corresponding to the fifth key point relative to the center point of the second bounding box is (-4, 4), then it can be determined that the adjustment direction of the fifth key point relative to the first position is the horizontal positive direction, and the adjustment distance can be expressed as (1, 0). Based on the adjustment direction and the adjustment distance, the second position of the fifth key point in the second bounding box can be adjusted by 1 pixel in the horizontal positive direction to obtain the second key point. The position of the second key point is located at the first position.

[0100] In step 108, the sample object is re-labeled based on the second bounding box and the second key points to obtain a sample object labeled with the second bounding box and the second key points.

[0101] In actual implementation, after determining the second bounding box of each sample object and the second key point located in the second bounding box through the above steps, the sample objects are re-labeled, and the corresponding second bounding box and second key point of each sample object are labeled by adding labels. The label of the second bounding box can be expressed in the form of (x, y, w, h), and the label of the second key point can include the pixel coordinate position of the second key point in the image sample and the offset position of the second key point relative to the center point of the second bounding box.

[0102] Continue reading Figure 3A , and the description will continue with the above step 101.

[0103] In step 102, based on the first bounding box, the second key point in the second bounding box of the sample object is adjusted to obtain a third key point.

[0104] Here, the first bounding box of the sample object is predicted by the first model, and the size of the first bounding box corresponds to the size of the image sample where the sample object is located, while the second bounding box is a bounding box annotated for the sample object, and the size of the second bounding box of each sample object is unified, and the length of the diagonal of each second bounding box is equal to the first length. It can be seen that the size of the first bounding box and the second bounding box annotated for the sample object are not necessarily the same. Therefore, when training the first model, the size and position of the second bounding box of the sample object need to be unified with the first bounding box in order to obtain a reference point (third key point) for position regression of the first key point.

[0105] In the embodiment of the present application, the purpose of marking the second bounding box and the second key point for the sample object is to make the key point position predicted by the first model more accurate. Therefore, when training the first model, what the present application ultimately needs to determine is the actual position of the first key point corresponding to each sample object in the predicted first bounding box, as well as the orientation and distance that need to be regressed in the first bounding box. Based on this, the embodiment of the present application can adjust the marked second bounding box to the size and position of the first bounding box. In this way, only one adjustment process is required to obtain the reference point (third key point) corresponding to the predicted first key point, and use the third key point as a reference to perform position regression and loss calculation on the first key point to implement the training process of the first model.

[0106] In some embodiments, see Figure 3C , step 102 can be implemented through steps 1021 to 1023.

[0107] In step 1021 , a scaling ratio of the second bounding box is determined based on a first size of the first bounding box and a second size of the second bounding box.

[0108] In actual implementation, the length of the diagonal line of the first bounding box may be used as the first size, and the length of the diagonal line of the second bounding box may be used as the second size.

[0109] As an example, the length d1 of the diagonal of the first bounding box and the length d2 of the diagonal of the second bounding box are determined, and the scaling ratio of the second bounding box is determined according to the ratio d2 / d1 of the length d2 to the length d1 .

[0110] In step 1022, the position of the second bounding box is calibrated based on the position of the first bounding box to obtain a third position.

[0111] In actual implementation, since it is necessary to determine the position corresponding to the marked second key point in the first bounding box to obtain the third key point as the reference position of the first key point, it is necessary to calibrate the position of the second bounding box based on the position of the first bounding box to obtain the third position for reference when the second bounding box is scaled, so that the second bounding box scaled based on the third position can overlap with the first bounding box.

[0112] As an example, when determining the third position of the second bounding box, the position between the points at the relative positions on the second bounding box and the first bounding box can be used for determination. For example, the coordinate position of the upper left corner of the first bounding box and the coordinate position of the upper left corner of the second bounding box are determined, and the position calibration is performed based on the position between the upper left corners of the two bounding boxes, and the upper left corner position of the first bounding box is determined as the third position of the upper left corner position of the second bounding box, or the coordinate position of the center point of the first bounding box and the coordinate position of the center point of the second bounding box are determined, and the position alignment is performed based on the position between the center points of the two bounding boxes, and the coordinate position of the center point of the first bounding box is determined as the third position of the center point position of the second bounding box. In order to improve the accuracy of the scaling position, two points can be determined on the first bounding box as the third positions of the points at the corresponding positions on the second bounding box, for example, the upper left corner of the first bounding box is the third position of the upper left corner of the second bounding box, and the lower right corner of the first bounding box is the third position of the lower right corner of the second bounding box.

[0113] In step 1023, based on the third position and the scaling ratio, the second bounding box and the second key point in the second bounding box are scaled, and the scaled second key point in the second bounding box is determined as the third key point.

[0114] In actual implementation, after determining the third position and the scaling ratio, the second bounding box and the second key point in the second bounding box can be scaled based on the scaling ratio until the second bounding box is the same size as the first bounding box. Then, based on the third position, the position of the scaled second bounding box and the scaled second key point is adjusted until the scaled second bounding box coincides with the first bounding box, and the position corresponding to the scaled second key point in the first bounding box is obtained, and this position is used as the position of the third key point to determine the third key point.

[0115] Continue reading Figure 3A , and the description will continue with the above step 102.

[0116] In step 103, a loss value is determined based on the first key point and the third key point.

[0117] In an embodiment of the present application, when training the first model, for the first key point predicted by the first model, the reference point directly used by the key point detection network in the related art (for example, the point corresponding to the upper left corner of the image sample) is no longer used for linear regression. Instead, the reference point (the third key point) determined by the pre-annotated second key point is used as the regression reference, so that the predicted first key point is regressed to the reference point (the third key point), thereby obtaining the key point position that can be predicted by the prediction model after parameter adjustment.

[0118] Therefore, the third key point serves as a reference point for the first key point. Based on the position between the first key point and the third key point, the loss value of the first model can be determined, so that the first model can be back-propagated through the loss value to adjust the parameters of the first model so that the position of the first key point predicted by the first model gets closer and closer to the third key point.

[0119] In some embodiments, see Figure 3D , step 103 can be implemented by following steps 1031 to 1033.

[0120] In step 1031 , a first offset between the first key point and the center point of the first bounding box and a second offset between the third key point and the center point of the scaled second bounding box are determined.

[0121] Here, the scaled second bounding box coincides with the first bounding box, and therefore, the center point of the first bounding box coincides with the center point of the scaled second bounding box.

[0122] In actual implementation, since the purpose of training the first model is to make the predicted first key point closer and closer to the third key point, the loss value can be calculated by the relative distance between the first key point and the third key point in the first bounding box. The smaller the relative distance between the first key point and the third key point, the smaller the corresponding loss value will be.

[0123] As an example, determine a first offset (x i ,y i ), and the second offset (x′) between the third keypoint and the center point of the second bounding box i , y′ i ). Among them, x i is the difference between the horizontal coordinates of the center point of the first bounding box and the horizontal coordinates of the first key point, y i is the difference between the vertical coordinates of the center point of the first bounding box and the vertical coordinates of the first key point; x′ iis the difference between the horizontal coordinates of the center point of the second bounding box and the horizontal coordinates of the third key point, y′ i is the difference between the vertical coordinates of the center point of the second bounding box and the vertical coordinates of the third key point.

[0124] In step 1032, a relative offset between the first key point and the third key point is determined based on the first offset and the second offset.

[0125] As an example, the horizontal offset x′ between the first key point and the third key point in the horizontal direction may be determined based on the first offset and the second offset: i -x i , the vertical offset y′ in the vertical direction i -y i . Horizontal offset x′ i -x i and the vertical offset y′ i -y i Constitutes the relative offset between the first and third keypoints.

[0126] In step 1033, a loss value is determined based on the relative offset.

[0127] In actual implementation, based on the relative offset and the Pythagorean theorem, the relative distance between the first key point and the third key point can be determined as The relative distance can also be called the Euclidean distance. Here, the calculated Euclidean distance can be directly used as the loss value of the first model.

[0128] In other embodiments, the method of determining the loss value based on the relative offset between the first key point and the third key point is not limited to the Euclidean distance, but may also include the following distance metrics: Manhattan Distance, also known as the city block distance, is a measure of the sum of the absolute wheelbases of two points in a standard coordinate system. If the coordinates of point c are known, the offsets of point a and point b relative to point c can be expressed as: |ax-bx|+|ay-by|, where (ax, ay) is the x and y coordinate offset of point a relative to point c, and (bx, by) is the x and y coordinate offset of point b relative to point c. Mahalanobis Distance is a covariance-based distance metric that can take into account the covariance structure of the data.

[0129] In practical applications, the choice of distance metric depends on the nature of the specific task of the first model of key point prediction and the distribution of the data. For example, if the data distribution is similar to the layout of city streets, then Manhattan distance may be more appropriate; if the data has many dimensions and the importance of each dimension is different, then Hamming distance or Mahalanobis distance may be more applicable. In machine learning, the most appropriate distance metric is usually selected based on the model objectives and data characteristics.

[0130] In the above manner, the loss is calculated through the relative offset between the first key point and the third key point, and then the back propagation and the process of adjusting the parameters of the first model can make the position regression of the first key point limited by the size of the first bounding box not deviate too far from the first bounding box, thereby reducing the occurrence of gradient explosion in the model training process to a certain extent.

[0131] In some embodiments, step 1033 may be implemented in the following manner.

[0132] If there are multiple first key points, determine the relative offset corresponding to each first key point; determine the weight of each first key point, and fuse the relative offsets corresponding to multiple first key points based on the weight to obtain a loss value.

[0133] As an example, the Euclidean distance is calculated based on the relative offset, and the Euclidean distance is used as the loss value. Since the number of first key points predicted for the sample object may be multiple (for example, M, where M is an integer greater than or equal to 2), it is necessary to calculate the relative offset corresponding to each first key point, and calculate the sum of the Euclidean distances based on the relative offset corresponding to each first key point.

[0134] In some cases, in different application scenarios, there may be different levels of attention for different first key points. Therefore, based on actual business needs, corresponding weights can be assigned to each first key point, and the Euclidean distances determined by the relative offsets corresponding to the M first key points can be weighted summed up to achieve the fusion processing of the M relative offsets and obtain the loss value of the first model.

[0135] As an example, take the face as the sample object, and predict the five first key points. Since the facial features of a person are all equally important key points, the weights of the five first key points can be set to 1. If the Euclidean distance is used as the loss value, the loss value Wherein, i=(1…5) represents the numbers 1 to 5 corresponding to the five first key points.

[0136] In step 104, a first model is trained based on the loss value.

[0137] In some embodiments, step 104 may be implemented in the following manner.

[0138] The first model is iteratively trained with the loss value approaching the loss threshold and the first key point predicted by the first model being close to the third key point as the training target, and the iterative training is stopped when the iterative training end condition is met.

[0139] The training end condition includes at least one of the following: the loss value reaches a loss threshold, and the number of iterative training times reaches a number threshold.

[0140] In the embodiment of the present application, the loss value is determined by the relative offset between the first key point and the third key point, such as the Euclidean distance between the first key point and the third key point. Therefore, the closer the first key point predicted by the first model is to the referenced third key point, the higher the accuracy of the key point position predicted by the first model, and the closer the first key point is to the third key point, the smaller its relative offset is, and the closer the Euclidean distance is to 0. Therefore, when training the first model, the loss threshold can be set to 0, and the first model can be iteratively trained with the loss value approaching 0 and the first key point being close to the third key point as the training target.

[0141] As an example, when the first model is iteratively trained, the model is back-propagated through the calculated loss value, and the parameters of the first model are adjusted so that the predicted first key point is offset to the position of the third key point. During training, the first model can directly infer the offset of the first key point to the referenced third key point based on the adjusted parameters, adjust the position of the first key point according to the offset, and obtain the position of the first key point that can be predicted by the first model after adjusting the parameters.

[0142] For example, see Figure 6 , Figure 6 The key point P shown is the first key point predicted by the first model, the key point G is the third key point referenced by the key point P, and the position of the key point G′ is the regression position inferred after adjusting the parameters of the first model. The regression formula of the key point can be expressed as:

[0143]

[0144] Among them, (P x , P y ) represents the coordinate position of the key point P; (G′ x , G′ y ) represents the coordinate position of the key point G′; P w represents the width of the first bounding box; P h Indicates the height of the first bounding box; d x (P) represents the offset in the horizontal direction; dy (P) represents the offset in the vertical direction.

[0145] Through the above regression formula, the formula for the regression amount (offset) can be obtained as follows:

[0146]

[0147] Therefore, during the first model training process, after adjusting the parameters based on the loss value, the offset of the first key point can be directly inferred by the above regression formula.

[0148] In actual implementation, the loss value reaching the loss threshold can be directly used as the training end condition. During the iterative training of the first model, if the loss value reaches the loss threshold, it means that the prediction accuracy of the first model has reached the expected result, and the iterative training is ended to obtain the second model for key point prediction.

[0149] In some cases, if the loss value reaching the loss threshold is the only condition for ending training, and the training of the first model may not reach the ideal state, that is, the loss value of the model is difficult to reach the loss threshold, then the iterative training end condition of the number of iterative training times reaching the number threshold can be added. When the number of iterative training times reaches the set number threshold but the loss value has not reached the loss threshold, the iterative training can also be ended to obtain the second model for key point prediction.

[0150] Through the above embodiment, the key point prediction of the sample object is performed through the first model to obtain the first bounding box and the first key point predicted for the sample object, and then the third key point for the first key point to be used as a position reference is obtained by adjusting the second bounding box of the sample object to the predicted first bounding box, and the loss value of the model is determined based on the first key point and the third key point, and the first model is back-propagated using the loss value to implement the training of the first model and obtain the second model for key point prediction. Through the above method, when training the first model, the linear regression method in the related art is no longer used to make the predicted first key point perform position regression, but the third key point determined by the pre-marked second key point is used as the regression reference point of the first key point, which reduces the loss convergence caused by gradient explosion or disappearance in the early stage of the training of the first model, making the initial stage of the training of the first model more stable, thereby improving the accuracy of the second model for key point prediction.

[0151] In some embodiments, after the iterative training of the first model is completed, a second model for key point prediction can be obtained (i.e., the second model obtained by training the first model). When the second model is used to predict the key points of the object in the image to be predicted, the image to be predicted is input into the second model, and the key points corresponding to the object in the image to be predicted can be predicted by the second model. For example, when the object in the image to be predicted is a face, the key points of the facial features such as the left eye, right eye, nose, left lip corner, right lip corner, etc. can be predicted by the second model; for another example, when the object in the image to be predicted is a pet (such as a dog), the left eye, right eye, nose, mouth, tail, etc. of the dog can be predicted by the second model. Since the second model uses the third key point determined by the pre-labeled second key point (that is, the key point position corresponding to the class standard object in the predicted bounding box) as the regression reference point of the first key point to train the first model, the training process is more stable and the occurrence of loss difficulty in convergence due to gradient explosion or disappearance in the early stage of training of the first model is reduced. Therefore, the key point position obtained by using the trained second model to predict the key points of the object in the predicted image is more accurate.

[0152] In a specific embodiment, a human face is used as a sample object for illustration.

[0153] The first model can be built based on the Yolo series detection framework. Specifically, the built and trained yolov5-face model can be used as the first model of the embodiment of the present application, and the yolov5-face model can be fine-tuned again through the training method of the embodiment of the present application.

[0154] The training process for the yolov5-face model may include an image sample preparation stage before model inference and a key point prediction stage using image samples.

[0155] See also Figure 7 , the image sample preparation stage can include:

[0156] S710, traverse all face bounding boxes in the image sample, and calculate the average of the lengths of the diagonals of all face bounding boxes.

[0157] First, the input size of the yolov5-face model is globally defined as 640x640x3, that is, the height h=640, width w=640, and number of channels c=3 of the image sample input to the yolov5-face model. In actual application, the balance can be achieved based on computing power and accuracy.

[0158] Taking the input size of the yolov5-face model as the standard, traverse all image samples, and scale the image samples to a uniform resolution or size (i.e., the input size of the yolov5-face model) by scaling regardless of the original size of the image samples. Usually, the annotation of the face bounding box is expressed in the form of (x, y, w, h), where x and y represent the coordinates of the center point of the face bounding box, and the face bounding box refers to the smallest rectangular box surrounding the face boundary. The center point is the midpoint of the face bounding box and the intersection of the diagonal of the face bounding box; w and h represent the width and height of the face bounding box, respectively. According to the w and h of the face bounding box, calculate the length d of the diagonal line, and count the mean d′ of the length d of the diagonal lines of all face bounding boxes in all image samples.

[0159] Suppose there is an image containing a face, where the initial annotation information of the face includes the face bounding box bbox: (x, y, w, h), the coordinates of the five key points of the face point5 (p1(x1, y1), p2(x2, y2), p3(x3, y3), p4(x4, y4), p5(x5, y5)). According to the input size requirements of the yolov5-face model, the values ​​of the face bounding box bbox and the five key point coordinates point5 are scaled to the corresponding input size, and the new bbox and point5 correspond to bbox. new and point5 new Then, according to the bbox of each face bounding box new , calculate the length d of its diagonal, and calculate the mean value of the length d of the diagonal of all face bounding boxes in all image samples, recorded as d′.

[0160] S720: Scale the face bounding box according to a ratio between a length of a diagonal line of the face bounding box and an average of the lengths of the diagonals.

[0161] Traverse all faces and scale the current face bounding box and five key points according to the ratio of d to d′ of the current face bounding box. The purpose of this step is to unify the size of the face bounding box and eliminate the interference of face size. The scaled face bounding box is (xs, ys, ws, hs), and the face key points are ps (p1 (x1, y1), p2 (x2, y2), p3 (x3, y3), p4 (x4, y4), p5 (x5, y5)).

[0162] For each face bounding box’s bboxnew and point5new, scale bboxnew and point5new according to the ratio of d and d′ to obtain the face bounding box’s bboxs and point5s.

[0163] S730, calculating the offset position of each key point from the center point of the face bounding box according to the scaled face bounding box and the key points.

[0164] S740, calculating the average values ​​corresponding to the five key points in all image samples, and obtaining the standard points corresponding to the key points.

[0165] According to bboxs and point5s, first get the coordinates of the center point of the face bounding box, and then calculate the horizontal pixel distance dx and vertical pixel distance dy of the five key points in point5s from the center point of the face bounding box. The dx and dy of each of the five key points are counted as the average value of all faces. For example, for the left eye p1 (x1, y1), the average distance (dx1, dy1) from the center point of the face bounding box is obtained, which is recorded as pd1, indicating the standard point of the left eye p1 (x1, y1).

[0166] Continue reading Figure 7 , the key point prediction stage using image samples can include:

[0167] S750, define the forward reasoning process of the yolov5-face model.

[0168] Assume that the input image sample I = (640, 640, 3), the face object of the image sample is input into a backbone network for forward reasoning (the forward reasoning process here is consistent with the general yolov5-face network reasoning, which will not be repeated here. It can be understood in conjunction with the yolov5-face network in the relevant technology). In the output layer of the yolov5-face model (usually there are multiple detection layers, only one layer is used as an example here), it is assumed that a 20*20*n high-dimensional data is obtained, where 20*20 is the value of the 640*640 spatial dimension after 32 times downsampling, and each cell (a cell can be understood as a pixel in the image sample, and each cell corresponds to n-dimensional information) contains n-dimensional information, including: the face bounding box information corresponding to the cell (length is 4), category information (when there are multiple categories, it is determined according to the category), confidence information (the probability value of the image in the frame is a face, the length is 1), and key point coordinate information (5 key points, each key point length is 2). Since this application only needs the face frame and five key points, only the 10-dimensional information including the face bounding box information and key point coordinate information predicted by each cell is used here to obtain the face bounding box and key points of the face object in the predicted image sample.

[0169] S760, calculating the loss value of the key points based on the currently predicted face bounding box and key points, and the face bounding box and key points annotated for the face object.

[0170] S770, perform model training based on the loss value.

[0171] S780, model accuracy tuning.

[0172] Calculate the loss value of key point prediction. The reference point of key point regression is no longer the origin in the image sample, but the standard point corresponding to each key point. For example, the reference point of left eye regression becomes pd1. Specifically, when obtaining pd1, pd1 needs to be scaled according to the ratio between the labeled face bounding box d and the current predicted face bounding box d, that is, the labeled face bounding box is scaled to the size of the current predicted face bounding box. After scaling, with the reference point, the specific loss calculation can use the Euclidean distance, followed by back propagation until the model converges.

[0173] During inference, the yolov5-face model will directly obtain the offset of each key point (predicted key point) relative to the reference point, and then calculate the final face key point based on this offset. Since the yolov5-face model is trained with the pre-labeled standard human key point positions as a reference, it is more stable in the early stages of training and will not encounter loss difficulties due to gradient explosion or disappearance. Therefore, the yolov5-face model after training is more accurate in predicting face key points.

[0174] The following continues to describe the exemplary structure of the model training device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the model training device 455 of the memory 450 may include:

[0175] A prediction module 4551 is used to predict key points of the sample object to obtain a first bounding box of the sample object and a first key point located in the first bounding box;

[0176] An adjustment module 4552 is used to adjust the second key point in the second bounding box of the sample object based on the first bounding box to obtain a third key point;

[0177] A loss determination module 4553, configured to determine a loss value based on the first key point and the third key point;

[0178] The training module 4554 is used to train the first model based on the loss value.

[0179] In some embodiments, the model training device 455 also includes: a labeling module, used to determine the third bounding box of the initially labeled sample object and the fourth key point located in the third bounding box; based on the first length of the diagonal of the third bounding box, the third bounding box and the fourth key point are scaled to obtain a second bounding box and a fifth key point located in the second bounding box; based on the first position of the key point of the sample object in the second bounding box, the second position of the fifth key point in the second bounding box is adjusted to obtain the second key point; based on the second bounding box and the second key point, the sample object is re-labeled to obtain a sample object labeled with the second bounding box and the second key point.

[0180] In some embodiments, the labeling module is further configured to determine the second length of the diagonal of the third bounding box of each sample object if there are multiple sample objects; and average the second lengths of the third bounding boxes of multiple sample objects to obtain the first length.

[0181] In some embodiments, the labeling module is also used to determine the horizontal distance and vertical distance between the fifth key point of each sample object and the center point of the second bounding box if there are multiple sample objects; average the multiple horizontal distances to obtain the average horizontal distance, and average the multiple vertical distances to obtain the average vertical distance; determine the first position based on the average horizontal distance and the average vertical distance.

[0182] In some embodiments, the annotation module is also used to determine the adjustment direction and adjustment distance for the fifth key point based on the first position in the second bounding box and the second position of the fifth key point in the second bounding box; based on the adjustment direction and adjustment distance, the second position of the fifth key point in the second bounding box is adjusted to obtain the second key point in the second bounding box, and the second key point is located at the first position in the second bounding box.

[0183] In some embodiments, the adjustment module 4552 is also used to determine the scaling ratio of the second bounding box based on the first size of the first bounding box and the second size of the second bounding box; calibrate the position of the second bounding box based on the position of the first bounding box to obtain a third position; scale the second bounding box and the second key point in the second bounding box based on the third position and the scaling ratio, and determine the second key point in the scaled second bounding box as the third key point.

[0184] In some embodiments, the loss determination module 4553 is also used to determine a first offset between the first key point and the center point of the first bounding box, and a second offset between the third key point and the center point of the scaled second bounding box; based on the first offset and the second offset, determine the relative offset between the first key point and the third key point; based on the relative offset, determine the loss value.

[0185] In some embodiments, the loss determination module 4553 is also used to determine the relative offset corresponding to each first key point if there are multiple first key points; determine the weight of each first key point, and fuse the relative offsets corresponding to multiple first key points based on the weight to obtain a loss value.

[0186] In some embodiments, the training module 4554 is also used to iteratively train the first model with the loss value approaching the loss threshold and the first key point predicted by the first model being close to the third key point as the training target, and stop the iterative training when the iterative training end condition is met; wherein the training end condition includes at least one of the following: the loss value reaches the loss threshold and the number of iterative training times reaches the number threshold.

[0187] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device 400 executes the training method of the model described in the embodiment of the present application.

[0188] The present application embodiment provides a computer-readable storage medium, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the training method of the model provided by the present application embodiment, for example, Figure 3A The training method of the model is shown.

[0189] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0190] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0191] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0192] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0193] In summary, through the embodiments of the present application, after the key point prediction is performed on the sample object and the first bounding box and the first key point predicted for the sample object are obtained, the second bounding box marked for the sample object is adjusted to the predicted first bounding box to obtain the third key point for which the first key point can be used as a position reference, the loss value of the model is determined based on the first key point and the third key point, and the first model is back-propagated using the loss value to achieve the training of the first model. Through the above method, when training the first model, the linear regression method in the related technology is no longer used to make the predicted first key point perform position regression, but the third key point determined by the pre-marked second key point is used as the regression reference point of the first key point to achieve the position regression of the first key point. This method reduces the situation in which the loss is difficult to converge due to gradient explosion or disappearance caused by linear regression in the early stage of the training of the first model, making the initial stage of the training of the first model more stable, thereby improving the accuracy of the first model for key point prediction. Moreover, during the training process, the loss is calculated through the relative offset between the first key point and the third key point, thereby back-propagating and adjusting the parameters of the first model. This can make use of the size of the first bounding box to limit the position regression of the first key point so that it will not deviate too far from the first bounding box, thereby reducing the occurrence of gradient explosion during the model training process to a certain extent.

[0194] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Performing key point prediction on the sample object to obtain a first bounding box of the sample object and a first key point located in the first bounding box; Based on the first bounding box, adjusting the second key point in the second bounding box of the sample object to obtain a third key point; Determine a loss value based on the first key point and the third key point; Based on the loss value, a first model is trained.

2. The method according to claim 1, characterized in that Before predicting key points of the sample object, the method further includes: Determine a third bounding box of the sample object initially marked and a fourth key point located in the third bounding box; Scaling the third bounding box and the fourth key point based on a first length of a diagonal of the third bounding box to obtain the second bounding box and a fifth key point located in the second bounding box; Based on the first position of the key point of the sample object in the second bounding box, adjusting the second position of the fifth key point in the second bounding box to obtain the second key point; The sample object is re-labeled based on the second bounding box and the second key point to obtain the sample object labeled with the second bounding box and the second key point.

3. The method according to claim 2, characterized in that Before scaling the third bounding box and the fourth key point based on the first length of the diagonal of the bounding box, the method further includes: If there are multiple sample objects, determining a second length of a diagonal line of a third bounding box of each sample object; The first length is obtained by averaging the second lengths of the third bounding boxes of the plurality of sample objects.

4. The method according to claim 2, characterized in that: Before adjusting the second position of the fifth key point in the second bounding box based on the first position of the key point of the sample object in the second bounding box, the method further includes: If there are multiple sample objects, determine the horizontal distance and the vertical distance between the fifth key point of each sample object and the center point of the second bounding box; averaging the plurality of horizontal distances to obtain an average horizontal distance, and averaging the plurality of vertical distances to obtain an average vertical distance; The first position is determined based on the average horizontal distance and the average vertical distance.

5. The method according to claim 2, characterized in that: The adjusting the second position of the fifth key point in the second bounding box based on the first position of the key point of the sample object in the second bounding box to obtain the second key point in the second bounding box includes: Determining an adjustment direction and an adjustment distance for the fifth key point based on the first position in the second bounding box and the second position of the fifth key point in the second bounding box; Based on the adjustment direction and the adjustment distance, the second position of the fifth key point in the second bounding box is adjusted to obtain the second key point in the second bounding box, and the second key point is located at the first position in the second bounding box.

6. The method according to claim 1, characterized in that The step of adjusting the second key point in the second bounding box annotated by the sample object based on the first bounding box to obtain the third key point includes: determining a scaling ratio of the second bounding box based on a first size of the first bounding box and a second size of the second bounding box; Based on the position of the first bounding box, calibrate the position of the second bounding box to obtain a third position; The second bounding box and the second key point in the second bounding box are scaled based on the third position and the scaling ratio, and the scaled second key point in the second bounding box is determined as a third key point.

7. The method according to claim 1, characterized in that The determining of the loss value based on the first key point and the third key point includes: Determining a first offset between the first key point and a center point of the first bounding box, and a second offset between the third key point and a center point of the scaled second bounding box; Determining a relative offset between the first key point and the third key point based on the first offset and the second offset; Based on the relative offset, a loss value is determined.

8. The method according to claim 7, characterized in that The determining of the loss value based on the relative offset includes: If there are multiple first key points, determine the relative offset corresponding to each of the first key points; The weight of each of the first key points is determined, and relative offsets corresponding to the plurality of first key points are fused based on the weight to obtain a loss value.

9. A model training device, characterized in that: The device comprises: A prediction module, configured to perform key point prediction on a sample object to obtain a first bounding box of the sample object and a first key point located in the first bounding box; an adjustment module, configured to adjust the second key point in the second bounding box of the sample object based on the first bounding box to obtain a third key point; A loss determination module, configured to determine a loss value based on the first key point and the third key point; A training module is used to train the first model based on the loss value.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 8 when executing computer executable instructions or computer programs stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images