Model training method, device, equipment, medium and program

Abstract

The present embodiment discloses a model training method and related device in the field of artificial intelligence. The method includes the steps of: performing object keypoint detection processing on target training images using m reference object keypoint detection models to obtain keypoint detection results including predicted positions of keypoints corresponding to each of a plurality of object key parts in the m target training images; for each object key part, determining whether the object key part is a target object key part based on the predicted positions of the keypoints corresponding to the object key part in the m keypoint detection results; if Yes, determining a labeling position corresponding to the target object key part; taking the target training images and the labeling positions corresponding to each of the target object key parts therein as training samples; and training a target object keypoint detection model based on the training samples. The method can improve the performance of the trained target object keypoint detection model.

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on November 24, 2021, application number 202111398623.5, entitled "Model training method and related apparatus," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of artificial intelligence, and in particular to model training. [Background technology]

[0003] The technology known as human body keypoint detection technology or human body pose estimation technology is used to recognize the positions of important joint points of the human skeleton in an image (e.g., nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, left and right ankles, etc.). Human body keypoint detection technology is now widely applied in many scenes. For example, in the scene of a haptic game, human body keypoint detection technology is used to detect changes in the movement of the player's limbs so that subsequent corresponding game logic can be executed according to the detection results.

[0004] The human body keypoint detection technology is mainly realized based on a human body keypoint detection model. That is, the human body keypoint detection model is used to identify the positions of important joint points on the human skeleton in an input image. In practical operation, labeling human body keypoint data takes a lot of time and effort, and it is not easy to label enough training data for training the human body keypoint detection model. Moreover, in order to ensure the detection efficiency of human body keypoints, the human body keypoint detection model to be actually input and used should not be too complex. Therefore, when training a human body keypoint detection model to be actually used in the related technology, a human body keypoint detection model with a complex structure and high performance is usually used to perform human body keypoint detection processing on an unlabeled image to identify the positions of human body keypoints in the image and use them as pseudo labels for the image, and then the image and its pseudo label are used as training samples, and the training samples are used to train a human body keypoint detection model with a simpler structure to be actually used. Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if the human body keypoint detection model has a complex structure, the model performance is limited, and it is difficult to ensure the accuracy of the determined pseudo label, and the pseudo label generated by the above method usually contains a lot of noise. Therefore, if a human body keypoint detection model to be actually used is trained using training samples including such pseudo labels, the performance of the trained human body keypoint detection model will be affected.

[0006] The model training method and related apparatus provided in the embodiments of the present application can improve the accuracy of the determined pseudo labels, and thus improve the performance of a human body keypoint detection model trained based on training samples containing the pseudo labels. [Means for solving the problem]

[0007] In view of this, a model training method provided in a first aspect of the present application includes the steps of: performing object keypoint detection processing on a target training image using m (m is an integer greater than 1) reference object keypoint detection models, respectively, to obtain keypoint detection results corresponding to each of the m reference object keypoint detection models, where the keypoint detection results include predicted positions of keypoints corresponding to each of a plurality of object key features in the target training image; determining, for each of the object key features, whether the object key feature is a target object key feature based on the predicted positions of the keypoints corresponding to the object key feature in the m keypoint detection results, and if Yes, determining a labeling position corresponding to the target object key feature based on the predicted positions of the keypoints corresponding to the target object key feature in the m keypoint detection results; using the target training images and the corresponding keypoint labeling results including labeling positions corresponding to each of the target object key features as training samples; and training a target object keypoint detection model based on the training samples.

[0008] A model training apparatus provided in a second aspect of the present application includes a keypoint detection module for performing object keypoint detection processing on a target training image using m (m is an integer greater than 1) reference object keypoint detection models, respectively, to obtain keypoint detection results corresponding to the m reference object keypoint detection models, the keypoint detection results including predicted positions of keypoints corresponding to each of a plurality of object key sites in the target training image; and a model training apparatus for performing, for each of the object key sites, a prediction process based on the predicted positions of the keypoints corresponding to the object key sites in the m keypoint detection results. the m keypoint detection results including a labeling position corresponding to each of the target object key portions; a sample generation module used to generate training samples from the target training images and the corresponding keypoint labeling results including a labeling position corresponding to each of the target object key portions; and a model training module used to train a target object keypoint detection model based on the training samples.

[0009] A third aspect of the present application provides a computer device comprising a processor and a memory, the memory being adapted to store a computer program, and the processor being adapted to execute steps of the model training method of the first aspect based on the computer program.

[0010] A fourth aspect of the present application provides a computer-readable storage medium for storing a computer program, the computer program being used to execute steps of the model training method of the first aspect above.

[0011] A fifth aspect of the present application provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium, the computer instructions being read by a processor of a computing device from the computer readable storage medium and executed by the processor to cause the computing device to perform the steps of the model training method of the first aspect above. Effect of the Invention

[0012] As can be seen from the above technical solution, the embodiment of the present application has the following advantages:

[0013] In the embodiment of the present application, a model training method is provided. In the method, when generating training samples for training a target object keypoint detection model to be actually used, m (m is an integer greater than 1) reference object keypoint detection models are used to perform object keypoint detection processing on a target training image, respectively, to obtain keypoint detection results corresponding to each of the m reference object keypoint detection models. Here, each keypoint detection result includes a predicted position of a keypoint corresponding to each of a plurality of object key parts in the target training image. Then, based on the principle that the predicted positions of the same object key part in each keypoint detection result should basically be the same, for each object key part, it is evaluated whether the position prediction result of each reference object keypoint detection model for the object key part is reliable based on the predicted position of the keypoint corresponding to the object key part in each keypoint detection result. That is, it is determined whether the object key part is a target object key part. If it is determined that the position prediction result of each reference object keypoint detection model for the object key part is reliable, further, a labeling position, i.e., a pseudo label, corresponding to the target object key part is determined. Further, the target training image and the labeling positions corresponding to each of the target object key parts therein are used to construct training samples. In this way, the object key parts whose position prediction results are unreliable are ignored, and only the labeling positions of the object key parts whose position prediction results are reliable are used to obtain pseudo labels, thereby effectively reducing noise in the determined pseudo labels and improving the accuracy of the pseudo labels. Therefore, if a target object keypoint detection model with a relatively simple structure that is to be actually used is trained based on the training samples including the pseudo labels, the performance of the trained target object keypoint detection model can be guaranteed to a certain extent. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 is a schematic diagram of an application scenario of the model training method provided in the embodiment of the present application. [Diagram 2] FIG. 2 is a schematic flow diagram of the model training method provided in the present embodiment. [Diagram 3] FIG. 3 is a schematic diagram of the working principle of the reference object keypoint detection model provided in the embodiment of the present application. [Figure 4] FIG. 4 is a schematic diagram of the training flow of the target object keypoint detection model provided in the embodiment of the present application. [Diagram 5] FIG. 5 is a diagram showing the overall implementation architecture of the virtual game provided in the embodiment of the present application. [Figure 6] FIG. 6 is a schematic diagram of the training architecture of a human body keypoint detection model applied to a virtual game provided in an embodiment of the present application. [Figure 7] FIG. 7 is a schematic diagram of the test result of the human body keypoint detection model provided in the embodiment of the present application. [Figure 8] FIG. 8 is a structural schematic diagram of the model training device provided in the present embodiment. [Figure 9] FIG. 9 is a structural schematic diagram of another model training device provided in an embodiment of the present application. [Figure 10] FIG. 10 is a structural schematic diagram of yet another model training device provided in an embodiment of the present application. [Figure 11] FIG. 11 is a structural schematic diagram of a terminal device provided in an embodiment of the present application. [Figure 12] FIG. 12 is a schematic diagram of the server structure provided in this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In order to make the scheme of the present application more easily understood by those skilled in the art, the technical scheme of the embodiments of the present application will be described below clearly and comprehensively in combination with the drawings in the embodiments of the present application. Of course, the described embodiments are only a part of the embodiments of the present application, and are not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive ideas are all within the scope of protection of the present application.

[0016] The terms "first," "second," "third," "fourth," etc. (if any) in the present specification, claims, and drawings are used to distinguish between similar objects and are not necessarily intended to describe a particular order or sequence. It is to be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present invention described herein may be performed in other orders than those shown or described. In addition, the terms "comprises," "has," and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or means may include, but is not limited to, those steps or means expressly recited, or other steps or means that are not expressly recited, or that are inherent to the process, method, product, or apparatus.

[0017] With the research and progress of artificial intelligence technology, it is being researched and applied in many fields, such as commonly seen smart homes, smart wearable devices, virtual assistants, smart speakers, intelligent marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles (IoV), autonomous driving, intelligent transportation systems (ITS), etc. With the development of technology, it is certain that artificial intelligence technology will be applied in more fields and exert more important value.

[0018] The scheme provided in the embodiments of the present application relates to computer visual technology of artificial intelligence, and can be applied to the above-mentioned various fields, which will be specifically described in the following embodiments.

[0019] In the related art, the keypoint detection results for unlabeled images using a complex object keypoint detection model are directly used as pseudo labels for the unlabeled images, and the training samples consisting of the unlabeled images and their corresponding pseudo labels are used to train an object keypoint detection model with a simple structure that is to be actually used. The pseudo labels generated by this method usually contain a lot of noise and have low accuracy, which adversely affects the model performance of the trained object keypoint detection model.

[0020] In order to solve the problems in the related art, the embodiment of the present application provides a model training method. In the model training method, first, object keypoint detection processing is performed on a target training image using m (m is an integer greater than 1) reference object keypoint detection models, and keypoint detection results corresponding to each of the m reference object keypoint detection models are obtained. Here, each keypoint detection result includes a predicted position of a keypoint corresponding to each of a plurality of object key parts in the target training image. Next, for each object key part, it is determined whether the object key part is a target object key part based on the predicted position of the keypoint corresponding to the object key part in the m keypoint detection results. If yes, a labeling position corresponding to the target object key part is determined based on the predicted position of the keypoint corresponding to the target object key part in the m keypoint detection results. Furthermore, the target training image and the labeling positions corresponding to each of the target object key parts therein are used as training samples, and a target object keypoint detection model with a relatively simple structure that is to be actually used is trained based on the training samples.

[0021] When the training sample for training the target object keypoint detection model to be actually used is generated by the above model training method, a plurality of reference object keypoint detection models with complex structures are used to perform object keypoint detection processing on the target training image, respectively, to obtain a plurality of keypoint detection results. Then, based on the principle that the predicted positions of the same object key parts in each keypoint detection result should basically be the same, for each object key part, it is evaluated whether the position prediction result of each reference object keypoint detection model for the object key part is reliable based on the predicted position of the keypoint corresponding to the object key part in each keypoint detection result. That is, it is determined whether the object key part is a target object key part. If it is determined that the position prediction result of each reference object keypoint detection model for the object key part is reliable, further, a labeling position, i.e., a pseudo label, corresponding to the target object key part is determined. Then, a training sample is constructed using the target training image and the labeling positions corresponding to each target object key part therein. In this way, the object key parts whose position prediction results are unreliable are ignored, and only the labeling positions corresponding to the object key parts whose position prediction results are reliable are used to determine the pseudo labels, so that the noise present in the determined pseudo labels can be effectively reduced and the accuracy of the pseudo labels can be improved. Therefore, if a target object keypoint detection model with a relatively simple structure that is to be actually used is trained based on training samples including such pseudo labels, the performance of the trained target object keypoint detection model can be guaranteed to a certain extent.

[0022] It should be noted that the model training method provided in the present embodiment can be executed by a computer device having image processing capabilities. The computer device can be a terminal device or a server. The terminal device can be specifically a computer, a smartphone, a tablet computer, a PDA (Personal Digital Assistant), etc. The server can be specifically an application server or a web server. When actually deployed, it may be a standalone server, or a group server or cloud server consisting of multiple physical servers. The processing data (training images, keypoint detection results, keypoint labeling results, etc.) according to the present embodiment can be stored in a blockchain.

[0023] In order to facilitate understanding of the model training method provided in the embodiments of the present application, the following describes an example of an application scenario of the model training method, taking as an example a case where the execution entity of the model training method is a server.

[0024] Please refer to FIG. 1. FIG. 1 is a schematic diagram of an application scenario of the model training method provided in the embodiment of the present application. As shown in FIG. 1, the application scenario includes a server 110 and a database 120. The server 110 can obtain data from the database 120 through a network. Alternatively, the database 120 can be integrated into the server 110. The server 110 is used to execute the model training method provided in the embodiment of the present application to train a target object keypoint detection model with a relatively simple structure that is to be actually used. The database 120 is used to store unlabeled training images.

[0025] In practical operation, the server 110 can obtain target training images from the database 120. The target training images include objects to be detected, and the objects to be detected include some object key parts. For example, the target training images include a clear and complete human body to be detected.

[0026] Next, the server 110 performs object keypoint detection processing on the acquired target training image using m (m is an integer greater than 1) reference object keypoint detection models, respectively, to obtain keypoint detection results corresponding to the m reference object keypoint detection models, where each keypoint detection result includes a predicted position of a keypoint corresponding to each of a plurality of object key features in the target training image. In addition, the above-mentioned reference object keypoint detection model is a pre-trained object keypoint detection model, which can usually detect the position of each object key feature on the object to be detected in the image relatively accurately, and the model structure is generally relatively complex.

[0027] As an example, each of the above-mentioned reference object keypoint detection models may be a reference human body keypoint detection model for detecting the position of each important joint on the human body skeleton. Therefore, each of the reference human body keypoint detection models performs human body keypoint detection processing on the target training image respectively, and obtains keypoint detection results corresponding to each of the reference human body keypoint detection models. The keypoint detection results include predicted positions of keypoints corresponding to each important joint of the human body to be detected in the target training image.

[0028] Next, for each object key part, the server 110 judges whether the prediction result of the m reference object keypoint detection models for the position of the object key part is reliable based on the predicted position of the keypoint corresponding to the object key part in the m keypoint detection results. That is, it judges whether the object key part is a target object key part. If it is judged that the object key part is a target object key part, it determines a labeling position corresponding to the target object key part based on the predicted position of the keypoint corresponding to the target object key part in the m keypoint detection results.

[0029] It should be noted that the position prediction results of different reference object keypoint detection models for the same object key part in the target training image should basically be consistent. That is, the predicted positions of different reference object keypoint detection models for the same object key part should have a small distance from each other. If the distance between the predicted positions for the same object key part in the m keypoint detection results is large, it indicates that the predicted position corresponding to the object key part in the m keypoint detection results is unreliable, and there is probably a large noise. In order to prevent this noise from being introduced into the subsequent model training, the object key part may be ignored, and the labeling position corresponding to the object key part is not determined based on the predicted position for the object key part in the m keypoint detection results. On the contrary, if the distance between the predicted positions for the same object key part in the m keypoint detection results is small, it indicates that the predicted position corresponding to the object key part in the m keypoint detection results is reliable. Therefore, the object key part is regarded as a target object key part, and the labeling position corresponding to the target object key part is determined based on the predicted position corresponding to the target object key part in the m keypoint detection results.

[0030] After performing the above process for each object key feature in the target training image to obtain the labeling position corresponding to each target object key feature in the target training image, the labeling position corresponding to each target object key feature in the target training image is used to construct the keypoint labeling result corresponding to the target training image, i.e., the pseudo label corresponding to the target training image. Furthermore, the target training image and its corresponding keypoint labeling result can be used as a training sample.

[0031] Further, a target object keypoint detection model is trained based on the training sample constructed by the above method. The target object keypoint detection model is an object keypoint detection model to be actually used, for example, a human body keypoint detection model applied to a immersive game to recognize the player's body movements. It is considered that the object keypoint detection model to be actually used needs to have high detection efficiency. Therefore, an object keypoint detection model with a relatively simple model structure is usually used as the target object keypoint detection model.

[0032] 1 is merely an example, and in actual operation, the model training method provided in the embodiment of the present application can be applied to other scenes. The application scenes to which the model training method provided in the embodiment of the present application is applied are not limited here.

[0033] Next, the model training method provided in this application will be described in detail through a method embodiment.

[0034] Please refer to Figure 2. Figure 2 is a schematic flow diagram of a model training method provided in an embodiment of the present application. For convenience of explanation, the following embodiment also takes the case where the execution subject of the model training method is a server as an example. As shown in Figure 2, the model training method includes the following steps:

[0035] Step 201: Perform object keypoint detection processing on a target training image using m reference object keypoint detection models, respectively, to obtain keypoint detection results corresponding to the m reference object keypoint detection models, where the keypoint detection results include predicted positions of keypoints corresponding to a plurality of object key portions in the target training image, where m is an integer greater than 1.

[0036] In the embodiment of the present application, when the server generates corresponding pseudo labels for a target training image, it first performs object keypoint detection processing on the target training image using m reference object keypoint detection models, respectively, to obtain keypoint detection results for the target training image for each of the m reference object keypoint detection models, that is, obtain keypoint detection results corresponding to each of the m reference object keypoint detection models, where each keypoint detection result includes the predicted positions of keypoints corresponding to each of a plurality of object key portions in the target training image.

[0037] To elaborate, pseudo-labels are training data commonly used in semi-supervised learning. Usually, unlabeled data is processed by a more complex model with better performance to obtain a pseudo-label corresponding to the unlabeled data, but the pseudo-label may be inaccurate. In the embodiment of the present application, the pseudo-label corresponding to the target training image is determined based on the keypoint detection results of the m reference object keypoint detection models for the target training image. The gist of the present application is to process the keypoint detection results of the m reference object keypoint detection models for the target training image through a series of processing flows to obtain a pseudo-label that relatively accurately reflects the position of the object key part in the target training image, thereby improving the performance of the target object keypoint detection model trained based on the training sample containing the pseudo-label.

[0038] To explain further, the reference object keypoint detection model is a pre-trained model for detecting the location of an object key part on an object to be detected in an image, and can generally detect the location of the object key part relatively accurately. In general, a relatively complex model structure is set for the reference object keypoint detection model so that the reference object keypoint detection model can accurately detect the location of the object key part. Exemplarily, each of the reference object keypoint detection models is a reference human body keypoint detection model for detecting the location of each important joint on a human body in an image, and the keypoint detection result of the reference human body keypoint detection model is used to identify the posture of the human body. Note that the m reference object keypoint detection models in the embodiment of the present application may be trained based on the same training sample set or different training sample sets. The present application does not limit this.

[0039] To further explain, the target training image is an image including an object to be detected, and the object to be detected includes several object key parts. The object key parts include parts of the object to be detected that can reflect the posture of the object to be detected. For example, the target training image is an image including a clear and complete human body to be detected, and the human body to be detected includes several important joints, such as a nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, left and right ankles, etc.

[0040] To further explain, the keypoint detection result corresponding to the reference object keypoint detection model is obtained by the object keypoint detection process performed by the reference object keypoint detection model on the target training image, and the keypoint detection result includes the predicted positions of the keypoints corresponding to each object key portion on the object to be detected in the target training image. For example, assuming that the target training image is an image with a size of H×W, when the reference object keypoint detection model performs the object keypoint detection process on the target training image, a heat map with a size of K×H×W is output, where K is the number of object key portions to be detected, and the k-th heat map of the K×H×W heat maps is used to indicate the keypoint predicted position corresponding to the k-th object key portion in the target training image.

[0041] In an alternative embodiment, the server may obtain keypoint detection results corresponding to each reference object keypoint detection model in the following manner: for each reference object keypoint detection model, perform object keypoint detection processing on a target training image using the reference object keypoint detection model to determine the confidence that each pixel in the target training image belongs to an object key portion, and then determine keypoint detection results corresponding to the reference object keypoint detection model based on pixels in the target training image that have a confidence that they belong to an object key portion higher than a predetermined confidence threshold.

[0042] Specifically, FIG. 3 is a schematic diagram of the operation principle of the reference object keypoint detection model provided in the embodiment of the present application. As shown in FIG. 3, when a server inputs a target training image with a size of H×W into a certain reference object keypoint detection model, the reference object keypoint detection model performs an analysis process on the target training image and outputs a heat map with a size of K×H×W accordingly. The heat map is essentially composed of K heat maps with a size of H×W, and each heat map with a size of H×W contains the confidence that each pixel in the target training image belongs to the object key part corresponding to the heat map. For example, in the kth heat map with a size of H×W, each pixel has a corresponding confidence, and the confidence represents the probability that the corresponding pixel belongs to the kth object key part.

[0043] For each heat map with a size of H×W, the server can ignore pixels whose corresponding confidence is less than a predetermined confidence (e.g., 0.5). For example, the confidence corresponding to such pixels can be directly adjusted to 0, and the pixels are considered not to belong to the object key part corresponding to the heat map. Then, the server can determine the predicted location of the object key part corresponding to the heat map based only on pixels whose corresponding confidence is higher than the predetermined confidence. For example, the confidence corresponding to such pixels can be adjusted to 1, and the pixels are implicitly accepted as belonging to the object key part corresponding to the heat map. Thus, the location of the pixel in the heat map whose corresponding confidence is equal to 1 can reflect the predicted location of the object key part corresponding to the heat map. In this way, when the primary screening process of K heat maps with a size of H×W is completed by the above method, the heat map with a size of K×H×W obtained by the primary screening process can be the keypoint detection result corresponding to the reference object detection model.

[0044] In this way, the actual detection result of the reference object keypoint detection model is pre-processed to obtain the keypoint detection result corresponding to the reference object keypoint detection model, so that the noise in the keypoint detection result can be further reduced, and at the same time, the server can avoid performing unnecessary subsequent processing on such noise, so that the required processing resources can be reduced and the subsequent processing efficiency can be improved.

[0045] Step 202: For each of the object key parts, determine whether the object key part is a target object key part based on the predicted position of the keypoint corresponding to the object key part in the m keypoint detection results, and if Yes, determine a labeling position corresponding to the target object key part based on the predicted position of the keypoint corresponding to the target object key part in the m keypoint detection results.

[0046] After obtaining the keypoint detection results corresponding to each of the m reference object keypoint detection models, the server judges whether the prediction results for the position of each object key portion of the m reference object keypoint detection models are reliable based on the m keypoint detection results. That is, for each object key portion, the server judges whether the object key portion is a target object key portion based on the predicted position of the keypoint corresponding to the object key portion in the m keypoint detection results, and if it is determined that the object key portion is a target object key portion, the server further determines a labeling position corresponding to the target object key portion based on the predicted position of the keypoint corresponding to the target object key portion in the m keypoint detection results. Furthermore, the server can use the labeling position corresponding to each target object key portion in the target training image to construct a keypoint labeling result corresponding to the target training image.

[0047] From the viewpoint of spatial consistency, the position prediction results of different reference object keypoint detection models for the same object key part in the target training image should basically be consistent. That is, the predicted positions of the same object key part of different reference object keypoint detection models should be close to each other. If the distance between the predicted positions corresponding to the same object key part in the m keypoint detection results is far, it indicates that the predicted position corresponding to the object key part in the m keypoint detection results is unreliable, and there may be a large noise. In order to prevent this noise from being introduced into the subsequent model training, the embodiment of the present application chooses to ignore the object key part, and does not determine the labeling position corresponding to the object key part based on the predicted position corresponding to the object key part in the m keypoint detection results. That is, the pseudo label corresponding to the target training image is not generated based on the object key part. On the contrary, if the distance between the predicted positions corresponding to the same object key part in the m keypoint detection results is close to each other, it indicates that the predicted position corresponding to the object key part in the m keypoint detection results is reliable. Therefore, the object key portion can be regarded as a target object key portion, and a labeling position corresponding to the target object key portion can be determined based on a predicted position corresponding to the target object key portion in the m keypoint detection results, and then a pseudo label corresponding to the target training image can be determined based on the labeling position corresponding to the target object key portion.

[0048] In an alternative embodiment, the server may determine whether the position prediction result for each object key part of the m reference object keypoint detection models is reliable in the following manner. That is, for each object key part, determine whether it is a target object key part in the following manner. That is, determine a reference position corresponding to the object key part based on the predicted position of the keypoint corresponding to the object key part in the m keypoint detection results. Next, for each keypoint detection result, determine the distance between the predicted position of the keypoint corresponding to the object key part and the reference position corresponding to the object keypoint, and determine whether the keypoint detection result belongs to the target keypoint detection result corresponding to the object key part based on the distance. Furthermore, count the number of targets of the target keypoint detection result corresponding to the object key part in the m keypoint detection results. If the number of targets exceeds a predetermined number, the object key part can be determined to be a target object key part. If the number of targets does not exceed a predetermined number, the object key part can be determined to be a non-target object key part.

[0049] The predicted position of the keypoint corresponding to the object key part is obtained by object keypoint detection processing for the target training image by the reference object keypoint detection model, and is the position prediction result of the object key part output by the reference object keypoint detection model. The reference position corresponding to the object key part is specified based on the position prediction results of the object key part in the keypoint detection results of multiple reference object keypoint detection models, and is obtained, for example, by averaging the predicted positions corresponding to the object key part in the multiple keypoint detection results. It is used to reflect the distribution situation of the predicted positions corresponding to the object key part in the multiple keypoint detection results.

[0050] In order to facilitate understanding of the above-mentioned realization process, the above-mentioned realization process will be described below by taking as an example a case where the object key part "head" is determined to be a target object key part. When determining whether the head belongs to the target object key part, the server calculates the average value of the predicted positions of the keypoints corresponding to the head in the m keypoint detection results, and sets it as the reference position corresponding to the head. Next, with the reference position corresponding to the head as the center, the server specifies the distance between the predicted position of the keypoint corresponding to the head among the keypoint detection results and the reference position, and judges whether the distance is smaller than a first predetermined distance (for example, when normalization processing is performed for each position, the first predetermined distance can be 0.1). If it is smaller, the keypoint detection result is determined to belong to the target keypoint detection result corresponding to the head, and conversely, if it is not smaller, the keypoint detection result is determined to not belong to the target keypoint detection result corresponding to the head. When the above-mentioned process is completed for all m keypoint detection results, the number of targets in the target keypoint detection result corresponding to the head in the m keypoint detection results is tallied. If the number of targets corresponding to the head exceeds a predetermined number (e.g., m / 2), it can be determined that the head belongs to the target object key region. Conversely, if the number of targets corresponding to the head does not exceed a predetermined number, it can be determined that the head belongs to the non-target object key region.

[0051] It should be noted that the above-mentioned parameters such as the first predetermined distance and the predetermined quantity can be set according to actual needs, and the present application does not make any limitations thereon.In addition, the above-mentioned implementation of determining the reference position corresponding to the object key part and the implementation of determining whether the keypoint detection result belongs to the target keypoint detection result corresponding to the object key part can also be set according to actual needs, and the present application does not make any limitations thereon.

[0052] The above method judges whether the position prediction result of each object key part of the m reference object keypoint detection models is reliable, so that the reliability of the judgment result is guaranteed, and the position prediction result of the object key part with low reliability can be effectively filtered out. Therefore, the pseudo label corresponding to the target training image determined subsequently is guaranteed to have high reliability, and a large amount of noise can be avoided.

[0053] When determining whether each object key part is a target object key part using the above method, the server can further determine a labeling position corresponding to the target object key part using the following method: for each target keypoint detection result corresponding to the target object key part, identify the predicted position of the keypoint corresponding to the target object key part among them, and set it as the target predicted position corresponding to the target object key part; and further determine a labeling position corresponding to the target object key part based on each target predicted position corresponding to the target object key part.

[0054] For example, for a certain target object key portion, the server obtains a predicted position of a key point corresponding to the target object key portion in each target key point detection result corresponding to the target object key portion, and calculates the target object key portion. part Further, the server may calculate an average value of each of the target predicted positions corresponding to the target object key portion to obtain a labeling position corresponding to the target object key portion.

[0055] In this way, by determining a labeling position corresponding to the target object key part based only on the position prediction result of the target object key part in the target keypoint detection result corresponding to the target object key part, the accuracy of the determined labeling position can be guaranteed.

[0056] In actual operation, the server may use other methods to determine the labeling position corresponding to the target object key part. For example, the reference position corresponding to the target object key part (i.e., the average value of the predicted positions of the key points corresponding to the target object key part in the m key point detection results) may be used as the labeling position corresponding to the target object key part. The present application does not place any restrictions on the method by which the server determines the labeling position corresponding to the target object key part.

[0057] Optionally, if the target training images are video frames in a target video, the server may perform further determinations on the target object key portion based on time sequence continuity to determine whether to retain the target object key portion.

[0058] That is, the server can determine a video frame adjacent to the target training image in the target video as a reference training image, and obtain a keypoint labeling result corresponding to the reference training image. The keypoint labeling result includes a labeling position corresponding to each target object key feature in the reference training image. Then, for each target object key feature, the labeling position in the keypoint labeling result corresponding to the target training image is determined to be a first position, and the labeling position in the keypoint labeling result corresponding to the reference training image is determined to be a second position. Based on the distance between the first position and the second position, it is determined whether to keep the first position in the keypoint labeling result corresponding to the target training image and whether to keep the second position in the keypoint labeling result corresponding to the reference training image.

[0059] In practical operation, there should not be a large difference between adjacent video frames of a video, which means that the change degree of the object key part on the object to be detected in the adjacent video frames is also not large. Based on this, in the embodiment of the present application, further judgment processing can be performed on the target object key part on the key point labeling result corresponding to each of two adjacent video frames of a video.

[0060] Specifically, the server identifies the target video to which the target training image belongs, and identifies a video frame adjacent to the target training image in the target video as a reference training image. The reference training image may be a video frame immediately before the target training image, or a video frame immediately after the target training image. Then, obtain a keypoint labeling result corresponding to the reference training image. The method of generating the keypoint labeling result corresponding to the reference training image is the same as the method of generating the keypoint labeling result corresponding to the target training image described above, and the keypoint labeling result corresponding to the reference training image includes a labeling position corresponding to each target object key part in the reference training image.

[0061] For each target object key portion in the target training images and the reference training images, the server identifies a labeling position corresponding to the target object key portion in the keypoint labeling result corresponding to the target training image as a first position, and identifies a labeling position corresponding to the target object key portion in the keypoint labeling result corresponding to the reference training image as a second position. Then, the server calculates the distance between the first position and the second position and determines whether the distance is smaller than a second predetermined distance (e.g., when a normalization process is performed on each position, the second predetermined distance can be 0.2), and if it is smaller, the server may keep the labeling positions (i.e., the first position and the second position) corresponding to the target object key portion in the keypoint labeling result corresponding to each of the target training images and the reference training images. Conversely, if it is not smaller, the server may delete the labeling positions corresponding to the target object key portion in the keypoint labeling result corresponding to each of the target training images and the reference training images. That is, the server regards the target object key portion as an invalid object key portion.

[0062] The second predetermined distance may be set according to actual needs, and the present application does not limit it in any way.

[0063] In this way, the above method can further screen the target object key regions based on time series continuity, thereby contributing to improving the accuracy of the determined keypoint labeling results and further filtering out noise that may be present in the keypoint labeling results.

[0064] Step 203: The target training image and the corresponding keypoint labeling result are set as a training sample, and the keypoint labeling result includes a labeling position corresponding to each of the target object key features.

[0065] After obtaining the keypoint labeling results corresponding to the target training images in the above manner (including the labeling positions corresponding to each target object key feature in the target training images), the server can use the target training images and the corresponding keypoint labeling results to construct training samples for training a target object keypoint detection model to be actually used, where the keypoint labeling results corresponding to the target training images are essentially pseudo-labels corresponding to the target training images.

[0066] Step 204: Train a target object keypoint detection model based on the training samples.

[0067] In actual operation, the server can generate a large number of training samples through the above steps 201 to 203, and further use the generated training samples to train a target object keypoint detection model that is to be actually used.

[0068] To further explain, the target object keypoint detection model is a model that requires training, for detecting the location of the object key part on the object to be detected in the image, and is a model that is intended to be used in practice. In general, in order to ensure detection efficiency in practical operation, the model structure of the target object keypoint detection model is generally relatively simple. That is, the complexity of the target object keypoint detection model is generally lower than that of the reference object keypoint detection model. For example, the target object keypoint detection model is a model for detecting the change of the player's limb movement applied to a immersive game, and is used to detect the important joint point position of the human body in the image captured by a camera and identify the movement posture of the human body.

[0069] In one possible implementation, the server realizes training for a target object keypoint detection model according to the flow shown in Figure 4. As shown in Figure 4, the server ObjectSpecifically, when training a keypoint detection model, the following steps need to be performed:

[0070] Step 2041: Using the target object keypoint detection model, perform object keypoint detection processing on the target training images in the training sample to obtain training keypoint detection results corresponding to the target training images, the training keypoint detection results including training predicted positions of keypoints corresponding to each of a plurality of object key features.

[0071] Specifically, when the server uses the training sample to train the target object keypoint detection model, the target training image in the training sample is input to the target object keypoint detection model, and the target object keypoint detection model can perform analysis processing on the input target training image to output a training keypoint detection result corresponding to the target training image, the training keypoint detection result including the training predicted positions of the keypoints corresponding to each of a plurality of object key portions in the target training image.

[0072] It should be noted that the operation principle of the target object keypoint detection model and the above-mentioned reference object keypoint detection model is basically the same, and only the model structure is different between the two. In general, the structure of the target object keypoint detection model is simpler than that of the reference object keypoint detection model. Based on this, the operation principle of the target object keypoint detection model may also be the same as that of FIG. 3. That is, a target training image with a size of H×W is input to the target object keypoint detection model, and the target object keypoint detection model processes the target training image, and the output training keypoint detection result is a heat map with a size of K×H×W. Here, the k-th heat map yk is used to represent the predicted position of the keypoint corresponding to the k-th object key part.

[0073] Step 2042: Construct a target loss function based on the training keypoint detection results corresponding to the target training images and the keypoint labeling results in the training samples.

[0074] The server uses the target object keypoint detection model to perform keypoint detection processing on the target training image to obtain training keypoint detection results corresponding to the target training image, and then constructs a target loss function based on the difference between the training keypoint detection results and the keypoint labeling results in the training samples.

[0075] In an alternative embodiment, the server constructs a target loss function in the following manner: for each keypoint in the training keypoint detection result, determine whether the object key part corresponding to the keypoint is a target object key part in the keypoint labeling result, and if yes, construct a loss term corresponding to the keypoint based on a training predicted position corresponding to the keypoint and a labeling position of the target object key part corresponding to the keypoint, and set a corresponding first loss weight for the loss term corresponding to the keypoint; if no, construct a loss term corresponding to the keypoint based on a training predicted position corresponding to the keypoint, and set a corresponding second loss weight for the loss term corresponding to the keypoint, where the first loss weight is greater than 0 and the second loss weight is equal to 0; and construct a target loss function based on a loss term corresponding to each keypoint in the training keypoint detection result and a corresponding loss weight.

[0076] As can be seen from the description of step 202, in the present embodiment, when determining the keypoint labeling result corresponding to the target training image, each object key part is judged to be a target object key part, and only the labeling position corresponding to the target object key part is determined, and only the labeling position corresponding to the target object key part is used to determine the keypoint labeling result. In this process, some keypoints corresponding to the object key parts are ignored. Although the keypoints corresponding to these object key parts may actually exist, in the present embodiment, through a series of judgment processes, the keypoints corresponding to these object key parts are deemed to be not necessarily reliable, and therefore are not added to the keypoint labeling result corresponding to the target training image. In order to prevent the keypoints corresponding to the above-mentioned ignored object key parts from being mistakenly recognized as not actually existing when constructing the target loss function, which may lead to erroneous error backpropagation and affect the performance of the trained target object keypoint detection model, in the present embodiment, a weighted mean squared error (MSE) loss function is innovatively designed as the target loss function.

[0077] Specifically, for each keypoint in the training keypoint detection result, the server first judges whether the object key part corresponding to the keypoint is any of the target object key parts in the keypoint labeling result. That is, it judges whether the labeling position of the object key part corresponding to the keypoint exists in the keypoint labeling result. If Yes, the server constructs a loss term corresponding to the keypoint based on the difference between the training predicted position corresponding to the keypoint and the labeling position of the object key part corresponding to the keypoint in the keypoint labeling result, and sets the loss term to a first loss weight greater than 0 (e.g., 1). If No, the server constructs a loss term corresponding to the keypoint based on the training predicted position corresponding to the keypoint and a predetermined reference labeling position (which may be set randomly), and sets the loss term to a second loss weight equal to 0. Then, the server constructs a target loss function based on the loss terms corresponding to each keypoint in the training keypoint detection result and the corresponding loss weights.

[0078] The specific formula for constructing the target loss function is shown in Equation (1).

number

[0079] TIFF0007635466000002.tif103170

[0080] Optionally, when the target training images are video frames in a target video, the server can also construct a first reference loss function by focusing on the continuity of time series, and jointly train a target object keypoint detection model based on the target loss function and the first reference loss function.

[0081] That is, the server determines a video frame adjacent to the target training image in the target video as a reference training image, and obtains a training keypoint detection result corresponding to the reference training image, where the training keypoint detection result corresponding to the reference training image is obtained by performing an object keypoint detection process on the reference training image using the target object keypoint detection model, and includes training predicted positions of keypoints corresponding to each of a plurality of object key parts in the reference training image.Then, the server can construct a first reference loss function based on the training keypoint detection results corresponding to the target training image and the reference training image.

[0082] In practical operation, there should not be a large difference between adjacent video frames of a video. This means that the degree of change of the object key part on the object to be detected in adjacent video frames is also not large. Based on this, in the present embodiment, a first reference loss function for constraining the continuity between frames is innovatively designed. That is, based on the training key point detection result for two adjacent video frames by the target object key point detection model, a first reference loss function for auxiliary training of the target object key point detection model is constructed, and the first reference loss function constrains the degree of change of the object key part position between frames not to be too large.

[0083] In a specific implementation, for each object key part, the server can determine a training predicted position of the keypoint corresponding to the object key part in the target training image as a third position, and determine a training predicted position of the keypoint corresponding to the object key part in the reference training image as a fourth position. Then, based on the distance between the third position and the fourth position, construct a loss term corresponding to the object key part, and set a loss weight corresponding to the loss term. Furthermore, the server can construct the first reference loss function based on the loss term corresponding to each object key part and the corresponding loss weight.

[0084] Exemplarily, for each object key part, the server determines the predicted position of the keypoint corresponding to the object key part in the training keypoint prediction result corresponding to the target training image as a third position, and determines the predicted position of the keypoint corresponding to the object key part in the training keypoint prediction result corresponding to the reference training image as a fourth position. Then, the server calculates the distance between the third position and the fourth position, constructs a loss term corresponding to the object key part based on the distance, and determines a loss weight corresponding to the loss term based on the distance. For example, if the distance is smaller than a third predetermined distance (for example, when a normalization process is performed for each position, the third predetermined distance can be 0.1), the server can determine the loss weight corresponding to the loss term to be 0, and if the distance is not smaller than the third predetermined distance, the server can determine the loss weight corresponding to the loss term to be 1. Of course, for the two cases where the distance is smaller than the third predetermined distance and not smaller than the third predetermined distance, the server may set other appropriate loss weights, as long as the loss weight set for the case where the distance is smaller than the third predetermined distance is smaller than the loss weight set for the case where the distance is not smaller than the third predetermined distance. Then, the server constructs the first reference loss function according to the loss terms and loss weights corresponding to each object key portion.

[0085] A specific formula for constructing the first reference loss function is shown in Equation (2).

number

[0086] TIFF0007635466000004.tif76170

[0087] In this way, by introducing the first reference loss function to constrain the continuity between frames in the above method, the trained target object keypoint detection model can be quickly converged, which can improve the model training efficiency and also improve the model performance of the trained target object keypoint detection model.

[0088] Optionally, the present embodiment can further introduce the concept of knowledge distillation into the training process of the target object keypoint detection model to further improve the model performance of the trained target object keypoint detection model.Knowledge distillation is a model training method that uses knowledge learned from a complex model (also called a teacher model) to teach a simple model (also called a student model), aiming to make the simple model have performance equivalent to the complex model, and to greatly reduce the number of parameters of the simple model compared to the complex model, thereby realizing model compression and acceleration.

[0089] That is, the server can obtain a first intermediate processing result generated when the target training image is processed by any reference object keypoint detection, and a second intermediate processing result generated when the target training image is processed by the target object keypoint detection model, and further construct a second reference loss function based on the first intermediate processing result and the second intermediate processing result.

[0090] For example, the server selects a reference object keypoint detection model that is close to the model structure of the target object keypoint detection model to be trained from the m reference object keypoint detection models, and sets it as a teacher model. Then, the target training image is processed with the teacher model to obtain an intermediate processing result generated as a first intermediate processing result. For example, when the target training image is processed with the teacher model, features extracted from the target training image in one of the feature extraction layers are obtained as the first intermediate processing result. When the server processes the target training image using the target object keypoint detection model, the server can process the target training image with the target object keypoint detection model to obtain an intermediate processing result generated as a second intermediate processing result. For example, when the target training image is processed with the target object keypoint detection model, features extracted from the target training image in one of the feature extraction layers are obtained as the second intermediate result. To supplement, the first intermediate processing result and the second intermediate processing result obtained here must have the same number of dimensions. Also, they must be obtained from the same processing stage. For example, the first intermediate processing result and the second intermediate processing result are obtained in the encoding phase of the reference object keypoint detection model and the target object keypoint detection model, respectively, and / or the first intermediate processing result and the second intermediate processing result are obtained in the decoding phase of the reference object keypoint detection model and the target object keypoint detection model, respectively. The embodiment of the present application is not limited to the first intermediate processing result and the second intermediate processing result.

[0091] After obtaining the first intermediate processing result and the second intermediate processing result, the server can construct a second reference loss function based on the difference between the first intermediate processing result and the second intermediate processing result. When training the target object keypoint detection model with the second reference loss function, reference The training goal is to minimize the loss function and adjust the model parameters of the target object keypoint detection model that needs to be trained.

[0092] In this way, the above method introduces a second reference loss function constructed based on the concept of knowledge distillation, and uses a reference object keypoint detection model with excellent model performance to guide the training of the target object keypoint detection model, which contributes to improving the model performance of the target object keypoint detection model, and also contributes to quickly convergence of the trained target object keypoint detection model, and improves the efficiency of model training.

[0093] Step 2043: Adjust model parameters of the target object keypoint detection model based on the target loss function.

[0094] In step 2042, the server constructs a target loss function for training the target object keypoint detection model, and then adjusts the model parameters of the target object keypoint detection model to promote convergence of the target object keypoint detection model, with the minimization of the target loss function as the training goal.

[0095] In addition, if the server further introduces a first reference loss function to constrain continuity between frames, the server can adjust the model parameters of the target object keypoint detection model based on the target loss function and the first reference loss function.

[0096] Specifically, the server can construct a total loss function for the model training stage based on the target loss function and the first reference loss function according to the following equation (3): L=L mse +λ L consistency ……(3)

[0097] where L is the total loss function in the model training stage, L mse is the target loss function, L consistency is the first reference loss function, and λ is a weight that is set in advance for the first reference loss function.

[0098] Furthermore, the minimization of the total loss function is set as a training goal, and model parameters of the target object keypoint detection model are adjusted to promote convergence of the target object keypoint detection model.

[0099] In addition, if the server further introduces a second reference loss function constructed based on the concept of knowledge distillation, the server can adjust the model parameters of the target object keypoint detection model based on the target loss function and the second reference loss function.

[0100] Specifically, the server can construct a total loss function of the model training stage based on the target loss function and the second reference loss function in a manner similar to the above-mentioned method of constructing a total loss function based on the target loss function and the first reference loss function, and then adjust model parameters of the target object keypoint detection model with the minimization of the total loss function as a training goal to promote the convergence of the target object keypoint detection model.

[0101] If the server simultaneously introduces the first and second reference loss functions, the server can adjust the model parameters of the target object keypoint detection model according to the target loss function, the first and second reference loss functions. That is, the server sets weights corresponding to the first and second reference loss functions, respectively, and further constructs a total loss function in the model training stage according to the target loss function, the first reference loss function and its corresponding weight, and the second reference loss function and its corresponding weight. Then, the server trains the model to minimize the total loss function. the goal Then, the model parameters of the target object keypoint detection model are adjusted to promote convergence of the target object keypoint detection model.

[0102] The server repeatedly executes the above steps 2041 to 2043 based on different target training images to realize the repeated training of the target object keypoint detection model until the target object keypoint detection model satisfies a model training end condition. At this time, the target object keypoint detection model obtained by training becomes a target object keypoint detection model that can actually be used. Here, the training end condition may be, for example, that the model performance of the trained target object keypoint detection model satisfies a predetermined condition (for example, the detection accuracy rate of the target object keypoint detection model reaches a predetermined accuracy rate threshold, etc.). Also, for example, it may be that the model performance of the target object keypoint detection model does not improve significantly any more (for example, the target object keypoint detection models trained for several cycles are tested using a test sample set, and it is determined that the difference in detection accuracy of the target object keypoint detection models trained for several cycles is smaller than a predetermined difference threshold, etc.). Furthermore, for example, it may be that the number of repeated training of the target object keypoint detection model reaches a predetermined number. The present application does not place any restrictions on the training end condition.

[0103] When the training sample for training the target object keypoint detection model to be actually used is generated by the above model training method, a plurality of reference object keypoint detection models with complex structures are used to perform object keypoint detection processing on the target training image, respectively, to obtain a plurality of keypoint detection results. Then, based on the principle that the predicted positions of the same object key parts in each keypoint detection result should basically be the same, for each object key part, it is evaluated whether the position prediction result of each reference object keypoint detection model for the object key part is reliable based on the predicted position of the keypoint corresponding to the object key part in each keypoint detection result. That is, it is determined whether the object key part is a target object key part. If it is determined that the position prediction result of each reference object keypoint detection model for the object key part is reliable, further, a labeling position, i.e., a pseudo label, corresponding to the target object key part is determined. Then, a training sample is constructed using the target training image and the labeling positions corresponding to each target object key part therein. In this way, the object key parts whose position prediction results are unreliable are ignored, and only the labeling positions corresponding to the object key parts whose position prediction results are reliable are used to determine the pseudo labels, so that the noise present in the determined pseudo labels can be effectively reduced and the accuracy of the pseudo labels can be improved. Therefore, if a target object keypoint detection model with a relatively simple structure that is to be actually used is trained based on training samples including such pseudo labels, the performance of the trained target object keypoint detection model can be guaranteed to a certain extent.

[0104] In order to make the model training method provided in the embodiments of the present application easier to understand, the following will comprehensively and illustratively describe the model training method by taking as an example the case of using the model training method to train a human body keypoint detection model for detecting a player's posture in an immersive game.

[0105] A haptic game refers to a game experienced through the body. It is a new type of electronic game that breaks through the traditional operation method of simply inputting with controller buttons, and operates through changes in limb movements. In the process of playing a haptic game, the camera takes pictures including the player's limbs, and transmits the pictures to a game server in the background. The game server uses a human body keypoint detection model to detect the positions of the important joint points of the player's skeleton in the pictures, and based on this, recognizes the limb movements made by the player, and executes the corresponding game logic. Figure 5 shows the overall implementation architecture of the haptic game.

[0106] TIFF0007635466000005.tif85170

[0107] The core modules of the technical solution provided in the embodiment of this application are mainly two: one is a pseudo-label generation network, which uses multiple existing complex and high-performance human body keypoint detection models to perform human body keypoint detection processing on video frames in unlabeled videos, and generates pseudo-labels corresponding to the video frames according to the detection results of each human body keypoint detection model, which are used in the subsequent model training process; the other is an inter-frame continuity constraint loss function, which is based on the premise that the degree of change in the person's movements between video frames should not be large, and designs an inter-frame continuity loss function to enable the model to realize unsupervised learning.

[0108] Next, we will explain each of the two core modules in detail.

[0109] As shown in FIG. 6, the pseudo-label generation network includes a human body keypoint prediction module and a prediction result integration module.

[0110] TIFF0007635466000006.tif46170

[0111] The prediction result integration module is used to complete the integration of m prediction results by the following steps. In the integration process, the embodiment of the present application performs strict filtering processing on the prediction results in the spatial dimension and the time dimension respectively, to ensure that the generated pseudo label has higher accuracy. The specific integration method is as follows:

[0112] TIFF0007635466000007.tif34170

[0113] TIFF0007635466000008.tif94170

[0114] TIFF0007635466000009.tif81170

[0115] (4) By going through the above spatial and temporal filtering processes, a highly accurate pseudo label can be obtained. However, the above process results in many ignored human body parts. These human body parts actually exist, and in this application, the detection results of the human body keypoint detection model for them are deemed unreliable, so they are simply not input into the subsequent model training process. If model training is performed based on the MSE loss function as it is, it may misidentify these keypoints as not actually existing, leading to erroneous error backpropagation, which may affect the performance of the trained human body keypoint detection model. To solve this problem, in this embodiment, a weighted MSE loss function is designed, and when a human body part in the pseudo label is ignored, the loss weight of the loss term corresponding to the human body part is set to 0. In this way, erroneous backpropagation can be prevented. The weighted MSE loss function proposed in this embodiment is as follows:

[0116]

number

[0117] TIFF0007635466000011.tif100170

[0118] In addition, in the embodiment of the present application, it is found that there is no significant difference between two adjacent video frames in a video, that is, the degree of change of the human body parts in the adjacent video frames is not large. Based on this finding, the present application further designs a constraint function of inter-frame continuity for realizing unsupervised learning of the human body keypoint detection model, so as to ensure that the jump width of the keypoint coordinates between frames is not too large when the human body keypoint detection model predicts the video. The loss function is specifically as follows:

[0119]

number

[0120] TIFF0007635466000013.tif85170

[0121] To summarize the above, the loss function for the entire training stage of the scheme provided in the present embodiment is as follows: L=L mse +λ L consistency

[0122] However, λ is set in advance to L consistency is the weight set to

[0123] Then, the server trains the human body keypoint detection model that needs to be trained based on the loss function L.

[0124] We tested the human body keypoint detection model trained using the above method on the COCO-wholebody human body keypoint detection public test set, and the mean average precision (mAP) index for all classes increased from 0.457 to 0.521.

[0125] The present inventors further conducted a comparison test between a human body keypoint detection model currently applied to a immersive game and a human body keypoint detection model trained by the above method. The results of the comparison test are shown in FIG. 7. (a) is a detection result determined by a human body keypoint detection model trained by the scheme provided in the present embodiment, and (b) is a detection result determined by a conventional human body keypoint detection model. The comparison shows that (a) is more accurate than (b), and the keypoints corresponding to the left and right feet in (a) are not reversed in position, but the keypoints corresponding to the left and right feet in (b) are reversed in position.

[0126] It should be noted that the model training method provided in the embodiment of the present application can be used not only to train the human body keypoint detection model applied to the scene of the immersive game, but also to train the object keypoint detection model of other scenes. For example, the function of adding special effects to the human body or animals in the scene of the video shooting often requires the object keypoint detection model to detect the keypoints of the object in the image. The scheme provided in the embodiment of the present application can also be used to train the object keypoint detection model in such a scene. Here, there is no limitation on the application scene to which the scheme provided in the embodiment of the present application is applied.

[0127] In addition, when the above examples of the present application are implemented in specific products or technologies, data related to user information (e.g., images captured by a camera) in the specific embodiments of the present application requires permission or consent from the user, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant country or region.

[0128] Regarding the above-described model training method, the present application further provides a corresponding model training device, so that the above-described model training method can be practically applied and realized.

[0129] Please refer to FIG. 8. FIG. 8 is a structural schematic diagram of a model training apparatus 800 corresponding to the model training method shown in FIG. 2 above. As shown in FIG. 8, the model training apparatus 800 includes a keypoint detection module 801, a part labeling module 802, a sample generation module 803, and a model training module 804. The keypoint detection module 801 is used to perform object keypoint detection processing on a target training image by m reference object keypoint detection models, respectively, to obtain keypoint detection results corresponding to the m reference object keypoint detection models. The keypoint detection results include predicted positions of keypoints corresponding to a plurality of object key parts in the target training image, where m is an integer greater than 1. The part labeling module 802 is used to determine whether the object key part is a target object key part based on the predicted position of the keypoint corresponding to the object key part in the m keypoint detection results for each of the object key parts, and if Yes, to determine a labeling position corresponding to the target object key part based on the predicted position of the keypoint corresponding to the target object key part in the m keypoint detection results. The sample generation module 803 is used to set the target training image and the corresponding keypoint labeling result as a training sample. The keypoint labeling result includes a labeling position corresponding to each of the target object key parts. The model training module 804 is used to train a target object keypoint detection model based on the training sample.

[0130] Optionally, based on the model training apparatus shown in FIG. 8 , the part labeling module 802 is specifically used for: determining a reference position corresponding to the target object key part according to the predicted position of the keypoint corresponding to the target object key part in the m keypoint detection results; determining, for each of the keypoint detection results, a distance between the predicted position of the keypoint corresponding to the object key part and the reference position corresponding to the object key part; and judging whether the keypoint detection result belongs to the target keypoint detection result corresponding to the object key part according to the distance; counting the number of targets of the target keypoint detection result corresponding to the object key part in the m keypoint detection results; and judging that the object key part is a target object key part if the number of targets exceeds a predetermined number; and judging that the object key part is an invalid object key part if the number of targets does not exceed the predetermined number.

[0131] Optionally, based on the model training apparatus shown in FIG. 8, the part labeling module 802 is specifically used for identifying, for each target keypoint detection result corresponding to the target object key part, a predicted position of the keypoint corresponding to the target object key part as a target predicted position corresponding to the target object key part, and determining a labeling position corresponding to the target object key part based on each target predicted position corresponding to the target object key part.

[0132] Optionally, on the basis of the model training apparatus shown in FIG. 8, please refer to FIG. 9. FIG. 9 is a structural schematic diagram of another model training apparatus 900 provided in an embodiment of the present application. As shown in FIG. 9, when the target training image is a video frame in a target video, the apparatus further includes a reference data acquisition module 901 and a labeling result screening module 902. The reference data acquisition module 901 is used to identify a video frame adjacent to the target training image in the target video as a reference training image, and obtain a keypoint labeling result corresponding to the reference training image. The labeling result screening module 902 is used to identify, for each of the target object key portions, a labeling position in the keypoint labeling result corresponding to the target training image as a first position, and identify a labeling position in the keypoint labeling result corresponding to the reference training image as a second position, and determine whether to retain the first position in the keypoint labeling result corresponding to the target training image and whether to retain the second position in the keypoint labeling result corresponding to the reference training image according to the distance between the first position and the second position.

[0133] Optionally, based on the model training apparatus shown in FIG. 8 , the keypoint detection module 801 is specifically used for, for each of the reference object keypoint detection models, performing object keypoint detection processing on the target training image through the reference object keypoint detection model to determine the confidence that each pixel in the target training image belongs to the object key portion, and based on the pixels in the target training image whose confidence that they belong to the object key portion is higher than a predetermined confidence threshold, determining a keypoint detection result corresponding to the reference object keypoint detection model.

[0134] Optionally, on the basis of the model training apparatus shown in FIG. 8, please refer to FIG. 10. FIG. 10 is a structural schematic diagram of another model training apparatus 1000 provided in an embodiment of the present application. As shown in FIG. 10, the model training module 804 includes a keypoint detection submodule 1001, a loss function construction submodule 1002, and a parameter adjustment submodule 1003. The keypoint detection submodule 1001 is used for performing object keypoint detection processing on the target training image in the training sample through the target object keypoint detection model to obtain a training keypoint detection result corresponding to the target training image. The training keypoint detection result includes a training predicted position of a keypoint corresponding to each of a plurality of object key parts. The loss function construction submodule 1002 is used for constructing a target loss function based on the training keypoint detection result corresponding to the target training image and the keypoint labeling result in the training sample. The parameter adjustment submodule 1003 is used for adjusting a model parameter of the target object keypoint detection model based on the target loss function.

[0135] Optionally, based on the model training device shown in Fig. 10, the loss function construction submodule 1002 specifically determines, for each keypoint in the training keypoint detection result, whether the object key part corresponding to the keypoint is a target object key part in the keypoint labeling result; if Yes, construct a loss term corresponding to the keypoint according to the training predicted position corresponding to the keypoint and the labeling position of the target object key part corresponding to the keypoint, and set a corresponding first loss weight for the loss term corresponding to the keypoint; if No, construct a loss term corresponding to the keypoint according to the training predicted position corresponding to the keypoint, and set a corresponding second loss weight for the loss term corresponding to the keypoint, and use the loss term corresponding to each keypoint in the training keypoint detection result and the corresponding loss weight to construct the target loss function. The first loss weight is greater than 0, and the second loss weight is equal to 0.

[0136] Optionally, based on the model training apparatus shown in FIG. 10 , the loss function construction submodule 1002 is further used to identify a video frame adjacent to the target training image in the target video as a reference training image, obtain a training keypoint detection result corresponding to the reference training image, and construct a first reference loss function based on the training keypoint detection results corresponding to the target training image and the reference training image respectively, and the parameter adjustment submodule 1003 is specifically used for adjusting model parameters of the target object keypoint detection model based on the target loss function and the first reference loss function.

[0137] Optionally, the loss function construction submodule 1002 is specifically used for: for each of the object key regions, determining a training predicted position of a keypoint corresponding to the object key region in the target training image as a third position; determining a training predicted position of a keypoint corresponding to the object key region in the reference training image as a fourth position; constructing a loss term corresponding to the object key region based on the distance between the third position and the fourth position; setting a loss weight corresponding to the loss term; and constructing the first reference loss function based on the loss terms corresponding to each of the object key regions and the corresponding loss weights.

[0138] Optionally, based on the model training apparatus shown in FIG. 10 , the loss function construction submodule 1002 is further used to obtain a first intermediate processing result generated when the target training image is processed by the reference object keypoint detection model and a second intermediate processing result generated when the target training image is processed by the target object keypoint detection model, and construct a second reference loss function based on the first intermediate processing result and the second intermediate processing result, and the parameter adjustment submodule 1003 is specifically used for adjusting model parameters of the target object keypoint detection model based on the target loss function and the second reference loss function.

[0139] When the model training device generates training samples for training a target object keypoint detection model to be actually used, a plurality of reference object keypoint detection models with complex structures are used to perform object keypoint detection processing on target training images to obtain a plurality of keypoint detection results. Then, based on the principle that the predicted positions of the same object key parts in each keypoint detection result should basically match, for each object key part, the device evaluates whether the position prediction result of each reference object keypoint detection model for the object key part is reliable based on the predicted position of the keypoint corresponding to the object key part in each keypoint detection result. That is, it is determined whether the object key part is a target object key part. If it is determined that the position prediction result of each reference object keypoint detection model for the object key part is reliable, the device further determines a labeling position, i.e., a pseudo label, corresponding to the target object key part. Then, the target training images and the labeling positions corresponding to each target object key part therein are used to construct training samples. In this way, the object key parts whose position prediction results are unreliable are ignored, and only the labeling positions corresponding to the object key parts whose position prediction results are reliable are used to determine the pseudo labels, so that the noise present in the determined pseudo labels can be effectively reduced and the accuracy of the pseudo labels can be improved. Therefore, if a target object keypoint detection model with a relatively simple structure that is to be actually used is trained based on training samples including such pseudo labels, the performance of the trained target object keypoint detection model can be guaranteed to a certain extent.

[0140] The present embodiment further provides a computer device for training a model, which may be a terminal device or a server. The following will focus on hardware implementation and introduce the terminal device and server provided in the present embodiment.

[0141] Please refer to FIG. 11. FIG. 11 is a structural schematic diagram of a terminal device provided in an embodiment of the present application. As shown in FIG. 11, for convenience of explanation, only parts related to the embodiment of the present application are shown. For details of specific techniques not described, please refer to the method part of the embodiment of the present application. The terminal may include any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a point of sales (POS) terminal, an in-vehicle computer, etc., but the terminal is taken as an example.

[0142] Figure 11 is a block diagram of a part of the structure of a computer related to a terminal provided in an embodiment of the present application. As shown in Figure 11, the computer includes components such as an RF (Radio Frequency) circuit 1110, a memory 1120, an input unit 1130 (including a touch panel 1131 and other input devices 1132), a display unit 1140 (including a display panel 1141), a sensor 1150, an audio circuit 1160 (which may be connected to a speaker 1161 and a microphone 1162), a WiFi (registered trademark) module 1170, a processor 1180, and a power supply 1190. As can be understood by those skilled in the art, the structure of the computer shown in Figure 11 is not intended to limit the computer, and the computer may include more or fewer components than those shown, any components may be combined, or the components may be arranged differently.

[0143] The memory 1120 is used to store software programs and modules, and the processor 1180 executes various functions, applications, and data processing of the computer by executing the software programs and modules stored in the memory 1120. The memory 1120 mainly includes a program storage area and a data storage area, and the program storage area can store an operating system and an application program required for at least one function (e.g., a voice playback function, an image playback function, etc.). The data storage area can store data generated in accordance with the use of the computer (e.g., voice data, a telephone directory, etc.). The memory 1120 may include a high-speed random access memory, or may include at least one non-volatile memory such as a magnetic disk storage device, a flash storage device, or other volatile solid-state storage device.

[0144] The processor 1180 is the control center of the computer, and is connected to each part of the entire computer through various interfaces and lines, and performs overall supervision of the computer by operating or executing software programs and / or modules stored in the memory 1120, and calling up data stored in the memory 1120 to execute various functions of the computer and process data. Optionally, the processor 1180 may include one or more processing units. The processor 1180 can be integrated with an application processor and a modulation / demodulation processor. The application processor mainly processes the operation system, user interface, application programs, etc., and the modulation / demodulation processor mainly processes wireless communication. It should be noted that the modulation / demodulation processor does not have to be integrated with the processor 1180.

[0145] In the embodiment of the present application, the processor 1180 included in the terminal further has the following functions: Perform object keypoint detection processing on a target training image using m (m is an integer greater than 1) reference object keypoint detection models, respectively, to obtain keypoint detection results corresponding to the m reference object keypoint detection models. The keypoint detection results include predicted positions of keypoints corresponding to each of a plurality of object key parts in the target training image. For each of the object key parts, determine whether the object key part is a target object key part based on the predicted positions of the keypoints corresponding to the object key part in the m keypoint detection results, and if Yes, determine a labeling position corresponding to the target object key part based on the predicted positions of the keypoints corresponding to the target object key part in the m keypoint detection results. The target training image and the corresponding keypoint labeling results including the labeling positions corresponding to each of the target object key parts are training samples. Train a target object keypoint detection model based on the training samples.

[0146] Optionally, the processor 1180 is further used to execute steps of any implementation of the model training method provided in the embodiments of the present application.

[0147] Please refer to FIG. 12. FIG. 12 is a structural schematic diagram of a server 1200 provided in an embodiment of the present application. The server 1200 may have relatively large differences due to different settings or performances, but may include one or more central processing units (CPUs) 1222 (e.g., one or more processors) and memory 1232, and may include one or more storage media 1230 (e.g., one or more mass storage devices) that store application programs 1242 or data 1244. The memory 1232 and the storage media 123 may be temporary or non-temporary memories. The program in the storage medium 1230 includes one or more modules (not shown), and each module includes a series of command operations in the server. In addition, the CPU 1222 is configured to communicate with the storage medium 1230 and execute the series of command operations in the storage medium 1230 on the server 1200.

[0148] Server 1200 may further include one or more power sources 1226, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1258, and / or one or more operating systems, such as Windows Server®, Mac OS X®, Unix®, Linux®, FreeBSD®, etc.

[0149] In the above embodiment, the steps performed by the server are based on the server structure shown in Fig. 12. The CPU 1222 is used to perform the following steps:

[0150] the step of performing object keypoint detection processing on a target training image using m (m is an integer greater than 1) reference object keypoint detection models to obtain keypoint detection results corresponding to each of the m reference object keypoint detection models, the keypoint detection results including predicted positions of keypoints corresponding to each of a plurality of object key features in the target training image; the step of determining, for each of the object key features, whether the object key feature is a target object key feature based on the predicted positions of the keypoints corresponding to the object key feature in the m keypoint detection results, and if Yes, determining a labeling position corresponding to the target object key feature based on the predicted positions of the keypoints corresponding to the target object key feature in the m keypoint detection results; the step of using the target training images and the corresponding keypoint labeling results including labeling positions corresponding to each of the target object key features as training samples; and the step of training a target object keypoint detection model based on the training samples.

[0151] Optionally, the CPU 1222 is further used to execute steps of any implementation of the model training method provided in the present embodiment.

[0152] The present embodiment further provides a computer-readable storage medium for storing a computer program, which is used to execute any one of the embodiments of the model training method described in each of the above embodiments.

[0153] The present embodiment further provides a computer program product or a computer program comprising computer instructions stored in a computer-readable storage medium, the computer instructions being read by a processor of a computing device from the computer-readable storage medium, and the processor executing the computer instructions, thereby causing the computing device to perform any one of the embodiments of the model training method described in each of the above embodiments.

[0154] As will be understood by those skilled in the art, for the sake of clarity and conciseness, the specific operation processes of the systems, devices and units described above can be referenced to the corresponding processes in the above method embodiments, and therefore will not be described repeatedly here.

[0155] It is understood that the systems, devices, and methods disclosed in some embodiments provided in the present application can be realized in other forms. For example, the above device embodiments are merely exemplary, and the division of the units is merely a division of a kind of logical function, and other division methods are possible when actually embodied. For example, multiple units or assemblies may be combined or integrated into another system, or some features may be omitted or not implemented. In addition, the couplings, direct couplings, or communication connections between the indicated or discussed mutually may be indirect couplings or communication connections via some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0156] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, and may be located in one place or distributed among multiple network units. Some or all of the units may be selected to achieve the purpose of the scheme of the present embodiment according to actual needs.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically independent, or two or more units may be integrated into one unit. The integrated units may be realized in the form of hardware or software functional units.

[0158] The integrated unit may be realized in the form of a software functional unit and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essence of the technical solution of the present application, that is, the part that contributes to the prior art, or all or part of the technical solution may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media capable of storing a computer program, such as a USB, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk, or an optical disk.

[0159] In this application, "at least one (item)" refers to one or more, and "multiple" refers to two or more. "And / or" is used to indicate a related relationship between related objects, and indicates that there are three types of relationships. For example, "A and / or B" indicates three types of situations: a situation in which only A exists, a situation in which only B exists, and a situation in which A and B exist simultaneously. However, A and B may be singular or plural. The symbol " / " generally indicates that the objects associated with the preceding and following are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or multiple items. For example, "at least one of a, b, or c" refers to a, b, c, a and b, a and c, b and c, or a, b, and c. However, a, b, and c may be singular or plural.

[0160] As mentioned above, the above examples are for illustrating the technical solutions of the present application, but are not intended to be limiting. Although the present application has been described in detail with reference to the above examples, those skilled in the art will understand that the technical schemes described in the above examples can be modified or some of the technical features therein can be replaced with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the respective embodiments of the present application.

Claims

1. 1. A method of model training implemented by a computing device, comprising: performing object keypoint detection processing on the target training image using m (m is an integer greater than 1) reference object keypoint detection models to obtain keypoint detection results corresponding to the m reference object keypoint detection models, the keypoint detection results including predicted positions of keypoints corresponding to a plurality of object key sites in the target training image; For each of the object key parts, it is determined whether the object key part is a target object key part based on the predicted position of the key point corresponding to the object key part in the m key point detection results, and if Yes, it is determined a labeling position corresponding to the target object key part based on the predicted position of the key point corresponding to the target object key part in the m key point detection results; The target training image and the corresponding keypoint labeling result including the labeling positions corresponding to each of the target object key parts are taken as training samples; training a target object keypoint detection model based on the training samples. Model training methods.

2. The step of determining whether the object key part is a target object key part based on a predicted position of a key point corresponding to the object key part in the m key point detection results, determining a reference position corresponding to the object key portion based on a predicted position of a key point corresponding to the object key portion in the m key point detection results; For each of the keypoint detection results, determining a distance between a predicted position of the keypoint corresponding to the object key part and a reference position corresponding to the object key part, and judging whether the keypoint detection result belongs to a target keypoint detection result corresponding to the object key part based on the distance; A step of counting the number of targets of the target keypoint detection results corresponding to the object key portion in the m keypoint detection results; determining that the object key part is a target object key part if the target number exceeds a predetermined number, and determining that the object key part is a non-target object key part if the target number does not exceed the predetermined number. The method of training a model according to claim 1 .

3. The step of determining a labeling position corresponding to the target object key portion based on a predicted position of a key point corresponding to the target object key portion in the m key point detection results includes: For each target keypoint detection result corresponding to the target object key portion, determining a predicted position of the keypoint corresponding to the target object key portion as a target predicted position corresponding to the target object key portion; determining a labeling position corresponding to the target object key portion based on each target predicted position corresponding to the target object key portion; The method of training a model according to claim 2.

4. If the target training images are video frames in a target video, further comprising: identifying a video frame adjacent to the target training image in the target video as a reference training image, and obtaining a keypoint labeling result corresponding to the reference training image; for each of the target object key portions, identifying a labeling position in the keypoint labeling result corresponding to the target training image as a first position, identifying a labeling position in the keypoint labeling result corresponding to the reference training image as a second position, and determining whether to retain the first position in the keypoint labeling result corresponding to the target training image and whether to retain the second position in the keypoint labeling result corresponding to the reference training image based on a distance between the first position and the second position. The method of training a model according to claim 1 .

5. The step of performing object keypoint detection processing on the target training image by m reference object keypoint detection models to obtain keypoint detection results corresponding to the m reference object keypoint detection models, for each of the reference object keypoint detection models, determining a confidence level that each pixel in the target training images belongs to the object key portion by performing object keypoint detection processing on the target training images using the reference object keypoint detection model; and identifying keypoint detection results corresponding to the reference object keypoint detection model based on pixels in the target training images that have a confidence level that they belong to the object key region higher than a predetermined confidence threshold. The method of training a model according to claim 1 .

6. The step of training a target object keypoint detection model based on the training samples includes: performing an object keypoint detection process on the target training images in the training sample using the target object keypoint detection model to obtain training keypoint detection results corresponding to the target training images, where the training keypoint detection results include training predicted positions of keypoints corresponding to each of a plurality of object key features; constructing a target loss function based on the training keypoint detection results corresponding to the target training images and the keypoint labeling results in the training samples; and adjusting model parameters of the target object keypoint detection model based on the target loss function. The method of training a model according to claim 1 .

7. The step of constructing a target loss function based on the training keypoint detection results corresponding to the target training images and the keypoint labeling results in the training samples includes: For each keypoint in the training keypoint detection result, determine whether an object key part corresponding to the keypoint is a target object key part in the keypoint labeling result. If yes, construct a loss term corresponding to the keypoint based on a training predicted position corresponding to the keypoint and a labeled position of the target object key part corresponding to the keypoint, and set a corresponding first loss weight to the loss term corresponding to the keypoint. If no, construct a loss term corresponding to the keypoint based on a training predicted position corresponding to the keypoint, and set a corresponding second loss weight to the loss term corresponding to the keypoint. constructing the target loss function based on a loss term corresponding to each keypoint in the training keypoint detection results and a corresponding loss weight; the first loss weight is greater than 0 and the second loss weight is equal to 0; The method of claim 6 .

8. If the target training images are video frames in a target video, further comprising: identifying a video frame adjacent to the target training image in the target video as a reference training image, and obtaining a training keypoint detection result corresponding to the reference training image; constructing a first reference loss function based on training keypoint detection results corresponding to the target training images and the reference training images, respectively; The step of adjusting model parameters of the target object keypoint detection model based on the target loss function comprises: adjusting model parameters of the target object keypoint detection model based on the target loss function and the first reference loss function. The method of training a model according to claim 6.

9. The step of constructing a first reference loss function based on training keypoint detection results corresponding to the target training images and the reference training images, respectively, comprises: For each of the object key features, determining a training predicted position of a key point corresponding to the object key feature in the target training images as a third position, determining a training predicted position of a key point corresponding to the object key feature in the reference training images as a fourth position, constructing a loss term corresponding to the object key feature based on a distance between the third position and the fourth position, and setting a loss weight corresponding to the loss term; and constructing the first reference loss function based on a loss term corresponding to each of the object key parts and a corresponding loss weight therefor; The method of training a model according to claim 8.

10. obtaining a first intermediate processing result generated when the target training image is processed by the reference object keypoint detection model, and a second intermediate processing result generated when the target training image is processed by the target object keypoint detection model; constructing a second reference loss function based on the first intermediate processing result and the second intermediate processing result; The step of adjusting model parameters of the target object keypoint detection model based on the target loss function comprises: adjusting model parameters of the target object keypoint detection model based on the target loss function and the second reference loss function. A method for training a model according to claim 6 or 8.

11. a keypoint detection module for performing object keypoint detection processing on a target training image using m (m is an integer greater than 1) reference object keypoint detection models to obtain keypoint detection results corresponding to the m reference object keypoint detection models, the keypoint detection results including predicted positions of keypoints corresponding to a plurality of object key sites in the target training image; For each of the object key parts, a part labeling module is used to determine whether the object key part is a target object key part based on the predicted position of the key point corresponding to the object key part in the m key point detection results, and if Yes, to determine a labeling position corresponding to the target object key part based on the predicted position of the key point corresponding to the target object key part in the m key point detection results; a sample generation module for taking the target training images and corresponding keypoint labeling results including labeling positions corresponding to each of the target object key features as training samples; and a model training module adapted to train a target object keypoint detection model based on the training samples. Model training equipment.

12. Specifically, the site labeling module: determining a reference position corresponding to the object key portion based on a predicted position of the key point corresponding to the object key portion in the m key point detection results; For each of the keypoint detection results, determine a distance between a predicted position of a keypoint corresponding to the object key part and a reference position corresponding to the object key part, and determine whether the keypoint detection result belongs to a target keypoint detection result corresponding to the object key part based on the distance; Counting the number of targets of the target keypoint detection results corresponding to the object key portion in the m keypoint detection results; When the number of targets exceeds a predetermined number, the object key part is determined to be a target object key part, and when the number of targets does not exceed the predetermined number, the object key part is determined to be a non-target object key part. A model training apparatus according to claim 11.

13. A processor and a memory are included. the memory is used to store a computer program; The processor is adapted to execute the model training method according to any one of claims 1 to 9 based on the computer program. Computer equipment.

14. A computer-readable storage medium for storing a computer program, the computer program being used to execute a model training method according to any one of claims 1 to 9. A computer-readable storage medium.

15. On the computer, To execute the model training method according to any one of claims 1 to 9, Computer program.

Images