Training method and apparatus for key point prediction model, device, medium and product

Abstract

A training method for a key point prediction model, the method being executed by a computer device and comprising: acquiring a sample image containing a sample object, the sample image being labeled with reference coordinates respectively corresponding to n key points of the sample object, and n being a positive integer (310); by means of a sample prediction model, predicting the probability distribution of the positions of the key points of the sample object in the sample image, so as to obtain a plurality of probability distribution parameters corresponding to each of the n key points, the plurality of probability distribution parameters corresponding to the j-th key point being parameters obtained by predicting the j-th key point on the basis of different image analysis aspects, and the plurality of probability distribution parameters corresponding to the j-th key point being used for representing mixed probability distribution of the position of the j-th key point, j≤n and j being a positive integer (320); and on the basis of reference coordinates corresponding to the j-th key point and the plurality of probability distribution parameters corresponding to the j-th key point, training the sample prediction model so as to obtain a key point prediction model, the key point prediction model being used for predicting coordinates of key points of objects to be recognized in images (330).

Description

Method, device, equipment, medium and product for training key point prediction model

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 202410823324.9, filed on June 25, 2024, and entitled "Method, device, equipment, medium and product for training key point prediction model", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] Embodiments of the present application relate to the field of artificial intelligence, and in particular to a method, device, equipment, medium and product for training a key point prediction model. BACKGROUND

[0004] Pose estimation aims to predict predefined key points on an object to be recognized, and is a basic task in computer vision. It is an important preprocessing operation for many downstream tasks such as motion capture. The pose estimation task can usually be achieved by training a key point prediction model on training samples.

[0005] In related technologies, a key point prediction model on training samples is trained by a method based on heat maps. A high-resolution heat map corresponding to a sample image can be generated by the key point prediction model on training samples. In the heat map, each pixel point corresponds to a value, which is used to represent the probability that a key point belongs to the pixel point. Then, the predicted coordinates of the key points are determined based on the heat map. Finally, the model is trained based on the difference between the predicted coordinates of the key points and the reference coordinates.

[0006] However, in the training scheme of the key point prediction model on training samples based on heat maps, each key point corresponds to a high-resolution heat map, and the amount of data that the model needs to predict is large, resulting in low training efficiency. SUMMARY

[0007] Embodiments of the present application provide a method, device, equipment, medium and product for training a key point prediction model, which can improve the training efficiency of the key point prediction model. The technical solution is as follows.

[0008] In one aspect, a method for training a key point prediction model is provided, which is executed by a computer device, and the method comprises:

[0009] obtaining a sample image containing a sample object, the sample image being labeled with reference coordinates corresponding to n key points of the sample object, n being a positive integer;

[0010] predict, by a sample prediction model, a probability distribution of a position of a key point of the sample object in the sample image, to obtain a plurality of probability distribution parameters corresponding to the n key points respectively, wherein a plurality of probability distribution parameters corresponding to a jth key point are parameters predicted based on different image analysis angles for the jth key point, and the plurality of probability distribution parameters corresponding to the jth key point are used to represent a mixed probability distribution of a position of the jth key point, j≤n and j is a positive integer; and

[0011] train, based on the reference coordinates corresponding to the jth key point and the plurality of probability distribution parameters corresponding to the jth key point, the sample prediction model to obtain a key point prediction model, wherein the key point prediction model is used to predict key point coordinates of a to-be-identified object in an image.

[0012] In another aspect, a device for training a key point prediction model is provided, and the device comprises:

[0013] a obtaining module configured to obtain a sample image containing a sample object, wherein the sample image is labeled with reference coordinates corresponding to n key points of the sample object respectively, and n is a positive integer;

[0014] a predicting module configured to predict, by a sample prediction model, a probability distribution of a position of a key point of the sample object in the sample image, to obtain a plurality of probability distribution parameters corresponding to the n key points respectively, wherein a plurality of probability distribution parameters corresponding to a jth key point are parameters predicted based on different image analysis angles for the jth key point, and the plurality of probability distribution parameters corresponding to the jth key point are used to represent a mixed probability distribution of a position of the jth key point, j≤n and j is a positive integer; and

[0015] a training module configured to train, based on the reference coordinates corresponding to the jth key point and the plurality of probability distribution parameters corresponding to the jth key point, the sample prediction model to obtain a key point prediction model, wherein the key point prediction model is used to predict key point coordinates of a to-be-identified object in an image.

[0016] In another aspect, a computer device is provided, which comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the above-mentioned method for training a key point prediction model.

[0017] In another aspect, a computer readable storage medium is provided, the storage medium having stored therein at least one instruction, at least one program, a set of codes or a set of instructions, which are loaded and executed by a processor to implement the training method of the key point prediction model.

[0018] In another aspect, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the training method of the key point prediction model.

[0019] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on the disclosed drawings.

[0021] FIG. 1 is a schematic diagram of a computer system according to an example embodiment of the present application;

[0022] FIG. 2 is a schematic diagram of a training method of a key point prediction model according to an example embodiment of the present application;

[0023] FIG. 3 is a flowchart of a training method of a key point prediction model according to an example embodiment of the present application;

[0024] FIG. 4 is a schematic diagram of a training method of a key point prediction model according to another example embodiment of the present application;

[0025] FIG. 5 is a schematic diagram of a training process of a key point prediction model according to an example embodiment of the present application;

[0026] FIG. 6 is a schematic diagram of a training method of a key point prediction model according to yet another example embodiment of the present application;

[0027] FIG. 7 is a flowchart of a palmprint recognition process according to an example embodiment of the present application;

[0028] FIG. 8 is a flowchart of an action, gesture, gait recognition process according to an example embodiment of the present application;

[0029] FIG. 9 is a schematic diagram of target detection according to an example embodiment of the present application;

[0030] FIG. 10 is a schematic diagram of region extraction according to an example embodiment of the present application;

[0031] FIG. 11 is a structural block diagram of a training device of a key point prediction model according to an example embodiment of the present application;

[0032] FIG. 12 is a structural block diagram of a training device of a key point prediction model according to another example embodiment of the present application;

[0033] FIG. 13 is a structural block diagram of a computer device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0035] The terms "first", "second", and the like are used to distinguish items or similar items having basically the same function and action, and it should be understood that there is no logical or time sequence dependency between "first" and "second", and the number and execution order are not limited.

[0036] It should be noted that, before collecting the relevant data of the user (for example, sample images, etc.), and during the process of collecting the relevant data of the user, a prompt interface, a pop-up window, or voice prompt information can be displayed, which is used to prompt the user that the relevant data of the user is being collected, so that the present application only starts to perform the relevant steps of obtaining the relevant data of the user after obtaining the confirmation operation of the user to the prompt interface or the pop-up window, otherwise (i.e., without obtaining the confirmation operation of the user to the prompt interface or the pop-up window), the relevant steps of obtaining the relevant data of the user are ended, that is, the relevant data of the user is not obtained. In other words, all the user data collected by the present application is collected under the condition that the user agrees and authorizes, and the collection, use, and processing of the relevant user data need to comply with relevant laws, regulations, and standards.

[0037] First, the terms involved in the embodiments of the present application are briefly introduced.

[0038] Artificial Intelligence (AI): is to use digital computer or digital computer controlled machine simulation, extension and expansion of human intelligence, perception of environment, acquisition of knowledge and use of knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0039] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model is also called large model, basic model, which can be widely applied to downstream tasks in various directions of artificial intelligence after fine tuning. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0040] Computer Vision (CV): Computer vision is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, follow identification and measure target machine vision, and further do image processing, so that the computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology brings important changes to the development of computer vision technology. Swin-transformer (shift window-based multi-level visual converter), ViT (Vision Transformer, vision converter), V-MOE (Vision Mixture of Experts, vision mixture of experts model) and other pre-training models in the field of vision can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition, optical character recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and map building, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0041] FIG. 1 is a structural block diagram of a computer system 100 according to an example embodiment of the present application. The computer system 100 can implement a system architecture of a training method of a key point prediction model. The computer system 100 includes a terminal 110 and a server 120.

[0042] The terminal 110 can be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (car machine), a wearable device, a PC (Personal Computer), an unmanned terminal, etc. The terminal 110 can install and run a client of a target application, which can be an application for training of a key point prediction model, or an application providing a function related to training of a key point prediction model, without limitation. In addition, the form of the target application is not limited, including but not limited to an App (Application) installed in the terminal 110, a mini program, etc., and can also be in the form of a web page.

[0043] The server 120 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs (Content Delivery Networks), and big data and artificial intelligence platforms. The server 120 can be a background server of the target application, for providing background services for the client of the target application.

[0044] The cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network to realize data calculation, storage, processing, and sharing in a wide area network or a local area network. The cloud technology is a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied based on a cloud computing business model, and can form a resource pool for on-demand use and flexible convenience. Cloud computing technology will become an important support. The background services of a technical network system require a large amount of computing and storage resources, such as video websites, picture websites, and more portal websites. With the high development and application of the Internet industry, in the future, every item can have its own identification mark and needs to be transmitted to a background system for logical processing. Different levels of data will be processed separately, and various types of industry data need strong system support, which can only be realized through cloud computing. Alternatively, the server 120 can also be implemented as a node in a blockchain system.

[0045] The terminal 110 and the server 120 can communicate through a network, such as a wired or wireless network.

[0046] The execution subject of each step of the training method of the key point prediction model provided in the embodiments of the present application can be a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. For example, the training method of the key point prediction model can be executed by the terminal 110 (e.g., the client of the target application installed and running in the terminal 110 executes the training method of the key point prediction model), or by the server 120, or by the terminal 110 and the server 120 in cooperation, which is not limited in the present application.

[0047] Those skilled in the art can know that the number of the terminal 110 can be more or less. For example, the terminal 110 can be only one, or several tens or hundreds, or more. The number and type of the terminal 110 are not limited in the embodiments of the present application.

[0048] In the related art, a sample key point prediction model is trained by a method based on a heat map. The sample key point prediction model can generate a high-resolution heat map corresponding to a sample image. In the heat map, each pixel point corresponds to a value, which is the probability that the key point belongs to the pixel point. Then, the prediction coordinates of the key point are determined based on the heat map. Finally, the model is trained based on the difference between the prediction coordinates of the key point and the reference coordinates. However, in the training scheme of the sample key point prediction model based on the heat map, each key point corresponds to a high-resolution heat map, and the amount of data that the model needs to predict is large, resulting in low training efficiency.

[0049] Therefore, the embodiments of the present application provide a training method of a key point prediction model. The probability distribution of the position of the key point of a sample object in an input image is predicted by a sample prediction model to obtain a plurality of probability distribution parameters for representing the mixed probability distribution of the key point. The sample prediction model is trained based on the reference coordinates of the key point and the plurality of probability distribution parameters. The amount of data of the probability distribution parameters predicted by the sample prediction model is much smaller than the amount of data of the heat map predicted in the related art, and the calculation amount of the sample prediction model is small, thereby improving the training efficiency of the sample prediction model.

[0050] FIG. 2 is a schematic diagram of the training method of the key point prediction model provided in an example embodiment of the present application. The method is executed by a computer device, which can be the terminal 110 and / or the server 120 shown in FIG. 1. In combination with FIG. 2, the steps executed by the computer device are briefly described as follows.

[0051] Step 1: The computer device acquires a sample image 200 containing an object 201.

[0052] The sample image 200 is labeled with reference coordinates corresponding to a plurality of key points of the object 201, including the reference coordinates of the key point 202.

[0053] Step 2: The computer device inputs the sample image 200 into the sample prediction model 210, and the sample prediction model 210 predicts the probability distribution of the position of the key point of the object 201 in the sample image 200, and obtains a plurality of probability distribution parameters corresponding to a plurality of key points.

[0054] The plurality of probability distribution parameters include the probability distribution parameters corresponding to the key point 202. Illustratively, the sample prediction model can predict the probability distribution of the position of the key point 202 from different image analysis angles, thereby obtaining different probability distribution parameters of the key point 202 under different image analysis angles, and the different probability distribution parameters correspond to different probability distributions. The probability distribution of the position of the key point 202 is used to represent the probability that the key point 202 belongs to each position in the sample image.

[0055] Suppose the probability distribution predicted by the sample prediction model 210 is a Gaussian distribution, then each probability distribution parameter can include a mean (in coordinate form), a standard deviation, and a weight, that is, each probability distribution parameter corresponding to the key point 202 represents a Gaussian distribution of the key point 202 predicted by the model, and the weight represents the confidence of each Gaussian distribution. Therefore, the plurality of probability distribution parameters can represent a mixed Gaussian distribution of the key point 202 (such as the mixed probability distribution shown in FIG. 2).

[0056] Step 3: The computer device trains the sample prediction model based on the reference coordinates of the key point 202 and the plurality of probability distribution parameters corresponding to the key point 202, and obtains a key point prediction model.

[0057] Illustratively, taking the probability distribution parameters including the mean (in coordinate form), the standard deviation, and the weight as an example for description. For the key point 202, the coordinate difference between the mean in each probability distribution parameter and the reference coordinates is calculated, and then the coordinate difference values corresponding to the plurality of probability distribution parameters are weighted and summed by the weights included in the plurality of probability distribution parameters, to obtain a total coordinate difference. The standard deviations included in the plurality of probability distribution parameters are weighted and summed by the weights included in the plurality of probability distribution parameters, to obtain a total standard deviation. Finally, based on the total standard deviation and the total coordinate difference, a target loss is calculated, and the key point prediction model is optimized by minimizing the target loss as a training target.

[0058] Optionally, the key point prediction model is optimized based on the target losses corresponding to the n key points, and the key point prediction model is obtained. The computer device inputs the target image into the key point prediction model, and the key point coordinates of the object to be recognized in the target image can be predicted.

[0059] FIG. 3 is a flowchart of a method for training a key point prediction model according to an example embodiment of the present application. The method is performed by a computer device, which can be the terminal 110 and / or the server 120 shown in FIG. 1. The method includes the following steps 310-330.

[0060] At step 310, a sample image containing a sample object is obtained. The sample image is labeled with reference coordinates corresponding to n key points of the sample object, where n is a positive integer.

[0061] The sample object refers to various entity objects contained in the sample image, including but not limited to a face, a human body, a vehicle, a building, etc. Optionally, the sample object can be a complete entity, such as a complete building, etc. The sample object can also be a part of an entity, such as a palm of a human body, etc.

[0062] The key points of the sample object refer to points on the target object that can represent the shape, structure, function, etc. of the target object, and are conducive to describing and identifying the target object. Different types of sample objects correspond to different key points. For example, the key points of a face include eyebrows, eyes, a nose, a mouth, and a chin, etc. The key points of a human body include a head, a neck, shoulders, elbows, wrists, hips, knees, and ankles, etc. The key points of a vehicle include a license plate position, a light position, and a wheel position, etc., which can be set according to task requirements or application scenarios.

[0063] It should be noted that the key points described above can be set according to task requirements or application scenarios, and the type and number of key points are not limited in the embodiments of the present application.

[0064] Optionally, the sample image can be at least one of an RGB (Red Green Blue) color image, a grayscale image, a depth image, etc., which is not limited in the embodiments of the present application.

[0065] Optionally, the number of sample objects in the sample image described above can be one or multiple.

[0066] In some embodiments, the reference coordinates corresponding to the n key points are represented by pixel coordinates of the key points. Optionally, the reference coordinates corresponding to the n key points of the sample object in the sample image are manually labeled. Alternatively, a target key point detection model is used to detect the key points of the sample object in the sample image to obtain candidate coordinates corresponding to the key points, and then the candidate coordinates corresponding to the key points are manually corrected to obtain the reference coordinates corresponding to the n key points. During the correction, the key points can be reduced, and the candidate coordinates of the key points can also be adjusted.

[0067] At step 320, the sample prediction model is used to predict the probability distribution of the key points of the sample object in the sample image, and a plurality of probability distribution parameters corresponding to the n key points are obtained. The plurality of probability distribution parameters corresponding to the jth key point are parameters predicted based on different image analysis angles for the jth key point, and the plurality of probability distribution parameters corresponding to the jth key point are used to represent the mixed probability distribution of the position of the jth key point, j≤n and j is a positive integer.

[0068] The sample prediction model is used to predict the probability distribution of the key points of the sample object in the sample image, and a plurality of probability distribution parameters are obtained. The model is trained based on these parameters and the reference coordinates of the key points, and finally the model of the key point prediction model is obtained. The model can be implemented as a regression model.

[0069] Illustratively, the probability distribution of the position of the key point refers to the probability of the key point belonging to each position in the sample image. The sample prediction model outputs a plurality of probability distribution parameters. The probability distribution parameter is a parameter used to model the probability distribution. Each probability distribution parameter represents a probability distribution. The plurality of probability distribution parameters can represent a combination of multiple probability distributions, i.e., a mixed probability distribution, which includes component parameters and component weights, etc. The plurality of probability distribution parameters can include at least one of a Gaussian distribution probability distribution parameter and a Laplace distribution probability distribution parameter. The component parameter is used to represent a sub-probability distribution of the mixed probability distribution of the key point, including a coordinate mean and a coordinate standard deviation, etc. The coordinate mean is used to represent the mean parameter of the sub-probability distribution corresponding to the component parameter. The coordinate standard deviation is used to represent the standard deviation parameter corresponding to the sub-probability distribution corresponding to the component parameter. The component weight is used to represent the confidence of the sub-probability distribution in the mixed probability distribution, and can be used to perform weighted fusion operations on different sub-probability distributions, etc.

[0070] The mixed probability distribution is used to represent the mixed probability of the key point belonging to each position in the sample image. For example, a mixed probability distribution is composed of probability distribution 1 and probability distribution 2. The probability of key point 1 belonging to position a in probability distribution 1 is 0.3, and the probability of key point 1 belonging to position a in probability distribution 2 is 0.4. Therefore, the mixed probability of key point 1 belonging to position a in the mixed probability distribution can be the average of 0.3 and 0.4, i.e., 0.35; or the mixed probability of key point 1 belonging to position a in the mixed probability distribution can be the weighted value of 0.3 and 0.4.

[0071] The number of probability distribution parameters corresponding to each key point output by the sample prediction model can be user-defined. For example, the sample prediction model outputs 5 sets of probability distribution parameters corresponding to key point 1, which indicates that the position of key point 1 is represented by 5 probability distributions.

[0072] Alternatively, the sample prediction model described above can be implemented as a regression model.

[0073] Illustratively, the image analysis angle is used to indicate the method of analyzing the sample image by using the sample prediction model. Different image analysis angles will be described below.

[0074] Optionally, the features of the sample image are extracted by the sample prediction model to obtain an image feature representation of the sample image; and the image feature representation is analyzed by the sample prediction model to predict a probability distribution of the positions of the key points of the sample object in the sample image, to obtain a plurality of probability distribution parameters corresponding to the n key points respectively.

[0075] 1. Different image analysis angles represent different image feature representations extracted by the sample prediction model.

[0076] Optionally, the plurality of probability distribution parameters corresponding to the jth key point are parameters predicted by the different image feature representations for the jth key point.

[0077] The image feature representation is an abstract representation of image information. Different feature representations focus on different aspects of the image, for example: color features mainly focus on the global or local color distribution of the image, shape features mainly focus on the outline or shape of the object in the image.

[0078] Selecting different image feature representations reflects the selection of different image information in the sample image by the model, that is, different image analysis angles are selected. In the sample prediction model, a plurality of (or understood as a plurality of groups) image feature representations corresponding to the sample image are extracted and analyzed respectively, and a plurality of (or a plurality of groups) probability distribution parameters corresponding to the plurality of (or the plurality of groups) image feature representations can be obtained.

[0079] Optionally, the image feature representation includes at least one of color features, texture features, shape features, spatial relationship features, edge features, etc.

[0080] Illustratively, the sample image is feature-extracted by the sample prediction model to obtain a plurality of image feature representations, wherein the probability distribution parameter 1 is predicted by analyzing the image feature representation 1; and the probability distribution parameter 2 is predicted by analyzing the image feature representation 2. The image feature representation 1 and the image feature representation 2 are different, for example: the image feature representation 1 is a color feature, and the image feature representation 2 is a texture feature; or the image feature representation 1 is a color feature and an edge feature, and the image feature representation 2 is a texture feature and an edge feature, etc.

[0081] Optionally, the image feature representation comprises at least one of a global image feature representation and a local image feature representation. The global image feature representation is used to represent the features of the entire sample image, and the local image feature representation is used to represent the features of the region related to the target key point, for example, the image feature representation corresponding to the jth key point is used to represent the features of the image region related to the jth key point.

[0082] Illustratively, the image feature representation 1 can be the features of the complete sample image, and the image feature representation 2 can be the features of part of the sample image; or, the image feature representation 1 is the features of the image region 1 in the sample image, and the image feature representation 2 is the features of the image region 2 in the sample image, and the image region 1 and the image region 2 are different.

[0083] 2. Different image analysis angles represent different mapping relationships of the image feature representation in the sample prediction model.

[0084] In some embodiments, the prediction of the jth key point based on different image analysis angles comprises the prediction of the jth key point based on different image feature representations, and the different image feature representations comprise at least two of color features, texture features, shape features, spatial relationship features or edge features. For example, through feature extraction of the sample image by the sample prediction model, different image feature representation combinations such as the combination of color features and texture features, or the combination of shape features and edge features can be obtained, and the multiple probability distribution parameters corresponding to the jth key point are predicted based on these different image feature representations.

[0085] Optionally, the multiple probability distribution parameters corresponding to the jth key point are parameters obtained by predicting the jth key point based on different mapping relationships.

[0086] The mapping relationship is used to represent the relationship between the image feature representation and the output probability distribution parameter, and for one (or a group) of image feature representations, there can be multiple mapping relationships, each mapping relationship has a different combination of weight bias, reflecting the different attention degrees of the model to one (or a group) of image feature representations, for example: for color features, the weight is 0.8 in mapping relationship A and the weight is 0.2 in mapping relationship B, which means that the mapping relationship A corresponds to a higher attention degree to the image color features in the output probability distribution parameter, and the mapping relationship B corresponds to a lower attention degree to the image color features in the output probability distribution parameter.

[0087] The selection of different mapping relationships reflects the different attention degrees of the model to the image feature representation, and can also be understood as the different image analysis angles of the model to the sample image. In the sample prediction model, after the image feature representation is extracted, the image feature representation can be output through multiple mapping relationships, thereby obtaining multiple probability distribution parameters.

[0088] Illustratively, the sample prediction model includes an output layer, and the output layer includes two (or two groups) of output nodes, each (or each group) of the output nodes represents a mapping relationship, and the image feature representation extracted by the sample prediction model is output to the output layer to obtain two (or two groups) of probability distribution parameters; wherein, the probability distribution parameter 1 refers to a parameter obtained by inputting the image feature representation corresponding to the jth key point into the output node 1, and the probability distribution parameter 2 refers to a parameter obtained by inputting the image feature representation corresponding to the jth key point into the output node 2.

[0089] In some embodiments, the prediction of the jth key point based on different image analysis angles includes the prediction of the jth key point based on different mapping relationships. The mapping relationship is used to represent the relationship between the image feature representation and the output probability distribution parameter. For example, the sample prediction model includes an output layer, and different output nodes of the output layer represent different mapping relationships. The extracted image feature representation is input into different output nodes to obtain different probability distribution parameters, and these different probability distribution parameters are obtained by predicting the jth key point based on different mapping relationships.

[0090] It should be noted that the above example of image analysis angle is only illustrative, and the embodiments of the present application are not limited thereto. For example, different mapping relationships can be used to map and output multiple image feature representations to obtain multiple probability distribution parameters, that is, image feature representation 1 is input into the output layer to obtain probability distribution parameter 1 through output node 1 and probability distribution parameter 2 through output node 2; image feature representation 2 is input into the output layer to obtain probability distribution parameter 3 through output node 3 and probability distribution parameter 4 through output node 4, and finally, probability distribution parameters 1 to 4 are obtained as parameters constituting a mixed probability distribution.

[0091] Optionally, the multiple probability distribution parameters include at least one of a Gaussian distribution probability distribution parameter and a Laplace distribution probability distribution parameter. Illustratively, the Gaussian distribution is taken as an example for description. The above probability distribution parameter includes a coordinate mean and a coordinate standard deviation, wherein the coordinate mean is used to represent the mean (or expectation) of the Gaussian distribution, and the coordinate standard deviation is used to represent the standard deviation of the Gaussian distribution, and the coordinate standard deviation is used to represent the dispersion degree between each predicted coordinate and the coordinate mean in the probability distribution.

[0092] In the embodiments of the present application, a plurality of Gaussian distribution (also referred to as normal distribution) or Laplace distribution probability distribution parameters are used to fit a mixed probability distribution, so as to represent the key point position. By combining a plurality of single distributions, the complexity of the key point position distribution can be accurately described, thereby improving the fitting accuracy of the model for the key point position. In addition, the Gaussian distribution and the Laplace distribution both have a clear mathematical expression, and the model can directly predict the expression parameters (i.e., the probability distribution parameters) to represent the distribution, thereby reducing the complexity of the model and improving the calculation efficiency of the model.

[0093] In step 330, the sample prediction model is trained based on the reference coordinates corresponding to the jth key point and the plurality of probability distribution parameters corresponding to the jth key point, to obtain a key point prediction model. The key point prediction model is used to predict the key point coordinates of the object to be recognized in the image.

[0094] In some embodiments, the target loss is determined based on the reference coordinates corresponding to the jth key point and the plurality of probability distribution parameters corresponding to the jth key point. The sample prediction model is trained based on the target loss, to obtain the key point prediction model.

[0095] The key point prediction model is a model trained by the sample prediction model based on the reference coordinates of the key point and the plurality of probability distribution parameters, and is used to predict the key point coordinates of the object to be recognized in the image.

[0096] The target loss is calculated by the target loss function, which can be implemented as a negative log-likelihood function. The negative log-likelihood function is used to evaluate the difference between the mixed probability distribution of the key point generated by the model and the reference coordinates. By minimizing the loss value calculated by the negative log-likelihood function, the optimal model parameters can be found, so that the mixed probability distribution generated by the model can better fit the position of the key point.

[0097] Alternatively, the probability distribution parameters include a coordinate mean and a coordinate standard deviation. The coordinate mean is used to represent the mean of the probability distribution corresponding to the key point, and the coordinate standard deviation is used to represent the standard deviation of the probability distribution corresponding to the key point. The method for determining the target loss further includes determining the target loss based on the difference between the coordinate mean included in the plurality of probability distribution parameters corresponding to the jth key point and the reference coordinates corresponding to the jth key point, and the coordinate standard deviation included in the plurality of probability distribution parameters corresponding to the jth key point.

[0098] The target loss is determined based on the average of the difference between the coordinate mean included in the plurality of probability distribution parameters corresponding to the jth key point and the reference coordinates corresponding to the jth key point, and the average of the coordinate standard deviation included in the plurality of probability distribution parameters corresponding to the jth key point.

[0099] Alternatively, the target loss is determined based on a weighted sum of differences between coordinate mean values included in the plurality of probability distribution parameters corresponding to the jth key point and reference coordinates corresponding to the jth key point, and a weighted sum of coordinate standard deviations included in the plurality of probability distribution parameters corresponding to the jth key point. It should be noted that only the method of training the model through the jth key point is described here. For a single sample image, if the number of key points is more than one, the target loss corresponding to each key point needs to be obtained, and then the model is trained through the target loss corresponding to each key point.

[0100] Illustratively, the parameters of the sample prediction model are updated with the training target of minimizing the target loss. Alternatively, in the case that the target loss is less than or equal to a preset loss, the training of the sample prediction model is stopped to obtain the key point prediction model; or in the case that the number of training times is greater than or equal to a preset number, the training of the sample prediction model is stopped to obtain the key point prediction model. The preset loss and the preset number can be set according to actual experience, and the embodiments of the present application are not limited thereto.

[0101] The key point prediction model is used to predict the key point coordinates of the object to be recognized in the image.

[0102] In some embodiments, after the key point prediction model is trained, the coordinates of the target key points of the target object in the target image can be predicted through the key point prediction model. Alternatively, the process of predicting the coordinates of the target key points further includes the following steps 1 to 4.

[0103] Step 1: detecting the target object in the target image to obtain a detection box corresponding to the target object.

[0104] The detection box is used to frame the target object in the target image.

[0105] Alternatively, the target image stored in the computer device is obtained; or the image acquired by the computer device is obtained as the target image; or the video frame in the video acquired by the computer device is obtained as the target image, etc.

[0106] Illustratively, the target object (such as a person, a vehicle, an animal, etc.) in the target image is recognized through a target detection algorithm such as SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), etc., and the corresponding detection box is output. The detection box is usually a rectangle, which is used to frame the position of the target object in the target image.

[0107] Step 2: determining the target region in the target image based on the detection box corresponding to the target object.

[0108] The target region includes the target object.

[0109] Optionally, the region framed by the detection frame in the target image is taken as the target region. Illustratively, the target region is the region inside the detection frame, which includes the entire target object, such as the entire human body, the entire building, etc.

[0110] Alternatively, further detection is performed on the region framed by the detection frame in the target image to obtain the target region, which includes the target part of the target object. Illustratively, the target part can be the palm region, the head region, etc. of the human body; or the target part is the window, the door, etc. of the building.

[0111] Step 3: The key point prediction model is used to predict the probability distribution of the position of the key point of the target object in the target region, to obtain a plurality of probability distribution parameters corresponding to the key point.

[0112] Optionally, the target region is input into the key point prediction model, and the key point prediction model is used to extract the features of the target region to obtain a target feature representation; the key point prediction model is used to analyze the target feature representation to predict the probability distribution of the position of the key point of the target object in the target region, to obtain a plurality of probability distribution parameters corresponding to the key point.

[0113] Illustratively, taking the human head key point as an example, the key point prediction model can predict a plurality of probability distribution parameters corresponding to the head key point, each probability distribution parameter is used to represent a probability distribution, and the plurality of probability distribution parameters represent a plurality of probability distributions corresponding to the head key point. For example, probability distribution parameter 1 represents probability distribution 1, and probability distribution parameter 2 represents probability distribution 2.

[0114] Step 4: The coordinates of the key point are determined based on the plurality of probability distribution parameters corresponding to the key point.

[0115] Optionally, the plurality of probability distribution parameters can be used to construct a mixed probability distribution of the head key point. For example, the mixed probability distribution refers to the combination of probability distribution 1 and probability distribution 2. The probability that the head key point belongs to position a in probability distribution 1 is 0.3, and the probability that the head key point belongs to position a in probability distribution 2 is 0.4. Therefore, the probability that the head key point belongs to position a in the mixed probability distribution can be the average of 0.3 and 0.4, i.e. 0.35; or the probability that the head key point belongs to position a in the mixed probability distribution can be the weighted sum of 0.3 and 0.4.

[0116] After obtaining the mixed probability distribution, a position with the highest probability value in the mixed probability distribution is taken as the coordinate of the key point, or an expectation value (i.e., a weighted average position) of the mixed probability distribution is calculated and taken as the coordinate of the key point, or a median position of the mixed probability distribution is taken as the coordinate of the key point, and the like, which are not limited by the embodiments of the present application.

[0117] In the above embodiments, after the key point prediction model is trained, the key point prediction model can be used to predict the key point positions of the object in the image. Before predicting the key point positions, the target object in the image can be detected to determine the display region of the target object in the image, so as to determine the target region that needs to be analyzed by the key point prediction model, avoid the waste of computing power caused by analyzing the entire target image by the key point prediction model, improve the prediction efficiency of the key point prediction model, and at the same time, reduce the influence of the background region (which can be understood as a region other than the target region) in the target image on the key point prediction, and improve the prediction accuracy of the model.

[0118] In summary, the training method of the key point prediction model provided in the embodiments of the present application predicts the probability distribution of the key point positions of the sample object in the input image by using the sample prediction model, obtains a plurality of probability distribution parameters of the mixed probability distribution for representing the key point, and trains the sample prediction model based on the reference coordinates of the key point and the plurality of probability distribution parameters. The data amount of the probability distribution parameters predicted by the sample prediction model is much smaller than that of the heat map predicted in the related art, and the calculation amount of the sample prediction model is smaller, thereby improving the training efficiency of the sample prediction model. In addition, the plurality of probability distribution parameters predicted by the sample prediction model are obtained based on different image analysis angles, different image analysis angles can capture various features of the sample object, and errors caused by a single probability distribution parameter are reduced, thereby improving the prediction accuracy of the key point prediction model trained.

[0119] In some embodiments, after training the sample prediction model based on the reference coordinates corresponding to the jth key point and the plurality of probability distribution parameters corresponding to the jth key point, the following operations can be performed: detecting a target object in a target image, for example, identifying the target object in the target image by a target detection algorithm such as SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), etc., and obtaining a detection box corresponding to the target object, the detection box being used to frame the target object in the target image; determining a target region in the target image based on the detection box corresponding to the target object, for example, taking the region framed by the detection box in the target image as the target region, the target region containing the target object; inputting the target region into the key point prediction model to predict the probability distribution of the position of the key point of the target object in the target region, and obtaining the plurality of probability distribution parameters corresponding to the key point; and determining the coordinates of the key point based on the plurality of probability distribution parameters corresponding to the key point, for example, by constructing a mixed probability distribution of the key point, taking the position with the highest probability value in the mixed probability distribution as the coordinates of the key point, or calculating the expected value (i.e., the weighted average position) of the mixed probability distribution as the coordinates of the key point, etc.

[0120] In some embodiments, the probability distribution parameters include component parameters and component weights, the component parameters being used to represent the sub-probability distributions of the mixed probability distribution of the key point, and the component weights being used to represent the confidence of the sub-probability distributions in the mixed probability distribution. For illustration, please refer to FIG. 4, the embodiment shown in FIG. 3 can also be implemented as steps 410 to 432 in FIG. 4.

[0121] Step 410: obtaining a sample image containing a sample object, the sample image being labeled with reference coordinates corresponding to n key points of the sample object, n being a positive integer.

[0122] In some embodiments, the sample image refers to a sample image obtained after preprocessing. Optionally, the preprocessing method includes at least one of the following methods.

[0123] (1) Image size adjustment: adjusting the sample image to a fixed size, which helps to ensure that the model has consistent input size when processing all input images, thereby simplifying model design and speeding up processing.

[0124] (2) Image graying: converting a color image into a grayscale image, which helps to reduce the amount of calculation and improve the robustness of the model.

[0125] (3) Image enhancement: such as contrast enhancement, histogram equalization, etc. Image enhancement can enhance the features of the image and improve the accuracy of key point prediction.

[0126] (4) Noise removal: Various noises may exist in the sample image, such as Gaussian noise, etc. A filter (such as a Gaussian filter) can be used to smooth the image and remove noise to improve the effect of key point prediction.

[0127] (5) Edge detection: In some embodiments, edge points located at the edges of the image are included in the key points. Therefore, an edge detection algorithm (such as a Sobel operator, a Canny edge detector, etc.) can be used to detect the edges in the image, thereby helping the model to more accurately predict the key points.

[0128] (6) Object detection: In some embodiments, for the obtained original image, object detection can be performed on the original image to determine the presence area of the sample object, and then the original image is cropped according to the detection result to obtain a sample image containing the sample object, thereby reducing the amount of data that the model needs to process.

[0129] It should be noted that the above examples of preprocessing methods are only illustrative, and the embodiments of the present application are not limited thereto.

[0130] At step 420, the sample prediction model is used to predict the probability distribution of the positions of the key points of the sample object in the sample image, to obtain a plurality of component parameters corresponding to each of the n key points, and a plurality of component weights corresponding to the plurality of component parameters. The plurality of component parameters corresponding to the jth key point and the plurality of component weights corresponding to the plurality of component weights are used to represent the mixed probability distribution of the position of the jth key point, j≤n and j is a positive integer.

[0131] Illustratively, each component parameter is used to model a sub-probability distribution of the mixed probability distribution, and the component weight corresponding to the component parameter is used to represent the confidence of the sub-probability distribution corresponding to the component parameter in the mixed probability distribution.

[0132] For example: the mixed probability distribution is composed of probability distribution 1 and probability distribution 2, the weight of probability distribution 1 is 0.6, and the weight of probability distribution 2 is 0.4; the probability of key point 1 belonging to position a in probability distribution 1 is 0.3, and the probability of key point 1 belonging to position a in probability distribution 2 is 0.4, then the mixed probability of key point 1 belonging to position a in the mixed probability distribution can be the weighted fusion value of 0.3 and 0.4, i.e. 0.34.

[0133] In some embodiments, the component parameters include a coordinate mean and a coordinate standard deviation, the coordinate mean is used to represent the mean parameter of the sub-probability distribution corresponding to the component parameter, and the coordinate standard deviation is used to represent the standard deviation parameter of the sub-probability distribution corresponding to the component parameter.

[0134] Optionally, the key points include at least one of one-dimensional key points, two-dimensional key points, three-dimensional key points, and the like, which are not limited in the embodiments of the present application.

[0135] Taking the two-dimensional key points as an example, the coordinate standard deviation includes a first standard deviation and a second standard deviation. The first standard deviation is used to represent the dispersion degree between each predicted position and the horizontal coordinate of the coordinate mean in the probability distribution corresponding to the component parameter. The second standard deviation is used to represent the dispersion degree between each predicted position and the vertical coordinate of the coordinate mean in the probability distribution corresponding to the component parameter.

[0136] In the formula, each predicted position in the probability distribution corresponding to the component parameter can be each position in the sample image. Illustratively, assuming that the predicted positions include a (2, 3) and b (3, 5), the probability of a is 0.8, the probability of b is 0.2, and the coordinate mean is (1, 1), the first standard deviation can be The second standard deviation can be It should be noted that the calculation process here is only to illustrate the meaning of the first standard deviation and the second standard deviation. In the embodiments of the present application, the sample prediction model directly outputs the first standard deviation and the second standard deviation.

[0137] In the above embodiments, in the case of two-dimensional key points, the coordinate standard deviation in the probability distribution parameters output by the sample prediction model is subdivided into the first standard deviation and the second standard deviation, which respectively describe the dispersion degrees of the horizontal coordinate and the vertical coordinate. The model can more accurately describe the distribution of the key points in the two-dimensional space, which helps the model to more accurately predict the positions of the key points.

[0138] Optionally, the feature extraction network in the key point prediction model is used to extract the features of the sample image to obtain an image feature representation corresponding to the sample image. The mixed density network in the key point prediction model is used to analyze the image feature representation to predict the probability distribution of the positions of the key points of the sample object in the sample image, and obtain a plurality of component parameters corresponding to the n key points, and a plurality of component weights corresponding to the plurality of component parameters.

[0139] Illustratively, the feature extraction network can be implemented as a VGG (Visual Geometry Group), a ResNet (Residual Network), or the like. The mixed density network is used to output the parameters of the probability distribution. In the key point prediction task, due to factors such as noise, occlusion, and perspective change in the image, the positions of the key points have certain uncertainty. Therefore, the mixed density network can be used to predict a plurality of probability distributions of the key points, so as to more accurately describe the uncertainty.

[0140] In the above embodiments, the feature extraction network can efficiently extract image features related to key point prediction from sample images, which can accurately represent the shape, texture and other key information of the sample object, and is beneficial to subsequent key point prediction. The mixed density network can predict the key point probability distribution of the sample object in the sample image by analyzing the image feature representation. By introducing multiple component parameters and corresponding component weights, the distribution of the key points can be more accurately described, and the prediction error caused by a single distribution model can be avoided.

[0141] For illustration, please refer to FIG. 5, which shows a schematic diagram of a training process of a key point prediction model. As shown in FIG. 5, a sample image 500 containing a sample object 501 is input into a sample detection model 510, and the sample detection model 510 predicts the probability distribution of the position of the key points of the sample object 501 through a mixed density network 511 to obtain multiple component parameters corresponding to multiple key points respectively.

[0142] Suppose the number of components of the mixed probability distribution is L (L is an integer greater than 1), and the construction of the i-th sub-probability distribution in the mixed probability distribution requires: coordinate mean x i , covariance matrix Σ i , component weight w i . Wherein:

[0143] The covariance matrix Σ i can be decomposed into standard deviation , that is:

[0144] and are the standard deviations on the horizontal and vertical axes respectively, is the parameter used to construct the covariance matrix predicted by the model, that is, the i-th component parameter predicted by the sample prediction model includes: coordinate mean x i , standard deviation and component weight w i .

[0145] Step 431, determining the target loss based on the reference coordinates corresponding to the j-th key point, the multiple component parameters corresponding to the j-th key point, and the component weights corresponding to the multiple component parameters respectively.

[0146] For illustration, the multiple component parameters corresponding to the j-th key point and the component weights corresponding to the multiple component parameters form a set of distribution components, that is: {(x i ,Σ i ,w i )}, 0 < i ≤ L

[0147] This set of distribution components is used to characterize the mixed probability distribution corresponding to the j-th keypoint. For illustration, please refer to Figure 5. Taking keypoint 502 as an example, the distribution component set of keypoint 502 includes L component parameters output by the mixed density network 511: (x1, σ1), (x2, σ2), ..., (x... i , σ i ), ..., (x L , σ L ), where in (x i , σ i ) in x i This refers to the coordinate mean, σ i This refers to the coordinate standard deviation. Each component parameter represents a sub-probability distribution, as shown in Figure 5. Probability distribution component 502 refers to the sub-probability distribution with x1 as the coordinate mean and σ1 as the coordinate standard deviation. The distribution component set of keypoint 502 also includes the component weights corresponding to the L component parameters output by the hybrid density network 511: w1, w2, ..., w i ... w L By mixing the sub-probability distributions represented by the L component parameters based on the component weights, a mixed probability distribution 503 can be obtained.

[0148] The target loss can be calculated based on the distribution component set of the j-th key point mentioned above. Optionally, the method for determining the target loss further includes steps 1 to 4 below.

[0149] Step 1: Calculate the coordinate difference between the mean of multiple coordinates corresponding to the j-th key point and the reference coordinates corresponding to the j-th key point.

[0150] Indicatively, maximum likelihood estimation (MLE) can make the mixture probability distribution of the model as close as possible to the probability distribution in the real data. Therefore, for the true coordinates of each keypoint... The mean x-coordinate of each predicted component parameter can be obtained. i and reference coordinates The coordinate difference is:

[0151] Step 2: Based on the component parameters corresponding to the multiple component parameters, the multiple coordinate differences corresponding to the j-th key point are weighted and fused to obtain the target coordinate difference.

[0152] To illustrate, the target coordinate difference ε is obtained by weighting and summing the coordinate differences corresponding to the L component parameters using the component weights corresponding to the L component parameters.

[0153] Step 3: Based on the weights corresponding to the plurality of component parameters respectively, the plurality of coordinate standard deviations corresponding to the jth key point are weighted and fused to obtain a target standard deviation.

[0154] Illustratively, the L component parameters respectively include coordinate standard deviations are weighted and summed by the component weights corresponding to the L component parameters respectively to obtain a target standard deviation σ.

[0155] Taking the first standard deviation and the second standard deviation as examples of the coordinate standard deviation, the first standard deviation included in the L component parameters respectively is weighted and summed by the component weights corresponding to the L component parameters respectively to obtain a total first standard deviation, the second standard deviation included in the L component parameters respectively is weighted and summed by the component weights corresponding to the L component parameters respectively to obtain a total second standard deviation, and the total first standard deviation and the total second standard deviation are taken as the target standard deviation σ.

[0156] Step 4: Determine a target loss according to the target coordinate difference value and the target standard deviation.

[0157] Illustratively, the target loss refers to an error of a likelihood estimation, that is, an error of a likelihood estimation is determined based on the target coordinate difference value and the target standard deviation. Alternatively, in order to facilitate calculation, the error is generally taken as a logarithm to reduce optimization complexity.

[0158] In some embodiments, if the plurality of probability distribution parameters are probability distribution parameters corresponding to a Gaussian distribution, the target loss L is calculated according to the following formula one:

[0159] Formula one:

[0160] Wherein, σ in formula one refers to the target standard deviation calculated above, and ε refers to the target coordinate difference value calculated above.

[0161] In some embodiments, if the plurality of probability distribution parameters are probability distribution parameters corresponding to a Laplace distribution, the target loss L is calculated according to the following formula two:

[0162] Formula two:

[0163] Wherein, σ in formula two refers to the target standard deviation calculated above, and ε refers to the target coordinate difference value calculated above.

[0164] ​​In some embodiments, the plurality of probability distribution parameters comprises a probability distribution parameter corresponding to a Gaussian distribution and a probability distribution parameter corresponding to a Laplace distribution. The target coordinate difference and the target standard deviation corresponding to the Gaussian distribution can be determined according to the probability distribution parameter corresponding to the Gaussian distribution, and the Gaussian distribution loss can be determined according to the above Formula I. The target coordinate difference and the target standard deviation corresponding to the Laplace distribution can be determined according to the probability distribution parameter corresponding to the Laplace distribution, and the Laplace distribution loss can be determined according to the above Formula II. The target loss can be obtained by adding the Gaussian distribution loss and the Laplace distribution loss.

[0165] At step 432, the sample prediction model is trained by the target loss to obtain the key point prediction model.

[0166] Illustratively, the sample prediction model is trained by minimizing the target loss to obtain the key point prediction model. As shown in FIG. 5, the loss value L is calculated, and the sample detection model 510 is trained with the training target of minimizing the loss value L to obtain the key point prediction model.

[0167] In summary, the embodiments of the present application provide a training method of a key point prediction model. The sample prediction model is used to predict the probability distribution of the position of the key point of the sample object in the input image to obtain a plurality of probability distribution parameters of a mixed probability distribution for representing the key point. The sample prediction model is trained based on the reference coordinates of the key point and the plurality of probability distribution parameters. The probability distribution parameters comprise component parameters and corresponding component weights. The component weights represent the confidence of the probability distribution, so that the model can distinguish the importance of different probability distributions and give higher weights to more reliable probability distributions. Therefore, the model can more accurately describe the uncertainty of the position of the key point and improve the accuracy of the predicted key point coordinates.

[0168] In some embodiments, when the sample prediction model is trained, the sample image is regionally divided, and the probability distribution of the position of the key point is predicted in each region, so that the model for predicting the position of the key point can be more targetedly optimized. Illustratively, please refer to FIG. 6. The embodiments shown in FIG. 3 or FIG. 4 can also be implemented as steps 610 to 634 in FIG. 6.

[0169] At step 610, a sample image containing a sample object is obtained. The sample image is labeled with reference coordinates corresponding to n key points of the sample object, and n is a positive integer.

[0170] At step 621, the sample image is divided into a plurality of image regions.

[0171] Optionally, the sample image is evenly divided into a plurality of image regions. Illustratively, taking a rectangular sample image as an example, the sample image can be divided into a plurality of sub-images of the same size, each of which contains a part of the image.

[0172] Alternatively, the sample image is cropped using a sliding window to obtain a plurality of image regions. Illustratively, the window is slid from left to right and from top to bottom on the sample image, and each time it is moved by a certain step size. For each window position, a corresponding image region is cropped.

[0173] At step 622, the sample prediction model is used to predict the probability distribution of the position of the key points of the sample object in each image region, obtaining a plurality of key point prediction results corresponding to the plurality of image regions respectively; the key point prediction result includes a plurality of probability distribution parameters corresponding to n key points respectively, wherein the plurality of probability distribution parameters corresponding to the jth key point are parameters predicted by the sample prediction model based on different image analysis angles for the jth key point, and the plurality of probability distribution parameters corresponding to the jth key point are used to represent the mixed probability distribution of the position of the jth key point, j≤n and j is a positive integer.

[0174] Taking a target image region in the plurality of image regions as an example, the feature of the target image region is extracted by the sample prediction model to obtain a sub-image feature representation; the probability distribution of the position of the key points of the sample object in the target image region is predicted by analyzing the sub-image feature representation by the sample prediction model, and the key point prediction result corresponding to the target image region is obtained.

[0175] The prediction method of the probability distribution of the position of the key points of the sample object in the target image region can refer to the description of the prediction method of the probability distribution of the position of the key points of the sample object in the sample image in steps 320 and 420, which will not be described here.

[0176] Illustratively, assuming that the current includes image region a and image region b, for key point 1, the sample prediction model outputs probability distribution parameter 1 and probability distribution parameter 2 of key point 1 in image region a, and probability distribution parameter 3 and probability distribution parameter 4 of key point 1 in image region b.

[0177] At step 631, based on the plurality of probability distribution parameters corresponding to the jth key point in the key point prediction result, a plurality of region probabilities corresponding to the plurality of image regions are determined, and the region probability is used to represent the probability that the jth key point belongs to the image region.

[0178] The region probability is used to represent the probability that the jth key point belongs to a certain image region, which can be determined based on the plurality of probability distribution parameters corresponding to the jth key point in the key point prediction result.

[0179] Taking a target image region in the plurality of image regions as an example, the mixed probability distribution of the target image region can be constructed according to the plurality of probability distribution parameters corresponding to the jth key point in the key point prediction result of the target image region.

[0180] Optionally, the probability distribution parameters include component parameters and component weights, the component parameters are used to represent sub-probability distributions of the mixed probability distribution of the key point, and the component weights are used to represent confidence degrees of the sub-probability distributions in the mixed probability distribution. The component parameters include a coordinate mean value and a coordinate standard deviation, the coordinate mean value is used to represent a mean parameter of the sub-probability distribution corresponding to the component parameter, and the coordinate standard deviation is used to represent a standard deviation parameter of the sub-probability distribution corresponding to the component parameter.

[0181] For example, the component parameter 1 of the target image region represents a Gaussian distribution a, and the weight of the component parameter 1 is 0.8; the component parameter 2 of the target image region represents a Gaussian distribution b, and the weight of the component parameter 2 is 0.2; the mixed probability distribution of the target image region is obtained by weighted summation of the Gaussian distribution a and the Gaussian distribution b according to the weights.

[0182] After obtaining the mixed probability distribution of the target image region, in the mixed probability distribution, an average value of probabilities corresponding to each predicted position is taken as a region probability, or a maximum probability in the probabilities of each predicted position is taken as the region probability, or a median in the probabilities of each predicted position is taken as the region probability, which is not limited in the embodiments of the present application.

[0183] In step 632, a region loss corresponding to each of the plurality of image regions is determined based on the reference image region to which the reference coordinate corresponding to the jth key point belongs and the region probabilities corresponding to the plurality of image regions.

[0184] The reference image region is one of the plurality of image regions.

[0185] The region loss is determined based on the reference image region to which the reference coordinate corresponding to the jth key point belongs and the region probabilities corresponding to the plurality of image regions, and can be calculated by using a cross-entropy loss function or the like, and is used to measure the rationality of the distribution of the key point in different image regions and guide the model to learn more effectively.

[0186] For example, the region loss can be calculated by using at least one of a cross-entropy loss function or the like. That is, if an image region is a reference image region, the training target is to make the region probability of this region output by the model as close to 1 as possible. If an image region is not a reference image region, the training target is to make the region probability output by the model as close to 0 as possible.

[0187] Optionally, for each image region, a loss is calculated using a cross-entropy loss function, obtaining a plurality of region losses respectively corresponding to the plurality of image regions.

[0188] At step 633, based on the reference coordinates corresponding to the jth key point and the plurality of probability distribution parameters corresponding to the jth key point in the key point prediction result, a plurality of target losses respectively corresponding to the plurality of image regions are determined.

[0189] The target loss calculation method for the image region can refer to the description of the target loss calculation method for the sample image in step 431, which will not be repeated here.

[0190] The target loss is determined based on the reference coordinates corresponding to the jth key point and the plurality of probability distribution parameters corresponding to the jth key point, and is used to evaluate the difference between the mixed probability distribution of the key point generated by the model and the reference coordinates, so as to optimize the model with the training target of minimizing the target loss.

[0191] At step 634, the sample prediction model is trained according to the region loss and the target loss, obtaining a key point prediction model.

[0192] Illustratively, the region losses respectively corresponding to the plurality of image regions are added to obtain a total region loss, and the target losses respectively corresponding to the plurality of image regions are added to obtain a total target loss; the sample detection model is trained with the loss sum of the total region loss and the total target loss as the training target, and the key point prediction model can be obtained.

[0193] In summary, the training method of the key point prediction model provided in the embodiments of the present application divides the sample image into a plurality of image regions, and trains the sample prediction model based on these regions. By dividing the image into a plurality of regions, the model can focus more on the local details in each region, thereby more accurately predicting the key points, reducing the complexity of full-image prediction, and improving the prediction accuracy. In addition, the embodiments not only consider the prediction accuracy of the key points in the region (target loss), but also consider the distribution probability of the key points relative to the reference region (region loss), thereby guiding the model to learn more effectively and improving the prediction accuracy of the key point prediction model obtained by training.

[0194] The key point prediction model trained by the training method provided in the embodiments of the present application can be applied to various scenes, such as palmprint recognition scenes, action, gesture, and gait recognition scenes. The palmprint recognition process and the action, gesture, and gait recognition process provided in the embodiments of the present application are introduced below.

[0195] 1. Palmprint recognition process.

[0196] Before each user starts palmprint identification, the palm detection model and the palm key point prediction model trained by the training method are used to constitute a palm following identification component. For illustration, please refer to FIG. 7, which shows a palmprint identification flowchart. As shown in FIG. 7, the palmprint identification flow includes the following steps.

[0197] S701: Taking a photo.

[0198] The computer device acquires multiple continuous image frames by taking a photo.

[0199] S702: Palm following identification.

[0200] The computer device first performs palm detection on an initial image frame in the multiple continuous images by using the palm detection model, and after the palm detection, the palm region is framed in the initial image frame by using the detection frame. Then, the palm key point prediction model is used to perform key point detection on the framed palm region in the initial image frame to determine multiple hand key points. Then, in the multiple image frames after the initial image frame, the palm key point prediction model is used to predict the positions of the multiple hand key points in the multiple image frames to determine the positions of the detection frames of the framed palm regions in the multiple image frames after the initial image frame, so as to realize the following identification of the palm.

[0201] S703: Extracting a region of interest (ROI) region.

[0202] For illustration, according to the detection frames identified in the multiple continuous image frames, the ROI region is determined, for example, the palm regions framed by the detection frames corresponding to the multiple continuous image frames are taken as the ROI region; or the palm region framed by the detection frame in the last image frame is taken as the ROI region.

[0203] S704: Transmission to the background and palmprint identification.

[0204] The palm photo of the user is registered in the background registration library, and the palm in the ROI region is compared with each palm photo in the background registration library for palmprint identification, so as to finally determine the identity of the user.

[0205] 2. Action, gesture, gait recognition flow.

[0206] After the human body or hand region is located by using the target detection model, the key point prediction model trained by the training method is used to locate the key points of the human body or hand to complete the subsequent action, gesture, and gait recognition. For illustration, please refer to FIG. 8, which shows an action, gesture, and gait recognition flowchart. As shown in FIG. 8, the action, gesture, and gait recognition flow includes the following steps.

[0207] S801: photographing a photo.

[0208] The computer device photographs a photo to be recognized as an image frame.

[0209] S802: target detection.

[0210] As shown in FIG. 9, the computer device performs target detection on the image frame 900 to be recognized by using a target detection model, and a target object 901 is framed in the image frame 900 by a detection frame 902.

[0211] S803: human pose estimation.

[0212] The computer device performs key point detection on the framed target object region in the image frame to be recognized by using a key point prediction model, and determines a plurality of object key points.

[0213] S804: extraction of a region of interest (ROI).

[0214] Based on the plurality of object key points, an ROI region is extracted from the framed target object region. As shown in FIG. 10, hand key points in the framed target object region are recognized, and a detection frame 1002 for framing a palm 1001 is obtained.

[0215] S805: action recognition / gesture recognition / gait recognition.

[0216] Action recognition / gesture recognition / gait recognition is performed based on the ROI region. For example, by gait recognition, a falling condition and a disease signal of a target object in an image can be determined, so as to detect a health condition of the target object. Action recognition or gesture recognition can be applied to automatic teaching in fitness, sports and dance, and the like, and by action recognition or gesture recognition, whether the action of the target object is correct or not can be detected, so as to assist the target object in learning or exercising.

[0217] As shown in FIG. 11, the device includes:

[0218] An acquisition module 1100 is configured to acquire a sample image containing a sample object, and the sample image is labeled with reference coordinates corresponding to n key points of the sample object, where n is a positive integer.

[0219] The prediction module 1120 is configured to predict, by using the sample prediction model, a probability distribution of a position of a key point of the sample object in the sample image, to obtain a plurality of probability distribution parameters corresponding to the n key points respectively, wherein a plurality of probability distribution parameters corresponding to a jth key point are parameters predicted by using different image analysis angles for the jth key point, and the plurality of probability distribution parameters corresponding to the jth key point are used to represent a mixed probability distribution of the position of the jth key point, j≤n and j is a positive integer.

[0220] The training module 1130 is configured to train the sample prediction model based on the reference coordinates corresponding to the jth key point and the plurality of probability distribution parameters corresponding to the jth key point, to obtain a key point prediction model, and the key point prediction model is used to predict key point coordinates of an object to be identified in an image.

[0221] Illustratively, referring to FIG. 12, in some embodiments, the probability distribution parameters include component parameters and component weights, the component parameters are used to represent a sub-probability distribution of the mixed probability distribution of the key point, and the component weights are used to represent a confidence of the sub-probability distribution in the mixed probability distribution; the prediction module 1120 is configured to extract a feature of the sample image by using a feature extraction network in the key point prediction model, to obtain an image feature representation corresponding to the sample image; and analyze the image feature representation by using a mixed density network in the key point prediction model, to predict a probability distribution of a position of a key point of the sample object in the sample image, to obtain a plurality of component parameters corresponding to the n key points respectively, and component weights corresponding to the plurality of component parameters respectively.

[0222] In some embodiments, the prediction module 1120 is configured to determine a target loss based on the reference coordinates corresponding to the jth key point, the plurality of component parameters corresponding to the jth key point, and the component weights corresponding to the plurality of component parameters respectively; and train the sample prediction model by using the target loss, to obtain the key point prediction model.

[0223] In some embodiments, the component parameter comprises a coordinate mean value and a coordinate standard deviation, the coordinate mean value being used to represent a mean parameter of a sub-probability distribution corresponding to the component parameter, and the coordinate standard deviation being used to represent a standard deviation parameter corresponding to the sub-probability distribution corresponding to the component parameter; the prediction module 1120 is configured to calculate coordinate difference values between a plurality of coordinate mean values corresponding to the jth key point and reference coordinates corresponding to the jth key point; based on component weights respectively corresponding to the plurality of component parameters, the plurality of coordinate difference values corresponding to the jth key point are weighted and fused to obtain a target coordinate difference value; based on the component weights respectively corresponding to the plurality of component parameters, the plurality of coordinate standard deviations corresponding to the jth key point are weighted and fused to obtain a target standard deviation; and the target loss is determined according to the target coordinate difference value and the target standard deviation.

[0224] In some embodiments, the key point comprises a two-dimensional key point, and the coordinate standard deviation comprises a first standard deviation and a second standard deviation; the first standard deviation is used to represent a dispersion degree between each predicted position in the probability distribution corresponding to the component parameter and a horizontal coordinate of the coordinate mean value, and the second standard deviation is used to represent a dispersion degree between each predicted position in the probability distribution corresponding to the component parameter and a vertical coordinate of the coordinate mean value.

[0225] In some embodiments, the plurality of probability distribution parameters comprises at least one of a Gaussian distribution probability distribution parameter and a Laplace distribution probability distribution parameter.

[0226] In some embodiments, the prediction module 1120 is configured to split the sample image into a plurality of image regions; predict, by using the sample prediction model, a probability distribution of a position of a key point of the sample object in each image region to obtain a key point prediction result corresponding to each image region, the key point prediction result comprising a plurality of probability distribution parameters respectively corresponding to the n key points.

[0227] In some embodiments, the training module 1130 is configured to determine, based on the plurality of probability distribution parameters corresponding to the j-th key point in the key point prediction result, a region probability corresponding to each of the plurality of image regions, the region probability representing a probability that the j-th key point belongs to the image region; determine, based on a reference image region to which a reference coordinate corresponding to the j-th key point belongs and the region probability corresponding to each of the plurality of image regions, a region loss corresponding to each of the plurality of image regions, the reference image region being one of the plurality of image regions; determine, based on the reference coordinate corresponding to the j-th key point and the plurality of probability distribution parameters corresponding to the j-th key point in the key point prediction result, a target loss corresponding to each of the plurality of image regions; and train the sample prediction model according to the region loss and the target loss to obtain the key point prediction model.

[0228] In some embodiments, the apparatus further includes a determination module 1140 configured to detect a target object in a target image to obtain a detection frame corresponding to the target object, the detection frame being used to frame the target object in the target image; determine, based on the detection frame corresponding to the target object, a target region in the target image, the target region containing the target object; predict, by the key point prediction model, a probability distribution of a position of a key point of the target object in the target region to obtain a plurality of probability distribution parameters corresponding to the key point; and determine, based on the plurality of probability distribution parameters corresponding to the key point, a coordinate of the key point.

[0229] In summary, the training apparatus of the key point prediction model provided in the embodiments of the present application predicts, by a sample prediction model, a probability distribution of a position of a key point of a sample object in an input image to obtain a plurality of probability distribution parameters representing a mixed probability distribution of the key point, thereby training the sample prediction model based on a reference coordinate of the key point and the plurality of probability distribution parameters. The data amount of the probability distribution parameters predicted by the sample prediction model is much smaller than that of a heat map predicted in the related art, and the calculation amount of the sample prediction model is smaller, thereby improving the training efficiency of the sample prediction model. In addition, the plurality of probability distribution parameters predicted by the sample prediction model are obtained based on different image analysis angles, which can capture various features of the sample object and reduce errors caused by a single probability distribution parameter, thereby improving the prediction accuracy of the key point prediction model obtained by training.

[0230] It should be noted that the specific limitations in the above embodiment of the training apparatus of one or more key point prediction models can refer to the limitations of the training method of the key point prediction model in the above, which will not be repeated here. The modules of the above apparatus can be realized by software, hardware and their combination, and each module can be embedded in the form of hardware or independent of the processor of the computer device, or can be stored in the form of software in the memory of the computer device, so as to call and execute the corresponding operations of each module by the processor.

[0231] The embodiment of the present application also provides a computer device, which comprises a processor and a memory, and the memory stores a computer program; the processor is used for executing the computer program in the memory to realize the training method of the key point prediction model provided by each method embodiment.

[0232] For example, FIG. 13 is a structural block diagram of a computer device 1300 provided by an example embodiment of the present application. Optionally, the computer device 1300 is a server 1300.

[0233] Generally, the server 1300 comprises a processor 1301 and a memory 1302.

[0234] The processor 1301 can comprise one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 can be realized in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 1301 can also comprise a main processor and a coprocessor, the main processor is a processor for processing data in the wake-up state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 can be integrated with a graphics processor (GPU), and the GPU is used to render and draw the content required to be displayed by the display screen. In some embodiments, the processor 1301 can also comprise an artificial intelligence (AI) processor, which is used to process machine learning related computing operations.

[0235] The memory 1302 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 1302 can also include high-speed random access memory and can include non-volatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 stores at least one instruction for execution by the processor 1301 to implement the training method of the key point prediction model provided by any of the method embodiments.

[0236] In some embodiments, the server 1300 can further include an input interface 1303 and an output interface 1304. The processor 1301, the memory 1302, and the input interface 1303 and the output interface 1304 can be connected through a bus or a signal line. Various peripheral devices can be connected to the input interface 1303 and the output interface 1304 through the bus, the signal line, or the circuit board. The input interface 1303 and the output interface 1304 can be used to connect at least one peripheral device related to input / output (I / O) to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302, and the input interface 1303 and the output interface 1304 are integrated on the same chip or circuit board; in some other embodiments, any one or both of the processor 1301, the memory 1302, and the input interface 1303 and the output interface 1304 can be implemented on a separate chip or circuit board, and the embodiments of the present application are not limited in this regard.

[0237] Those skilled in the art can understand that the structure shown in FIG. 13 does not constitute a limitation on the computer device 1300, and can include more or fewer components than those shown, or combine certain components, or use different component arrangements.

[0238] In an example embodiment, the present application provides a chip including programmable logic circuitry and / or program instructions, which, when the chip is running on a computer device, is used to implement the training method of the key point prediction model provided by any of the method embodiments.

[0239] In an example embodiment, the present application provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the training method of the key point prediction model provided by any of the method embodiments.

[0240] In an example embodiment, the present application provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the processor of the computer device to load and execute a training method of a key point prediction model provided by the above-mentioned method embodiment.

[0241] To sum up, the present application provides a training method and device of a key point prediction model, a computer device, a computer readable storage medium, and a computer program product. The computer device acquires a sample image containing a sample object and labeled with reference coordinates of n key points of the sample object respectively. The sample prediction model is used to predict the probability distribution of the positions of the key points of the sample object in the sample image, and a plurality of probability distribution parameters corresponding to the n key points are obtained. These parameters are obtained based on different image analysis angles and are used to represent the mixed probability distribution of the key points. Then, the sample prediction model is trained based on the reference coordinates of the key points and the plurality of probability distribution parameters, and a key point prediction model for predicting the coordinates of the key points of an object to be identified in an image is obtained. Since the data amount of the probability distribution parameters predicted by the sample prediction model is much smaller than that of the heat map predicted in the related art, the calculation amount of the sample prediction model is small, thereby improving the training efficiency of the sample prediction model. Different image analysis angles can capture various features of the sample object, and errors caused by a single probability distribution parameter are reduced, thereby improving the prediction accuracy of the key point prediction model obtained by training.

[0242] Further, when the probability distribution parameter includes component parameters used to represent sub-probability distributions of the mixed probability distribution of the key points and component weights used to represent the confidence of the sub-probability distributions in the mixed probability distribution, the computer device extracts features of the sample image through a feature extraction network in the key point prediction model to obtain an image feature representation corresponding to the sample image, and then analyzes the image feature representation through a mixed density network in the key point prediction model to predict the probability distribution of the positions of the key points of the sample object in the sample image, and obtains a plurality of component parameters corresponding to the n key points and a plurality of component weights corresponding to the plurality of component parameters. The feature extraction network can efficiently extract image features related to key point prediction from the sample image, and these features can accurately represent key information such as the shape and texture of the sample object, which is beneficial to subsequent key point prediction. The mixed density network can predict the probability distribution of the key points of the sample object in the sample image by analyzing the image feature representation. By introducing a plurality of component parameters and corresponding component weights, the distribution of the key points can be described more accurately, and prediction errors caused by a single distribution model can be avoided.

[0243] Further, when training the sample prediction model based on the key point-based reference coordinates and the plurality of probability distribution parameters, the computer device determines a target loss based on the reference coordinates corresponding to the jth key point, the plurality of component parameters corresponding to the jth key point, and the component weights corresponding to the plurality of component parameters, and then trains the sample prediction model through the target loss to obtain the key point prediction model. The component weights represent the confidence of the probability distribution, so that the model can distinguish the importance of different probability distributions and give higher weights to more reliable probability distributions, so that the model can more accurately describe the uncertainty of the key point position and improve the accuracy of the predicted key point coordinates.

[0244] Further, when the component parameters include a coordinate mean value representing a mean value parameter of the sub-probability distribution corresponding to the component parameter, and a coordinate standard deviation representing a standard deviation parameter of the sub-probability distribution corresponding to the component parameter, the computer device calculates a coordinate difference value between the plurality of coordinate mean values corresponding to the jth key point and the reference coordinates corresponding to the jth key point, weights and fuses the plurality of coordinate difference values corresponding to the jth key point based on the component weights corresponding to the plurality of component parameters to obtain a target coordinate difference value, weights and fuses the plurality of coordinate standard deviations corresponding to the jth key point based on the component weights corresponding to the plurality of component parameters to obtain a target standard deviation, and finally determines a target loss according to the target coordinate difference value and the target standard deviation. By comprehensively considering the influence of different sub-probability distributions through weighted fusion, the calculation of the target loss is more accurate, which can more effectively guide the model training and improve the prediction accuracy of the model for the key point position.

[0245] Further, when the key point includes a two-dimensional key point, the coordinate standard deviation includes a first standard deviation representing the dispersion degree between the horizontal coordinates of each predicted position in the probability distribution corresponding to the component parameter and the coordinate mean value, and a second standard deviation representing the dispersion degree between the vertical coordinates of each predicted position in the probability distribution corresponding to the component parameter and the coordinate mean value. In the case of a two-dimensional key point, the coordinate standard deviation in the probability distribution parameter output by the sample prediction model is subdivided into the first standard deviation and the second standard deviation, which respectively describe the dispersion degrees of the horizontal coordinates and the vertical coordinates, so that the model can more accurately describe the distribution of the key point in the two-dimensional space and help the model more accurately predict the position of the key point.

[0246] Further, the plurality of probability distribution parameters comprises at least one of a Gaussian distribution parameter and a Laplace distribution parameter. The mixed probability distribution is fitted by using a plurality of Gaussian distribution parameters or Laplace distribution parameters to represent the key point position, and the complexity of the key point position distribution can be accurately described by combining a plurality of single distributions, thereby improving the fitting accuracy of the model for the key point position. In addition, the Gaussian distribution and the Laplace distribution both have a clear mathematical expression, and the model can directly predict the expression parameters (i.e., the probability distribution parameters) to represent the distribution, thereby reducing the complexity of the model and improving the calculation efficiency of the model.

[0247] Further, the computer device divides the sample image into a plurality of image regions, predicts, by using the sample prediction model, a probability distribution of a position of a key point of the sample object in each image region, and obtains a plurality of key point prediction results corresponding to the plurality of image regions respectively, wherein the key point prediction result comprises a plurality of probability distribution parameters corresponding to n key points respectively. By dividing the image into a plurality of regions, the model can focus more on the local details in each region, thereby more accurately predicting the key point and reducing the complexity of the full-image prediction and improving the prediction accuracy.

[0248] Further, when training the sample prediction model based on the reference coordinates of the key points and the plurality of probability distribution parameters, the computer device determines, based on the plurality of probability distribution parameters corresponding to the jth key point in the key point prediction result, a plurality of region probabilities corresponding to the plurality of image regions respectively, wherein the region probability represents a probability that the jth key point belongs to an image region; determines, based on a reference image region to which the reference coordinates of the jth key point belong and the plurality of region probabilities corresponding to the plurality of image regions respectively, a plurality of region losses corresponding to the plurality of image regions respectively; determines, based on the reference coordinates of the jth key point and the plurality of probability distribution parameters corresponding to the jth key point in the key point prediction result, a plurality of target losses corresponding to the plurality of image regions respectively; and trains the sample prediction model based on the region losses and the target losses, to obtain the key point prediction model. This method not only considers the prediction accuracy of the key point in the region (the target loss), but also considers the distribution probability of the key point relative to the reference region (the region loss), thereby guiding the model to learn more effectively and improving the prediction accuracy of the key point prediction model obtained by training.

[0249] Further, after obtaining the key point prediction model, the computer device detects a target object in the target image to obtain a detection box corresponding to the target object, determines a target region in the target image based on the detection box corresponding to the target object, predicts a probability distribution of a position of a key point of the target object in the target region through the key point prediction model, obtains a plurality of probability distribution parameters corresponding to the key point, and determines the coordinates of the key point based on the plurality of probability distribution parameters corresponding to the key point. Before predicting the position of the key point, the target object in the image is detected to determine the display region of the target object in the image, so as to determine the target region that needs to be analyzed by the key point prediction model, avoid the waste of computing power caused by analyzing the entire target image by the key point prediction model, improve the prediction efficiency of the key point prediction model, and at the same time, reduce the influence of the background region in the target image on the key point prediction, and improve the prediction accuracy of the model.

[0250] Further, when predicting the jth key point based on different image analysis angles, the jth key point is predicted based on different image feature representations, and the different image feature representations include at least two of color features, texture features, shape features, spatial relationship features, or edge features. Different image feature representations focus on different aspects of the image. Selecting different image feature representations reflects the selection of the model for different image information in the sample image, can capture the features of the sample object from multiple dimensions, makes the prediction of the key point position by the model more comprehensive and accurate, reduces the limitations caused by a single feature representation, and improves the generalization ability and prediction accuracy of the model.

[0251] Further, when predicting the jth key point based on different image analysis angles, the jth key point is predicted based on different mapping relationships, and the mapping relationship is used to represent the relationship between the image feature representation and the output probability distribution parameter. Different mapping relationships reflect different degrees of attention of the model to the image feature representation. By outputting the probability distribution parameter through multiple mapping relationships, information in the image feature representation can be fully mined, the model can predict the position of the key point from different emphases, and multiple possibilities are considered comprehensively, so as to improve the accuracy and reliability of the model in predicting the position of the key point.

[0252] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete related hardware. The program can be stored in a computer readable storage medium. The computer readable storage medium mentioned above can be a read-only memory, a disk or an optical disk.

[0253] Those skilled in the art can understand that, in one or more examples described above, the functions described in the embodiments of the present application can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes computer storage medium and communication medium, and the communication medium includes any medium that facilitates the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0254] Any combination of the technical features in the above embodiments can be made, and for the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the description.

[0255] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for training a keypoint prediction model, executed by a computer device, the method comprising: Obtain a sample image containing a sample object, wherein the sample image is labeled with the reference coordinates corresponding to n key points of the sample object, where n is a positive integer; Using a sample prediction model, the probability distribution of the positions of key points of the sample object in the sample image is predicted, resulting in multiple probability distribution parameters corresponding to each of the n key points. The multiple probability distribution parameters corresponding to the j-th key point are parameters predicted based on different image analysis angles. These multiple probability distribution parameters characterize the mixed probability distribution of the position of the j-th key point, where j ≤ n and j is a positive integer. The sample prediction model is trained based on the reference coordinates corresponding to the j-th key point and multiple probability distribution parameters corresponding to the j-th key point to obtain a key point prediction model, which is used to predict the key point coordinates of the object to be identified in the image.

2. The method according to claim 1, wherein the probability distribution parameters include component parameters and component weights, the component parameters being used to characterize a sub-probability distribution of the mixed probability distribution of the key point, and the component weights being used to characterize the confidence level of the sub-probability distribution in the mixed probability distribution; The method involves using a sample prediction model to predict the probability distribution of the locations of key points of the sample object in the sample image, thereby obtaining multiple probability distribution parameters corresponding to the n key points, including: The features of the sample image are extracted by the feature extraction network in the key point prediction model to obtain the image feature representation corresponding to the sample image; The image feature representation is analyzed by the hybrid density network in the key point prediction model to predict the probability distribution of the location of the key points of the sample object in the sample image, and to obtain multiple component parameters corresponding to the n key points and the component weights corresponding to the multiple component parameters.

3. The method according to claim 2, wherein training the sample prediction model based on the reference coordinates corresponding to the j-th key point and multiple probability distribution parameters corresponding to the j-th key point to obtain a key point prediction model includes: The target loss is determined based on the reference coordinates corresponding to the j-th key point, the multiple component parameters corresponding to the j-th key point, and the component weights corresponding to the multiple component parameters. The key point prediction model is obtained by training the sample prediction model using the target loss.

4. The method according to claim 3, wherein the component parameters include coordinate mean and coordinate standard deviation, the coordinate mean is used to characterize the mean parameter of the sub-probability distribution corresponding to the component parameter, and the coordinate standard deviation is used to characterize the standard deviation parameter of the sub-probability distribution corresponding to the component parameter; The determination of the target loss based on the reference coordinates corresponding to the j-th key point, multiple component parameters corresponding to the j-th key point, and the component weights corresponding to the multiple component parameters includes: Calculate the coordinate difference between the mean of multiple coordinates corresponding to the j-th key point and the reference coordinates corresponding to the j-th key point; Based on the component weights corresponding to the multiple component parameters, the multiple coordinate differences corresponding to the j-th key point are weighted and fused to obtain the target coordinate difference. Based on the component weights corresponding to the multiple component parameters, the target standard deviation is obtained by weighted fusion of the multiple coordinate standard deviations corresponding to the j-th key point. The target loss is determined based on the target coordinate difference and the target standard deviation.

5. The method according to claim 4, wherein the key points include key points in two-dimensional form, and the coordinate standard deviation includes a first standard deviation and a second standard deviation; the first standard deviation is used to characterize the degree of dispersion between each predicted position in the probability distribution corresponding to the component parameter and the abscissa of the coordinate mean, and the second standard deviation is used to characterize the degree of dispersion between each predicted position in the probability distribution corresponding to the component parameter and the ordinate of the coordinate mean.

6. The method according to any one of claims 1 to 5, wherein the plurality of probability distribution parameters include at least one of the probability distribution parameters of a Gaussian distribution and the probability distribution parameters of a Laplace distribution.

7. The method according to any one of claims 1 to 6, wherein predicting the probability distribution of the positions of key points of the sample object in the sample image using a sample prediction model to obtain multiple probability distribution parameters corresponding to the n key points respectively includes: The sample image is divided into multiple image regions; The sample prediction model is used to predict the probability distribution of the location of the key points of the sample object in each image region, thereby obtaining the key point prediction results corresponding to the multiple image regions. The key point prediction results include multiple probability distribution parameters corresponding to the n key points.

8. The method according to claim 7, wherein training the sample prediction model based on the reference coordinates corresponding to the j-th key point and multiple probability distribution parameters corresponding to the j-th key point to obtain a key point prediction model includes: Based on the multiple probability distribution parameters corresponding to the j-th key point in the key point prediction results, the region probabilities corresponding to the multiple image regions are determined respectively, and the region probabilities are used to characterize the probability that the j-th key point belongs to the image region; Based on the reference image region to which the reference coordinates corresponding to the j-th key point belong and the region probabilities corresponding to the plurality of image regions respectively, the region loss corresponding to the plurality of image regions is determined, wherein the reference image region is one of the plurality of image regions; Based on the reference coordinates corresponding to the j-th key point and the multiple probability distribution parameters corresponding to the j-th key point in the key point prediction result, the target loss corresponding to the multiple image regions is determined respectively. The sample prediction model is trained based on the region loss and the target loss to obtain the key point prediction model.

9. The method according to any one of claims 1 to 8, wherein after training the sample prediction model based on the reference coordinates corresponding to the j-th key point and the plurality of probability distribution parameters corresponding to the j-th key point to obtain the key point prediction model, the method further includes: Detect a target object in a target image and obtain a detection box corresponding to the target object. The detection box is used to select the target object in the target image. Based on the detection box corresponding to the target object, a target region in the target image is determined, wherein the target object is contained within the target region; The key point prediction model is used to predict the probability distribution of the location of the key points of the target object in the target area, thereby obtaining multiple probability distribution parameters corresponding to the key points. The coordinates of the key point are determined based on multiple probability distribution parameters corresponding to the key point.

10. The method according to any one of claims 1 to 9, wherein the prediction of the j-th key point based on different image analysis angles includes the prediction of the j-th key point based on different image feature representations, wherein the different image feature representations include at least two of color features, texture features, shape features, spatial relationship features, or edge features.

11. The method according to any one of claims 1 to 10, wherein the prediction of the j-th key point based on different image analysis angles includes the prediction of the j-th key point based on different mapping relationships, wherein the mapping relationship is used to characterize the relationship between the image feature representation and the output probability distribution parameters.

12. A training apparatus for a keypoint prediction model, the apparatus comprising: The acquisition module is used to acquire a sample image containing a sample object, wherein the sample image is labeled with the reference coordinates corresponding to n key points of the sample object, where n is a positive integer; The prediction module is used to predict the probability distribution of the positions of key points of the sample object in the sample image through a sample prediction model, and obtain multiple probability distribution parameters corresponding to the n key points respectively. The multiple probability distribution parameters corresponding to the j-th key point are parameters obtained by predicting the j-th key point based on different image analysis angles. The multiple probability distribution parameters corresponding to the j-th key point are used to characterize the mixed probability distribution of the position of the j-th key point, where j≤n and j is a positive integer. and The training module is used to train the sample prediction model based on the reference coordinates corresponding to the j-th key point and multiple probability distribution parameters corresponding to the j-th key point to obtain a key point prediction model, which is used to predict the key point coordinates of the object to be identified in the image.

13. A computer device comprising a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement a training method for a keypoint prediction model as claimed in any one of claims 1 to 11.

14. A computer-readable storage medium storing at least one program, said at least one program being loaded and executed by a processor to implement a training method for a keypoint prediction model as claimed in any one of claims 1 to 11.

15. A computer program product comprising a computer program that, when executed by a processor, implements a training method for a keypoint prediction model as described in any one of claims 1 to 11.

Images