Feature extraction model training method, feature extraction model application method, and related apparatus

Abstract

Embodiments of the present application disclose a feature extraction model training method, a feature extraction model application method, and a related apparatus. The feature extraction model training method comprises: performing feature extraction on an image sample by means of an initial feature extraction model to obtain the features of key points; on the basis of the features of the key points, determining the probability distributions of the key points; performing sampling on the plurality of probability distributions to obtain a plurality of samples; fitting the plurality of samples by means of an initial normalizing flow model to obtain a simulated probability distribution; and on the basis of a difference between the simulated probability distribution and a real probability distribution, adjusting the model parameters of the initial normalizing flow model and the model parameters of the initial feature extraction model to obtain a feature extraction model. The input data and output data of an overall model composed of a normalizing flow model and the feature extraction model are in the same spatial dimension, and the overall model is trained, so that the capability of capturing intrinsic information of an image by the feature extraction model is improved, thereby improving the accuracy of the position coordinates of the key points.

Description

A training method, application method and related device for feature extraction model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on March 7, 2024, with application number 2024102616435 and application name “A training method and related device for a feature extraction model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to feature extraction technology. Background Art

[0003] Pose estimation is used to predict the coordinates of predefined key points on an entity, thereby identifying the entity's pose based on the coordinates of these key points. For example, human pose estimation involves identifying joints such as the head, shoulders, and elbows as key points, and then determining the person's pose based on the coordinates of these key points.

[0004] Related technologies typically use a regression-based pose estimation method. This involves using a regression model to predict the coordinates of key points in the image to be identified, and then determining the pose based on the coordinates of the key points. However, the accuracy of the key point coordinates determined using this method is low. Summary of the Invention

[0005] In order to solve the above technical problems, the present application provides a training method, an application method and related devices of a feature extraction model to improve the accuracy of the position coordinates of key points.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In one aspect, an embodiment of the present application provides a method for training a feature extraction model, the method comprising:

[0008] Acquire an image sample including a plurality of key points, wherein the key points have real position coordinates;

[0009] Performing feature extraction on the image sample using an initial feature extraction model to obtain features corresponding to the multiple key points;

[0010] Determining probability distributions corresponding to the multiple key points based on the features corresponding to the multiple key points, wherein the probability distributions are used to describe the probabilities of the corresponding key points at various positions in the image sample;

[0011] Sampling from the plurality of probability distributions to obtain a plurality of samples;

[0012] Fitting the initial normalized flow model to the plurality of samples to obtain a simulated probability distribution;

[0013] According to the difference between the simulated probability distribution and the true probability distribution, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted to obtain a feature extraction model corresponding to the initial feature extraction model, and the true probability distribution is determined based on the true position coordinates of the key points corresponding to each of the multiple samples.

[0014] On the other hand, an embodiment of the present application provides an application method of a feature extraction model, the method comprising:

[0015] Obtain the image to be recognized;

[0016] Performing feature extraction on the image to be identified by using a feature extraction model to obtain features corresponding to a plurality of key points to be identified included in the image to be identified, wherein the feature extraction model is trained using the above-mentioned feature extraction model training method;

[0017] Determining probability distributions corresponding to the plurality of key points to be identified, respectively, based on the features corresponding to the plurality of key points to be identified;

[0018] According to the probability distributions respectively corresponding to the multiple key points to be identified, the position coordinates of the multiple key points to be identified in the image to be identified are determined.

[0019] On the other hand, an embodiment of the present application provides a training device for a feature extraction model, the device comprising: an acquisition unit, a feature extraction unit, a conversion unit, a sampling unit, a fitting unit, and an adjustment unit;

[0020] The acquisition unit is used to acquire an image sample including a plurality of key points, wherein the key points have real position coordinates;

[0021] The feature extraction unit is configured to perform feature extraction on the image sample using an initial feature extraction model to obtain features corresponding to the multiple key points;

[0022] The conversion unit is configured to determine probability distributions corresponding to the multiple key points respectively based on the features corresponding to the multiple key points, wherein the probability distributions are used to describe the probabilities of the corresponding key points at various positions in the image sample;

[0023] The sampling unit is used to sample from the multiple probability distributions to obtain multiple samples;

[0024] The fitting unit is configured to perform fitting based on the multiple samples using an initial normalized flow model to obtain a simulated probability distribution;

[0025] The adjustment unit is used to adjust the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model according to the difference between the simulated probability distribution and the true probability distribution to obtain a feature extraction model corresponding to the initial feature extraction model, and the true probability distribution is determined based on the true position coordinates of the key points corresponding to each of the multiple samples.

[0026] On the other hand, an embodiment of the present application provides an application device of a feature extraction model, the device comprising: an acquisition unit, a feature extraction unit, a conversion unit, and a recognition unit;

[0027] The acquisition unit is used to acquire the image to be identified;

[0028] The feature extraction unit is configured to extract features from the image to be identified using the feature extraction model to obtain features corresponding to a plurality of key points to be identified included in the image to be identified, wherein the feature extraction model is trained using the above-mentioned feature extraction model training method;

[0029] The conversion unit is configured to determine the probability distribution corresponding to each of the plurality of key points to be identified based on the features corresponding to each of the plurality of key points to be identified;

[0030] The recognition unit is configured to determine the position coordinates of the multiple key points to be recognized in the image to be recognized according to the probability distributions respectively corresponding to the multiple key points to be recognized.

[0031] In another aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory.

[0032] The memory is used to store a computer program and transmit the computer program to the processor;

[0033] The processor is configured to execute the method described above according to instructions in the computer program.

[0034] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the method described in the above aspects.

[0035] In another aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above aspects.

[0036] It can be seen from the above technical solution that an image sample is obtained, which includes multiple key points with real position coordinates, so as to obtain a real probability distribution based on the real position coordinates of the multiple key points. The image sample is feature extracted by the initial feature extraction model to obtain the features of each key point. In order to avoid the input data and output data not being in the same spatial dimension, the present application no longer directly predicts the position coordinates of the key points based on the features of the key points, but determines the probability of the key points at each position in the image sample based on the features of each key point, that is, obtains the probability distribution of each key point. Multiple samples are sampled from multiple probability distributions, and the multiple samples are fitted by the initial normalized flow model to obtain a simulated probability distribution, which is equivalent to converting the position coordinates of the key points into a probability distribution on the image plane space, thereby causing the input data and output data to be in the same spatial dimension.

[0037] In order to make the simulated probability distribution output by the initial normalized flow model close to the true probability distribution, not only can the model parameters of the initial normalized flow model be adjusted to improve its ability to fit data, but the model parameters of the initial feature extraction model can also be adjusted to make the input data of the initial normalized flow model more realistic. That is, according to the difference between the simulated probability distribution and the true probability distribution, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted to obtain the normalized flow model and the feature extraction model accordingly. Thus, the overall model composed of the initial normalized flow model and the initial feature extraction model has its input data and output data in the same spatial dimension. By training this overall model, the feature extraction model corresponding to the initial feature extraction model will be obtained. The feature extraction model's ability to capture the intrinsic information in the image is improved during the training process, that is, its ability to capture the position coordinate distribution of the underlying key points is improved. Accordingly, the accuracy of the position coordinates of the key points determined by the features obtained based on the feature extraction model is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] FIG1 is a schematic diagram of an application scenario of a training method for a feature extraction model provided in an embodiment of the present application;

[0040] FIG2 is a flow chart of a method for training a feature extraction model according to an embodiment of the present application;

[0041] FIG3 is a schematic diagram of palmprint recognition provided by an embodiment of the present application;

[0042] FIG4 is a schematic diagram of palm detection provided by an embodiment of the present application;

[0043] FIG5 is a schematic diagram of a gesture recognition method provided by an embodiment of the present application;

[0044] FIG6 is a schematic diagram of an object recognition method according to an embodiment of the present application;

[0045] FIG7 a is a schematic diagram of the structure of a training device for a feature extraction model provided in an embodiment of the present application;

[0046] FIG7b is a schematic diagram of the structure of an application device of a feature extraction model provided in an embodiment of the present application.

[0047] FIG8 is a schematic diagram of the structure of a server provided in an embodiment of the present application;

[0048] FIG9 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The embodiments of the present application are described below with reference to the accompanying drawings.

[0050] Pose estimation is a fundamental task in computer vision, widely used in various visual tasks and serving as a preprocessing step for many downstream tasks, such as human motion analysis, liveness detection, and motion capture. In related techniques, the image to be identified is fed into a trained regression model, which then predicts the position coordinates of key points in the image to be identified. This prediction then determines the pose based on the key point coordinates. However, this approach often yields low accuracy in the key point coordinates.

[0051] After research, it was found that the input data and output data of the regression model are not in the same spatial dimension, that is, the input data (i.e., the image to be identified) is two-dimensional data, and the output data (i.e., the key point position coordinates) are one-dimensional data. The regression model needs to constrain the coordinate values ​​through an implicit, non-aligned constraint method. However, the use of this constraint method will cause the regression model to fail to capture the intrinsic information in the image to be identified well, and its ability to capture the underlying key point position distribution is limited. In other words, the accuracy of the key point position coordinates output by it is low.

[0052] In order to solve the above problems, an embodiment of the present application provides a training method and related devices for a feature extraction model, which is composed of an initial normalized flow model and an initial feature extraction model to form an overall model, wherein the input data and output data are in the same spatial dimension. By training the overall model, a feature extraction model corresponding to the initial feature extraction model is obtained, and its ability to capture the intrinsic information in the image is improved, that is, the representation ability in capturing the position coordinate distribution of the underlying key points is improved. Accordingly, the accuracy of the position coordinates of the key points determined by the features subsequently obtained based on the feature extraction model is higher.

[0053] The training and application methods of the feature extraction model provided in this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, the Internet of Things, identity recognition, the film and television industry, and healthcare. It should be noted that the following application scenarios are only examples, and the training and application methods of the feature extraction model provided in this embodiment can also be applied to other scenarios, without limitation here. The following three scenarios are used as examples for illustration.

[0054] Scenario 1: Autonomous driving scenario.

[0055] The vehicle captures the environmental images around the vehicle in real time. Taking the identification of pedestrians in front of the vehicle as an example, the features of the key points of the human body in the environmental image can be extracted through the feature extraction model, and then the position coordinates of the key points of the human body are determined based on the features of the key points of the human body, so as to obtain the posture of the pedestrian in front of the vehicle, and then determine the vehicle's driving speed and direction, etc.

[0056] Scenario 2: Identity recognition scenario.

[0057] As people pay more attention to privacy issues, palmprint recognition has gradually become popular in application scenarios such as payment and identity verification. The feature extraction model can be used to extract the features of the palm key points in the image to be identified, and then the position coordinates of the palm key points are determined based on the features of the palm key points. The position of the palm is circled in the image to be identified, and the image of the palm area is passed to the subsequent palmprint recognition module, thereby improving the accuracy of subsequent palmprint recognition.

[0058] Scene three, action recognition scene.

[0059] By using a trained feature extraction model to extract features from the image to be identified, the features of the key points of the entity included in the image to be identified are obtained. Then, based on the features of the key points of the entity, the position coordinates of the key points of the entity are determined to obtain the posture of the entity, thereby realizing action recognition, gesture recognition, gait recognition, etc. for entities such as human beings or animals, and then determining whether the entity has fallen or has disease signals. It can also be used to realize automatic teaching of sports or dance or production of movie special effects, etc.

[0060] The training method and application method of the feature extraction model provided in this application can be executed by a computer device with feature extraction capabilities, such as a terminal device and a server. Among them, the terminal device can specifically be a desktop computer, a laptop computer, a smart phone, a tablet computer, an Internet of Things device, an aircraft and a portable wearable device. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart car device, etc. The smart car device can be a car navigation terminal and a car computer, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc., but is not limited to this; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0061] In order to facilitate understanding of the training method of the feature extraction model provided in the embodiment of the present application, the application scenario of the training method of the feature extraction model is exemplarily introduced below, taking the execution subject of the training method of the feature extraction model as a server as an example.

[0062] Refer to Figure 1, which is a schematic diagram of an application scenario of a method for training a feature extraction model provided in an embodiment of the present application. As shown in Figure 1, the application scenario includes a server 100. Server 100 can be an independent server for training a feature extraction model. After completing the training of the feature extraction model, the trained feature extraction model can be deployed on a server or terminal device corresponding to the product to provide services such as feature extraction and posture recognition. Server 100 can also be a server that provides corresponding services for various products. For example, the services provided can be feature extraction of images of entities including human bodies or animals. The following is an example of training a feature extraction model using server 100.

[0063] The server 100 obtains an image sample, as shown in FIG1 . The image sample may be a plurality of images waiting for human posture recognition, and the image sample includes a plurality of key points with real position coordinates, so as to obtain a real probability distribution of the key points based on the real position coordinates. The image sample is subjected to feature extraction by the initial feature extraction model to obtain the features of each key point in the image sample. In order to avoid the input data and the output data not being in the same spatial dimension, the present application no longer directly predicts the position coordinates of the key points based on the features of the key points, but determines the probability of the key points at each position in the image sample based on the features of each key point, that is, obtains the probability distribution of each key point. A plurality of samples are sampled from a plurality of probability distributions, and the plurality of samples are fitted by the initial normalized flow model to obtain a simulated probability distribution, which is equivalent to converting the position coordinates of the key points into a probability distribution on the image plane space, thereby causing the input data and the output data to be in the same spatial dimension.

[0064] In order to make the simulated probability distribution output by the initial normalized flow model approximate the true probability distribution of key points in the image sample, not only can the model parameters of the initial normalized flow model be adjusted to improve its ability to fit data, but the model parameters of the initial feature extraction model can also be adjusted to make the input data of the initial normalized flow model more accurate. That is, according to the difference between the simulated probability distribution and the true probability distribution, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted to obtain the feature extraction model and the normalized flow model. Thus, the overall model composed of the initial normalized flow model and the initial feature extraction model has its input data and output data in the same spatial dimension. By training this overall model, a feature extraction model corresponding to the initial feature extraction model is obtained. The feature extraction model's ability to capture the intrinsic information in the image is improved during the training process, that is, its ability to capture the position coordinate distribution of the underlying key points is improved. Accordingly, the accuracy of the position coordinates of the key points determined by the features obtained based on the feature extraction model is also higher.

[0065] The feature extraction model training method provided in the embodiment of the present application can be executed by the server. However, in other embodiments of the present application, the terminal device can also have similar functions as the server to execute the feature extraction model training method provided in the embodiment of the present application, or the terminal device and the server can jointly execute the feature extraction model training method provided in the embodiment of the present application, and this embodiment does not limit this.

[0066] The following is a detailed introduction to a training method for a feature extraction model provided by this application through a method embodiment.

[0067] See Figure 2, which is a flow chart of a method for training a feature extraction model provided in an embodiment of the present application. For ease of description, the following embodiment will still be described using the server as the execution subject of the method for training the feature extraction model. As shown in Figure 2, the method for training the feature extraction model includes the following steps:

[0068] S201: Acquire an image sample including a plurality of key points.

[0069] Image samples contain multiple keypoints. Keypoints are specific parts of an entity used to describe its posture and movement in gesture recognition tasks. For example, in human gesture recognition, keypoints might be the head, neck, shoulders, elbows, wrists, waist, knees, and ankles. Accurately detecting the locations of keypoints improves the accuracy and robustness of gesture recognition, enabling better application in real-world scenarios.

[0070] Moreover, each key point included in the image sample has a real position coordinate. The real position coordinate is the coordinate that can accurately describe the position of the key point and can be obtained through manual calibration.

[0071] It is understandable that in the specific implementation of this application, if the image involves user information, facial images and other related data, when the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain separate permission or consent from the user, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0072] S202: Perform feature extraction on the image sample using the initial feature extraction model to obtain features corresponding to multiple key points.

[0073] The initial feature extraction model is an untrained feature extraction model used to extract features from input data. For example, feature extraction is performed on an image sample to obtain features corresponding to each keypoint in the image sample. The features corresponding to a keypoint indicate the keypoint's unique properties or characteristics, such as corners, edges, and textures. These features can be used to describe and identify keypoints, thereby extracting important information and structure from the image.

[0074] The embodiments of the present application do not specifically limit the initial feature extraction model. For example, the initial feature extraction model may include a backbone network and a convolution layer. The backbone network is used to perform feature extraction, and the convolution layer is used to adjust the dimension of the features output by the backbone network. For example, an image sample is input into the backbone network, and feature extraction is performed through the backbone network to obtain a feature map. The feature map is input into the convolution layer to obtain a feature f, and the size of the feature f is K×fw×fh. Wherein, K is the number of key points included in the image sample, fw is the width of feature f, and fh is the length of feature f. The feature map is the output obtained by performing a convolution operation on the convolution kernel in the backbone network and the input image (such as an image sample), reflecting certain features extracted from the input image under the action of the convolution kernel, such as color, texture, shape, etc.

[0075] S203: Determine the probability distribution corresponding to the multiple key points according to the features corresponding to the multiple key points.

[0076] To avoid situations where the input and output data are not in the same spatial dimension, this application no longer directly predicts the position coordinates of key points based on their features. Instead, it determines the probability of the key point at each position in the image sample based on the features of each key point, thus obtaining the probability distribution of each key point. Generally, there is a certain correlation between the features of key points and the probability distribution. For example, the features of some key points may be related to the peaks in the probability distribution.

[0077] A probability distribution describes the probability of a keypoint at each location in an image sample. For example, the probability distribution for an elbow keypoint identifies the probability that each pixel in the image sample belongs to that elbow keypoint. For example, the probability that the pixel at position (2,3) in the image sample belongs to the elbow keypoint is 50%, and the probability that the pixel at position (3,4) belongs to the elbow keypoint is 30%. It can be understood that in the probability distribution corresponding to each keypoint, the sum of the probabilities at all locations in the image sample is 1. The probability distribution of a keypoint refers to the probability or density of the keypoint's occurrence in an image. Probability distributions can clarify the distribution and density of keypoints in an image, thereby better understanding the image's content and structure.

[0078] The embodiment of the present application does not specifically limit the method of converting features into probability distribution. Those skilled in the art can set it according to actual needs. For example, the feature is converted into probability distribution through argmax function. For another example, the feature is converted into probability distribution through softmax function, which can be referred to formula (1). i =Softmax(f i ) (1)

[0079] Among them, P i is the probability distribution corresponding to the i-th key point, fi is the feature of the i-th key point.

[0080] Compared to converting features into probability distributions through the argmax function, converting features into probability distributions through the softmax function does not cause gradient truncation and can be trained end-to-end.

[0081] S204: Sampling is performed from multiple probability distributions to obtain multiple samples.

[0082] Each key point corresponds to a probability distribution. Taking a key point as an example, sampling is performed from the probability distribution corresponding to the key point to obtain a sample corresponding to the key point. Thus, sampling is performed for the probability distribution corresponding to each key point to obtain multiple samples.

[0083] As a possible implementation, since the probability distribution may be a discrete distribution, sampling in a discrete distribution is more difficult. Therefore, the discrete distribution can be viewed as a classification problem. Each position coordinate in the probability distribution is regarded as a category, and the probability value corresponding to each position coordinate is added to the random noise to obtain multiple perturbation probabilities. Among them, the random noise follows the Gumbel distribution, and the Gumbel distribution can be sampled from the category distribution. Moreover, the perturbation probability can be differentiated, so that the gradient information is not lost during the model training process. Finally, according to the annealing parameter, the multiple perturbation probabilities are normalized to obtain the samples corresponding to the probability distribution. The above process is performed on multiple probability distributions to obtain multiple samples.

[0084] Continuing with the aforementioned i-th key point as an example, the probability distribution P corresponding to the i-th key point is i , which is a discrete distribution, and sampling in a discrete distribution is very difficult. Therefore, the discrete distribution is regarded as a classification problem, that is, each position coordinate in the probability distribution is regarded as a category, and the probability value corresponding to each position coordinate can be expressed as π 11 ,π 12 ,,……,,π 1fh ,π 21 ,……π fwfh , the sum of all probability values ​​is 1. Gumbel Max can be applied to the probability distribution P i Extract samples from the equation (2). j +logπ j ),1≤π j ≤fw×fh (2)

[0085] Among them, z is the sample, fw is the width of the feature of the i-th key point, fh is the length of the feature of the i-th key point, π jis the probability value of the jth position coordinate in the probability distribution corresponding to the i-th key point, g j is an independent and identically distributed random variable sampled from the standard Gumbel distribution. The function G(x) of the Gumbel distribution can be expressed as (3).

[0086] By inverting formula (3) from the uniform distribution, we can get formula (4). j =-log(-log(∈ j )),∈ j ~U(0,1) (4)

[0087] Among them, g j is an independent and identically distributed random variable sampled from a standard Gumbel distribution, ∈ j is a random sample from a uniform distribution.

[0088] Since the argmax in Formula 2 is not differentiable, a differentiable approximate version of Gumbel Softmax is used instead of Gumbel Max, see Formula (5).

[0089] Among them, z is the sample, fw is the width of the feature of the i-th key point, fh is the length of the feature of the i-th key point, π j is the probability value corresponding to the jth position coordinate in the probability distribution corresponding to the i-th key point, g j is an independent and identically distributed random variable sampled from the standard Gumbel distribution, τ is the annealing parameter, τ>0.

[0090] It should be noted that when τ is closer to 0, Formula (5) is closer to Formula (2), but the problem of vanishing gradients will be more serious. When τ is larger, Formula (5) is closer to a uniform distribution, and the training stability will be stronger. As a possible implementation method, during the model training process, you can start with a slightly larger τ and then slowly anneal it to a value close to 0.

[0091] Therefore, when the probability distribution corresponding to the key point is a discrete distribution, in order to avoid the high difficulty of sampling in the discrete distribution, the embodiment of the present application regards the discrete distribution as a classification problem, that is, each position coordinate on the feature map is regarded as a category, and the probability value corresponding to each position coordinate is added to the random noise respectively to obtain multiple disturbance probabilities. Among them, the random noise obeys the Gumbel distribution, and the Gumbel distribution can be sampled from the category distribution, thereby converting the discrete distribution into a problem of sampling in the category. Moreover, the obtained disturbance probability can be differentiated, and the gradient information is not lost during the model training process, thereby improving the accuracy of the model. Finally, according to the annealing parameter, the multiple disturbance probabilities are normalized to obtain samples corresponding to the probability distribution. This ensures the accuracy of the model training.

[0092] S205: Fitting the initial normalized flow model to multiple samples to obtain a simulated probability distribution.

[0093] The initial normalized flow model is a normalized flow model that has not yet been trained. The normalized flow model can transform any data distribution into a simple base distribution by constructing a reversible transformation. Multiple samples are fed into the initial normalized flow model for fitting, resulting in a probability distribution describing the position coordinates of multiple key points, known as the simulated probability distribution.

[0094] Through the normalized flow model, the input data, that is, the potential distribution of the position coordinates of the key points, can be learned and understood. This potential distribution can describe the inherent structure and regularity of the data, so that the normalized flow model can better understand the input data and improve the accuracy of the position coordinates of the key points.

[0095] Continuing with the previous example, we can use multiple samples obtained through formula (5) to construct the probability distribution shape of the initial normalized flow model, which is centered on the position coordinates of the key points predicted by the initial normalized flow model. As a possible implementation method, we can obtain the simulated probability distribution through reparameterization, as shown in formula (6).

[0096] Among them, P feat To simulate the probability distribution, Flow.logP is the logarithmic probability of a given data point calculated by the initial normalized flow model, z is the sample, o = (o x , o y ) is the position coordinate of the key point predicted by the initial normalized flow model, σ o is the variance of the sample. As a possible implementation method, the variance can be set from large to small to make the model converge quickly.

[0097] S206: According to the difference between the simulated probability distribution and the true probability distribution, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted to obtain a feature extraction model corresponding to the initial feature extraction model.

[0098] The simulated probability distribution is the probability distribution of the position coordinates of the key points learned by the initial normalized flow model, and the true probability distribution is the probability distribution of the true position coordinates of the key points, which can be determined based on the true position coordinates of the key points corresponding to the samples. Therefore, the learning ability of the initial normalized flow model can be reflected by the difference between the simulated probability distribution and the true probability distribution. Furthermore, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model can be adjusted based on the difference between the simulated probability distribution and the true probability distribution, such as based on a training method that makes the difference between the simulated probability distribution and the true probability distribution smaller and smaller. The process of continuously adjusting the model parameters is the training process of the model, thereby obtaining a trained feature extraction model and / or normalized flow model, so that the simulated probability distribution obtained by fitting the normalized flow model becomes closer and closer to the true probability distribution, and the features extracted by the feature extraction model are more accurate.

[0099] In addition, related technologies also generate high-resolution heat maps based on the feature maps obtained by the feature extraction model. A heat map is generated for each key point. In the heat map, the locations where the key point is most likely to appear are marked with a high probability, and the remaining locations are marked with a low probability. Finally, based on the heat map, the argmax function is used to obtain the position coordinates of the key points. However, due to the limited size of the heat map, the position coordinates of the key points obtained using the argmax function often have quantization errors, resulting in low accuracy of the position coordinates.

[0100] Based on this, the embodiment of the present application believes that the features output by the feature extraction model (or the feature map output by the backbone network) have a probability distribution, so the features output by the feature extraction model are converted into a probability distribution, and the initial normalized flow model is used for learning to obtain a simulated probability distribution. The normalized flow model can fit the corresponding shape according to the input sample and can convert the distribution in one space into the distribution in another space, so that the initial normalized flow model can be trained based on the difference between the simulated probability distribution and the true probability distribution.

[0101] Moreover, the fitting ability of the initial normalized flow model is not only related to its own parameters, but also to the accuracy of the features output by the initial feature extraction model. If the accuracy of the features of the key points output by the initial feature extraction model is low, the initial normalized flow model will learn based on the probability distribution corresponding to the features with low accuracy, which will affect its learning effect. Therefore, during the model training process, it is necessary not only to adjust the model parameters of the initial normalized model, but also to adjust the model parameters of the initial feature extraction model at the same time, so that the initial feature extraction model can extract features more accurately and the initial normalized flow model can also fit the probability distribution more accurately. Therefore, this method does not use heat maps, does not compress the features of key points, and improves the accuracy of the position coordinates of the key points obtained subsequently.

[0102] It can be seen from the above technical solution that an image sample is obtained, which includes multiple key points with real position coordinates, so as to obtain a real probability distribution based on the real position coordinates of the multiple key points. The image sample is feature extracted by the initial feature extraction model to obtain the features of each key point. In order to avoid the input data and output data not being in the same spatial dimension, the present application no longer directly predicts the position coordinates of the key points based on the features of the key points, but determines the probability of the key points at each position in the image sample based on the features of each key point, that is, obtains the probability distribution of each key point. Multiple samples are sampled from multiple probability distributions, and the multiple samples are fitted by the initial normalized flow model to obtain a simulated probability distribution, which is equivalent to converting the position coordinates of the key points into a probability distribution on the image plane space, thereby causing the input data and output data to be in the same spatial dimension.

[0103] In order to make the simulated probability distribution output by the initial normalized flow model close to the true probability distribution, not only can the model parameters of the initial normalized flow model be adjusted to improve its ability to fit data, but the model parameters of the initial feature extraction model can also be adjusted to make the input data of the initial normalized flow model more realistic. That is, according to the difference between the simulated probability distribution and the true probability distribution, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted, and the normalized flow model and the feature extraction model are obtained accordingly. Thus, the overall model composed of the initial normalized flow model and the initial feature extraction model has input data and output data in the same spatial dimension. By training the overall model, the feature extraction model corresponding to the initial feature extraction model will be obtained. The ability of the feature extraction model to capture the intrinsic information in the image is improved during the training process, that is, the representation ability of capturing the position coordinate distribution of the underlying key points is improved. Accordingly, the accuracy of the position coordinates of the key points determined by the features obtained based on the feature extraction model is higher.

[0104] The embodiments of this application do not specifically limit the loss function used in the model training process. The following uses three loss functions as examples to illustrate them respectively.

[0105] Method 1.

[0106] See formula (7). L = L flow (7)

[0107] Among them, L is the overall loss function, L flow is the first difference, which is used to describe the difference between the simulated probability distribution and the true probability distribution, and can be expressed as formula (8). flow =-logP feat (8)

[0108] Among them, P feat To simulate the probability distribution, please refer to the above formula (6).

[0109] As a possible implementation, model training can be achieved through maximum likelihood estimation and formula (8). Maximum likelihood estimation is a commonly used parameter estimation method that estimates parameters by maximizing the likelihood function of the data. In the normalized flow model, maximum likelihood estimation can be used to estimate the parameters of the transformation function so that the simulated probability distribution generated by the normalized flow model is as similar as possible to the true probability distribution.

[0110] Method 2.

[0111] While using maximum likelihood estimation alone as the overall loss function, as described in Method 1, can also train a normalized flow model, it may not be sufficient to fully fit the complex distribution of the data. In this case, combining it with a standard probability distribution can provide more flexibility and degrees of freedom, enabling the normalized flow model to better fit the data. See A1-A4 for details.

[0112] A1: Determine the standard probability distribution corresponding to the multiple key points based on their true position coordinates.

[0113] The embodiments of the present application do not specifically limit the standard probability distribution, and those skilled in the art can set it according to actual needs. Taking the Gaussian distribution as an example, parameters such as the mean and variance of the Gaussian distribution can be used as prior information, that is, the variance and mean can be determined based on the real position coordinates of the key points, thereby obtaining a standard probability distribution that conforms to the Gaussian distribution, and then guiding the normalized flow model to fit a specific data distribution. By combining the Gaussian distribution, the normalized flow model can take advantage of the advantages of the Gaussian distribution in probability distribution modeling, such as simplicity, flexibility, and continuity. At the same time, the reversible transformation characteristics of the Gaussian distribution also enable the normalized flow model to achieve more complex data transformation and generation tasks.

[0114] A2: Sampling from multiple standard probability distributions to obtain multiple target sampling points.

[0115] Continuing to use Gaussian distribution as the standard probability distribution, taking one key point among multiple key points as an example, the target sampling point conforms to the standard probability distribution and can be expressed as formula (9).

[0116] Among them, P gt is the standard probability distribution, t=(t x ,,t y ) is the target sampling point obtained by sampling, μ=(μ x , μ y ) is the real position coordinate, σ g is the variance obtained based on the real position coordinates.

[0117] Thus, based on the above method, the target sampling point of each key point is obtained, and then multiple sampling points are obtained.

[0118] A3: Through the initial normalized flow model, multiple target sampling points are fitted to obtain the predicted probability distribution for multiple key points.

[0119] Continuing with a key point as an example, the target sampling point obtained by sampling the standard probability distribution corresponding to the key point is input into the initial normalized flow model, so as to guide the shape of the probability distribution output by the initial normalized flow model based on the standard probability distribution.

[0120] Through the initial normalized flow model, fitting is performed according to multiple target sampling points to obtain the predicted probability distribution for each key point, which can be expressed as formula (10).

[0121] Among them, P pred is the predicted probability distribution, Flow.logP is the logarithmic probability of the target sampling point calculated by the initial normalized flow model, t is the target sampling point obtained by sampling, μ is the real position coordinate, σ g is the variance obtained based on the target sampling point.

[0122] A4: Adjust model parameters of the initial normalized flow model and model parameters of the initial feature extraction model according to the first difference and the second difference.

[0123] The first difference is the difference between the simulated probability distribution and the true probability distribution, and the second difference is the difference between the predicted probability distribution and the standard probability distribution.

[0124] The loss function based on the first difference and the second difference can be expressed as formula (11): L = L flow +Lpred (11)

[0125] Among them, L is the overall loss function, L flow is the first difference, L pred is the second difference, which can be expressed as formula (12). pred =logP gt --logP pred (12)

[0126] Among them, L pred is the second difference, P gt is the standard probability distribution, see formula (9) for details, P pred To predict the probability distribution, refer to formula (10) for details. This is equivalent to applying the KL divergence constraint to bring the predicted probability distribution and the standard probability distribution closer.

[0127] Therefore, although the probability distribution of the true position coordinates is often unknown, it may not be possible to directly calculate the KL divergence. In this case, it can be assumed that the distribution of the true position coordinates is a standard probability distribution such as the Gaussian distribution, and sampled on the standard probability distribution to approximately calculate the KL divergence. Since the standard probability distribution has a simple mathematical form and is computationally convenient, it can be used as a reasonable approximation to evaluate the difference between the model predicted data and the true data distribution, so as to guide the initial normalized flow model to learn quickly and accurately. That is, when optimizing the model parameters of the initial normalized flow model, it is necessary not only to minimize the difference between the simulated probability distribution and the true probability distribution, but also to make the predicted probability distribution generated by the initial normalized flow model as close to the standard probability distribution as possible. This not only improves the training efficiency of the model, but also improves the fitting ability and flexibility of the normalized flow model, so that the normalized flow model can better adapt to situations with different data distributions.

[0128] Method three.

[0129] When training a model using Method 2, it's easy to hit a dead end because the model has no idea about the coordinates and variance at the beginning of training. Therefore, based on Method 2, we added the constraint of an ideal probability distribution (see B1-B2).

[0130] B1: According to the features corresponding to the multiple key points, the ideal probability distribution corresponding to the multiple key points is obtained.

[0131] The embodiments of the present application do not specifically limit the ideal probability distribution, and those skilled in the art can set it according to actual needs. Taking the image sample as an example of an image including the face of the subject, the ideal probability distribution can obey the Gaussian distribution or the Laplace distribution, which can be expressed as formula (13). Therefore, for the image sample including the face of the subject, although the key distance of the face of the subject is relatively close and the recognition is relatively complex, compared with other distributions, the Gaussian distribution or the Laplace distribution is more stable and convenient, and has higher stability while ensuring accuracy.

[0132] Among them, P optimal is the ideal probability distribution, gaussian is the Gaussian distribution, laplace is the Laplace distribution, σ g is the variance obtained based on the features of the key points, t is the sample obtained based on the features of the key points, and μ is the mean obtained based on the features of the key points.

[0133] B2: Adjusting model parameters of the initial normalized flow model and model parameters of the initial feature extraction model according to the first difference, the second difference, and the third difference.

[0134] Among them, the first difference is the difference between the simulated probability distribution and the true probability distribution, the second difference is the difference between the predicted probability distribution and the standard probability distribution, and the third difference is the difference between the standard probability distribution and the ideal probability distribution.

[0135] The loss function based on the first difference, the second difference and the third difference can be expressed as formula (14). flow +L pred +L optimal (14)

[0136] Among them, L is the overall loss function, L rlow is the first difference, L pred is the second difference, L optimal is the third difference, which can be expressed as formula (15). optimal =logP gt --logP optimal (15)

[0137] Among them, L optimal is the third difference, P gt is the standard probability distribution, see formula (9) for details, P optimal is the ideal probability distribution, see formula (13) for details.

[0138] Therefore, by implementing model training based on an overall loss function that includes the first, second, and third differences, it is possible to optimize the model parameters of the initial normalized flow model, not only to minimize the difference between the simulated probability distribution and the true probability distribution, but also to make the predicted probability distribution generated by the initial normalized flow model as close as possible to the standard probability distribution. Moreover, by constraining the ideal probability distribution, the initial normalized flow model must consider deviations from the ideal distribution while fitting the data. This can prevent the model from falling into undesirable local optimal solutions in the early stages of training and improve the model's stability and generalization ability.

[0139] As a possible implementation, the overall loss function can be expressed not only based on formula (14), but also based on formula (16). That is, the overall loss function is determined based on the first difference, the second difference, the third difference, the first weight, and the second weight, so as to adjust the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model based on the training direction of minimizing the overall loss function, thereby obtaining a feature extraction model and / or a normalized flow model. L = L flow +αL pred +βL optimal (16)

[0140] Among them, L is the overall loss function, L flow is the first difference, L pred is the second difference, L optimal The third difference is obtained by adjusting the second difference, α is a first weight, and is used to reduce the difference between the second difference and the first difference by adjusting the second difference. β is a second weight, and is used to reduce the difference between the third difference and the first difference by adjusting the third difference.

[0141] Therefore, through the first weight and the second weight, the differences between the first difference, the second difference and the third difference can be made smaller, thereby improving the stability of the model training process.

[0142] As a possible implementation method, if the number of iterations is less than a preset iteration threshold, and the difference between the overall loss function values ​​obtained in two adjacent model trainings is greater than a preset difference threshold, the second weight is increased to obtain an updated second weight.

[0143] The embodiment of the present application does not specifically limit the size of the preset iteration number threshold, and those skilled in the art can set it according to actual needs. If the number of iterations is less than the preset iteration number threshold, and the difference in the overall loss function value obtained in two adjacent model trainings is greater than the preset difference threshold, it means that in the early stage of model training, the model fluctuates greatly, that is, the model training is unstable, so the size of the second weight is increased to improve the stability of model training.

[0144] As can be seen from the above, the position coordinates of the key points can also be obtained based on the heat map method. In the related technology, although the softmax function can be used instead of the argmax function, that is, the heat map is converted into a probability distribution through the softmax function to avoid the generation of gradient truncation, thereby solving the problem of being unable to perform end-to-end training. However, the method of converting the features into a heat map and then obtaining the probability distribution based on the heat map, because the softmax function will change the shape of the probability distribution, there will be a certain difference between the heat map learned by the model and the final probability distribution, resulting in a decrease in the accuracy of the position coordinates of the key points. Moreover, in the process of determining the position coordinates of the key points, the heat map will be used. The heat map has a large amount of calculation, which will result in a long determination time. It is not suitable for scenarios with high time requirements such as real-time posture recognition.

[0145] Based on this, the embodiment of the present application no longer uses a heat map to determine the coordinate position of the key points. Instead, it extracts the features of the key points included in the image sample based on the initial feature extraction model, then converts the key point features into a probability distribution, and fits them through the initial normalized flow model. Since the normalized flow model can fit any probability distribution, it can be used to model the probability distribution of the key points. Thus, while training the fitting ability of the initial normalized flow model, the feature extraction ability of the initial feature extraction model is adjusted to obtain a feature extraction model.

[0146] After obtaining the image to be identified, feature extraction is performed on the image to be identified through a feature extraction model to obtain features corresponding to multiple key points to be identified included in the image to be identified. According to the features corresponding to the multiple key points to be identified, the probability distribution corresponding to the multiple key points to be identified is determined. According to the probability distribution of the multiple key points to be identified, the position coordinates of the multiple key points to be identified in the image to be identified are determined.

[0147] Therefore, the initial normalized flow model is followed by the feature extraction model. This eliminates the need for the normalized flow model when identifying the position coordinates of key points in the image to be recognized, reducing overall time consumption. Furthermore, this approach does not require heatmaps, and the model itself is more lightweight, eliminating any unnecessary time consumption. This approach is extremely time-efficient and suitable for time-sensitive scenarios such as real-time gesture recognition.

[0148] In order to facilitate further understanding of the technical solution provided by the embodiment of the present application, the following is an overall exemplary introduction to the training method of the feature extraction model provided by the embodiment of the present application, taking the execution subject of the training method of the feature extraction model as an example of a server.

[0149] The following first explains the model training process.

[0150] S1: Obtain an image sample including multiple key points.

[0151] Each image sample includes multiple key points with real position coordinates. Taking the human posture recognition scene as an example, the real position coordinates of key points such as the elbow in the image sample can be marked, that is, the position of key points such as the elbow in the image sample.

[0152] S2: Through the initial feature extraction model, feature extraction is performed on the image sample to obtain features corresponding to multiple key points.

[0153] S3: Determine the probability distribution corresponding to the multiple key points according to the features corresponding to the multiple key points.

[0154] The features of key points can be converted into probability distribution by the aforementioned formula (1).

[0155] S4: Sampling from multiple probability distributions to obtain multiple samples.

[0156] The aforementioned formula (5) can be used to sample from the probability distribution corresponding to each key point to obtain multiple samples.

[0157] S5: The initial normalized flow model is fitted based on multiple samples to obtain a simulated probability distribution.

[0158] The shape of the probability distribution of the initial normalized flow model centered on the coordinates of the predicted key point can be constructed based on multiple samples in the manner of the aforementioned formula (6).

[0159] S6: Determine the standard probability distribution of each key point based on its true position coordinates.

[0160] Taking the Gaussian distribution as an example, parameters such as the mean and variance of the Gaussian distribution can be used as prior information. That is, the variance and mean can be determined based on the actual position coordinates of the key points, thereby obtaining a standard probability distribution that conforms to the Gaussian distribution, and then guiding the normalized flow model to fit a specific data distribution.

[0161] S7: Sampling from multiple standard probability distributions to obtain multiple target sampling points.

[0162] The standard probability distribution corresponding to the target sampling point can be expressed as the aforementioned formula (9).

[0163] S8: Through the initial normalized flow model, fitting is performed according to multiple target sampling points to obtain the predicted probability distribution for multiple key points.

[0164] The predicted probability distribution can be expressed as the aforementioned formula (10).

[0165] S9: According to the features corresponding to the multiple key points, the ideal probability distribution corresponding to the multiple key points is obtained.

[0166] The ideal probability distribution can conform to Gaussian distribution or Laplace distribution, as shown in the above formula (13).

[0167] S10: Determine an overall loss function according to the first difference, the second difference, the third difference, the first weight, and the second weight.

[0168] The overall loss function is shown in the above formula (16).

[0169] S11: Based on the training direction of minimizing the overall loss function, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted to obtain the feature extraction model and the normalized flow model.

[0170] After the feature extraction model is trained, the position coordinates of key points can be identified.

[0171] S12: Obtain the image to be recognized.

[0172] S13: Perform feature extraction on the image to be identified through a feature extraction model to obtain features corresponding to a plurality of key points to be identified included in the image to be identified.

[0173] S14: Determine the probability distribution corresponding to the multiple key points to be identified according to the features corresponding to the multiple key points to be identified.

[0174] S15: Determine the position coordinates of the multiple key points to be identified in the image to be identified according to the probability distributions corresponding to the multiple key points to be identified.

[0175] The comparison between the solution of the embodiment of the present application and the other three solutions is shown in Table 1.

[0176] Table 1

[0177] Among them, Solution One is the aforementioned solution for obtaining the position coordinates of key points based on heat maps, but it does not use deconvolution. Solution Two is the aforementioned solution for obtaining the position coordinates of key points based on heat maps, but it uses deconvolution. Solution Three is the aforementioned solution for determining the position coordinates of key points based on regression models. Solution Four is an embodiment of S1-S15 of this application. The mean Average Precision (mAP) is a very important measurement indicator in the field of target detection in machine learning, which is used to measure the performance of target detection algorithms. Generally speaking, the mean average precision of all categories is obtained by taking a comprehensive weighted average of the mean accuracy (AP) of all category detections.

[0178] The following describes the application of the feature extraction model in combination with two application scenarios.

[0179] See Figure 3, which is a schematic diagram of palmprint recognition provided in an embodiment of the present application.

[0180] After obtaining the video to be identified, palm detection is performed on each frame of the video to be identified. As shown in Figure 4, the position of the palm in the image is circled by the detection frame, so that the image to be identified is cropped based on the detection frame to obtain an image that only includes the palm. For multiple frames of images, two operations, hand key point detection and estimation of the position of the detection frame, are continuously performed in a loop based on the palm image. Based on the detection frame obtained from the first frame of the image, the position coordinates of the hand key points of the 2nd to 5th frames of the image are identified in this area. Then, based on the position of the detection frame based on the 6th frame of the image, the position coordinates of the hand key points of the 7th to 10th frames of the image are identified based on the detection frame, and so on. It can be understood that the position coordinates of the hand key points can be obtained by the aforementioned S11-S15 method. After obtaining the position coordinates of the hand key points, the region of interest (ROI) is extracted to achieve palmprint recognition.

[0181] See FIG5 , which is a schematic diagram of gesture recognition provided in an embodiment of the present application.

[0182] After acquiring the image to be recognized, the object to be recognized in the image is identified, that is, target detection is achieved. As shown in Figure 6, the location of the object to be recognized in the image is circled by a detection frame, so that the image to be recognized can be cropped based on the detection frame to obtain an image containing only the object to be recognized. The position coordinates of the key points of the object to be recognized can be obtained through the aforementioned S11-S15 method to achieve object pose estimation, and finally extract the region of interest, thereby achieving action recognition, gesture recognition, gait recognition, etc.

[0183] In response to the training method and application method of the feature extraction model described above, the present application also provides corresponding training devices and application devices for the feature extraction model, so that the training method and application method of the feature extraction model described above can be applied and implemented in practice.

[0184] See Figure 7a, which is a schematic diagram of the structure of a feature extraction model training device provided in an embodiment of the present application. As shown in Figure 7a, the feature extraction model training device 700 includes: an acquisition unit 701, a feature extraction unit 702, a conversion unit 703, a sampling unit 704, a fitting unit 705, and an adjustment unit 706;

[0185] The acquisition unit 701 is configured to acquire an image sample including a plurality of key points, wherein the key points have real position coordinates;

[0186] The feature extraction unit 702 is configured to perform feature extraction on the image sample using an initial feature extraction model to obtain features corresponding to the multiple key points.

[0187] The conversion unit 703 is configured to determine probability distributions corresponding to the multiple key points according to the features corresponding to the multiple key points, wherein the probability distributions are used to describe the probabilities of the corresponding key points at various positions in the image sample;

[0188] The sampling unit 704 is configured to sample from the plurality of probability distributions to obtain a plurality of samples;

[0189] The fitting unit 705 is configured to fit the initial normalized flow model according to the plurality of samples to obtain a simulated probability distribution;

[0190] The adjustment unit 706 is used to adjust the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model according to the difference between the simulated probability distribution and the true probability distribution to obtain a feature extraction model corresponding to the initial feature extraction model, and the true probability distribution is determined based on the true position coordinates of the key points corresponding to each of the multiple samples.

[0191] It can be seen from the above technical solution that an image sample is obtained, which includes multiple key points with real position coordinates, so as to obtain the real probability distribution of the key points based on the real position coordinates. The image sample is feature extracted by the initial feature extraction model to obtain the features of each key point. In order to avoid the input data and the output data not being in the same spatial dimension, the position coordinates of the key points are no longer directly predicted based on the features of the key points. Instead, the probability of the key points at each position in the image sample is determined based on the features of each key point, that is, the probability distribution of each key point is obtained. Multiple samples are sampled from multiple probability distributions, and the multiple samples are fitted by the initial normalized flow model to obtain a simulated probability distribution, which is equivalent to converting the position coordinates of the key points into a probability distribution on the image plane space, thereby causing the input data and the output data to be in the same spatial dimension.

[0192] In order to enable the initial normalized flow model to gradually approach the potential distribution of key points, not only can the model parameters of the initial normalized flow model be adjusted to improve its ability to fit data, but the model parameters of the initial feature extraction model can also be adjusted to make the input data of the initial normalized flow model more realistic. That is, based on the difference between the simulated probability distribution and the true probability distribution, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted to obtain the feature extraction model and the normalized flow model. As a result, the overall model composed of the initial normalized flow model and the initial feature extraction model has its input data and output data in the same spatial dimension. The feature extraction model obtained through training has an improved ability to capture the intrinsic information in the image, that is, its ability to capture the position coordinate distribution of the underlying key points is improved, so that the accuracy of the position coordinates of the key points determined by the features obtained by the feature extraction model is higher.

[0193] As a possible implementation, the conversion unit 703 is further configured to determine the standard probability distributions corresponding to the multiple key points respectively according to the real position coordinates of the multiple key points;

[0194] The sampling unit 704 is further configured to sample from the plurality of standard probability distributions to obtain a plurality of target sampling points;

[0195] The fitting unit 705 is further configured to perform fitting based on the multiple target sampling points using the initial normalized flow model to obtain a predicted probability distribution for the multiple key points;

[0196] The adjustment unit 706 is specifically used to adjust the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model according to a first difference and a second difference, wherein the first difference is the difference between the simulated probability distribution and the true probability distribution, and the second difference is the difference between the predicted probability distribution and the standard probability distribution.

[0197] As a possible implementation, the conversion unit 703 is further configured to obtain ideal probability distributions corresponding to the multiple key points according to the features corresponding to the multiple key points.

[0198] The adjustment unit 706 is specifically used to adjust the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model according to the first difference, the second difference and the third difference, wherein the third difference is the difference between the standard probability distribution and the ideal probability distribution.

[0199] As a possible implementation manner, if the image sample is an image including a face of an object, the ideal probability distribution satisfies a Gaussian distribution or a Laplace distribution.

[0200] As a possible implementation manner, the adjusting unit 706 is specifically configured to:

[0201] determining an overall loss function according to the first difference, the second difference, the third difference, a first weight, and a second weight, wherein the first weight is used to reduce the difference between the second difference and the first difference, and the second weight is used to reduce the difference between the third difference and the first difference;

[0202] Based on the training direction of minimizing the overall loss function, model parameters of the initial normalized flow model and model parameters of the initial feature extraction model are adjusted.

[0203] As a possible implementation method, the device 700 also includes a parameter adjustment unit, which is used to increase the second weight to obtain an updated second weight if the number of iterations is less than a preset iteration threshold and the difference between the overall loss function values ​​obtained in two adjacent model trainings is greater than a preset difference threshold.

[0204] As a possible implementation, the sampling unit 704 is specifically configured to:

[0205] For each of the multiple probability distributions, a plurality of differentiable perturbation probabilities are obtained based on a plurality of probability values ​​and random noise included in the probability distribution, wherein the random noise obeys a Gumbel distribution; and the plurality of perturbation probabilities are normalized according to an annealing parameter to obtain samples corresponding to the probability distribution, wherein the annealing parameter is used to reduce the learning rate during training.

[0206] See Figure 7b, which is a schematic diagram of the structure of a feature extraction model application device provided in an embodiment of the present application. As shown in Figure 7b, the feature extraction model application device 800 includes: an acquisition unit 801, a feature extraction unit 802, a conversion unit 803, and an adjustment unit 804;

[0207] The acquisition unit 801 is used to acquire an image to be recognized;

[0208] The feature extraction unit 802 is configured to extract features from the image to be identified using the feature extraction model to obtain features corresponding to a plurality of key points to be identified included in the image to be identified, wherein the feature extraction model is trained using the above-mentioned feature extraction model training method;

[0209] The conversion unit 803 is configured to determine the probability distribution corresponding to each of the plurality of key points to be identified based on the features corresponding to each of the plurality of key points to be identified;

[0210] The recognition unit 804 is configured to determine the position coordinates of the multiple key points to be recognized in the image to be recognized according to the probability distributions corresponding to the multiple key points to be recognized.

[0211] The present application also provides a computer device, which can be a server or a terminal device. The following describes the computer device provided by the present application from the perspective of hardware entity. Figure 8 shows a schematic diagram of the structure of the server, and Figure 9 shows a schematic diagram of the structure of the terminal device.

[0212] Referring to Figure 8, which is a schematic diagram of a server structure provided in an embodiment of the present application, the server 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors 1422, such as central processing units (CPUs), a memory 1432, and one or more application programs 1442 or a storage medium 1430 for data 1444 (for example, one or more massive storage devices). Among them, the memory 1432 and the storage medium 1430 may be temporary storage or permanent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the processor 1422 may be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the server 1400.

[0213] The server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0214] The steps executed by the server in the above embodiment may be based on the server structure shown in FIG8 .

[0215] Among them, CPU 1422 is used to execute the training method and application method of the above-mentioned feature extraction model.

[0216] Optionally, the CPU 1422 may also execute the method steps of any specific implementation of the training method or application method of the feature extraction model in the embodiments of the present application.

[0217] Referring to Figure 9, which is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Taking the terminal device as a smartphone as an example, Figure 9 shows a block diagram of a partial structure of the smartphone, which includes: a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a wireless fidelity (WiFi) module 1570, a processor 1580, and a power supply 1590. Those skilled in the art will understand that the smartphone structure shown in Figure 9 does not constitute a limitation on smartphones, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0218] The following is a detailed introduction to the various components of a smartphone with reference to FIG9 :

[0219] The RF circuit 1510 may be used for receiving and sending signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is sent to the processor 1580 for processing. In addition, the designed uplink data is sent to the base station.

[0220] The memory 1520 may be used to store software programs and modules. The processor 1580 implements various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1520 .

[0221] The input unit 1530 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smartphone. Specifically, the input unit 1530 may include a touch panel 1531 and other input devices 1532. The touch panel 1531, also known as a touch screen, can collect user touch operations on or near it and drive the corresponding connection device according to a pre-set program. In addition to the touch panel 1531, the input unit 1530 may also include other input devices 1532. Specifically, other input devices 1532 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0222] The display unit 1540 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. The display unit 1540 may include a display panel 1541. Optionally, the display panel 1541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0223] The smartphone may also include at least one sensor 1550, such as a light sensor, a motion sensor, or other sensors. Other sensors that may be included in the smartphone, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., are not described in detail here.

[0224] Audio circuit 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and the smartphone. Audio circuit 1560 converts received audio data into electrical signals and transmits them to speaker 1561, which then converts them into sound signals for output. Microphone 1562, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 1560 and converted into audio data. The audio data is then processed by processor 1580 and transmitted to, for example, another smartphone via RF circuit 1510, or stored in memory 1520 for further processing.

[0225] Processor 1580 is the control center of the smartphone. It connects all parts of the smartphone using various interfaces and lines. It executes or runs software programs and / or modules stored in memory 1520 and accesses data stored in memory 1520 to perform various functions of the smartphone and process data. Optionally, processor 1580 may include one or more processing units.

[0226] The smartphone also includes a power supply 1590 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0227] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0228] In the embodiment of the present application, the memory 1520 included in the smart phone can store a computer program and transmit the computer program to the processor.

[0229] The processor 1580 included in the smart phone can execute the training method or application method of the feature extraction model provided in the above embodiment according to the instructions in the computer program.

[0230] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program, which is used to execute the training method or application method of the feature extraction model provided in the above embodiment.

[0231] Embodiments of the present application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the feature extraction model training method or application method provided in various optional implementations of the above aspects.

[0232] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the above-mentioned storage medium can be at least one of the following media: read-only memory (English: Read-Only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc., various media that can store computer programs.

[0233] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0234] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0235] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Based on the implementation methods provided in the above aspects, the present application can also be further combined to provide more implementation methods. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for training a feature extraction model, performed by a computer device, comprising: Acquire an image sample including a plurality of key points, wherein the key points have real position coordinates; Performing feature extraction on the image sample using an initial feature extraction model to obtain features corresponding to the multiple key points; Determining probability distributions corresponding to the multiple key points based on the features corresponding to the multiple key points, wherein the probability distributions are used to describe the probabilities of the corresponding key points at various positions in the image sample; Sampling from the plurality of probability distributions to obtain a plurality of samples; Fitting the plurality of samples using an initial normalized flow model to obtain a simulated probability distribution; According to the difference between the simulated probability distribution and the true probability distribution, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted to obtain a feature extraction model corresponding to the initial feature extraction model, and the true probability distribution is determined based on the true position coordinates of the key points corresponding to each of the multiple samples.

2. The method according to claim 1, further comprising: Determine, according to the real position coordinates of the multiple key points, the standard probability distributions corresponding to the multiple key points respectively; Sampling from a plurality of the standard probability distributions to obtain a plurality of target sampling points; Fitting the initial normalized flow model according to the plurality of target sampling points to obtain a predicted probability distribution for the plurality of key points; The adjusting the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model according to the difference between the simulated probability distribution and the true probability distribution includes: According to a first difference and a second difference, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model are adjusted, wherein the first difference is the difference between the simulated probability distribution and the true probability distribution, and the second difference is the difference between the predicted probability distribution and the standard probability distribution.

3. The method according to claim 2, further comprising: Obtaining ideal probability distributions corresponding to the plurality of key points, respectively, based on the features corresponding to the plurality of key points; The adjusting, according to the first difference and the second difference, the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model includes: Model parameters of the initial normalized flow model and model parameters of the initial feature extraction model are adjusted according to the first difference, the second difference, and the third difference, wherein the third difference is the difference between the standard probability distribution and the ideal probability distribution. 4 . The method according to claim 3 , wherein if the image sample is an image including a face of a subject, the ideal probability distribution satisfies a Gaussian distribution or a Laplace distribution.

5. The method according to claim 3 or 4, wherein adjusting the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model according to the first difference, the second difference, and the third difference comprises: determining an overall loss function according to the first difference, the second difference, the third difference, a first weight, and a second weight, wherein the first weight is used to reduce the difference between the second difference and the first difference, and the second weight is used to reduce the difference between the third difference and the first difference; Based on the training direction of minimizing the overall loss function, model parameters of the initial normalized flow model and model parameters of the initial feature extraction model are adjusted.

6. The method according to claim 5, further comprising: If the number of iterations is less than a preset iteration threshold, and the difference between the overall loss function values ​​obtained in two adjacent model trainings is greater than a preset difference threshold, the second weight is increased to obtain an updated second weight.

7. The method according to any one of claims 1 to 6, wherein sampling from the plurality of probability distributions to obtain a plurality of samples comprises: For each of the multiple probability distributions, a plurality of differentiable perturbation probabilities are obtained based on a plurality of probability values ​​and random noise included in the probability distribution, wherein the random noise obeys a Gumbel distribution; and the plurality of perturbation probabilities are normalized according to an annealing parameter to obtain samples corresponding to the probability distribution, wherein the annealing parameter is used to reduce the learning rate during training.

8. A method for applying a feature extraction model, executed by a computer device, the method comprising: Obtain the image to be recognized; Performing feature extraction on the image to be identified by using a feature extraction model to obtain features corresponding to a plurality of key points to be identified included in the image to be identified, wherein the feature extraction model is trained using the method according to any one of claims 1 to 7; Determining probability distributions corresponding to the plurality of key points to be identified, respectively, based on the features corresponding to the plurality of key points to be identified; According to the probability distributions respectively corresponding to the multiple key points to be identified, the position coordinates of the multiple key points to be identified in the image to be identified are determined.

9. A training device for a feature extraction model, comprising: Acquisition unit, feature extraction unit, conversion unit, sampling unit, fitting unit and adjustment unit; The acquisition unit is used to acquire an image sample including a plurality of key points, wherein the key points have real position coordinates; The feature extraction unit is configured to perform feature extraction on the image sample using an initial feature extraction model to obtain features corresponding to the multiple key points; The conversion unit is configured to determine probability distributions corresponding to the multiple key points respectively based on the features corresponding to the multiple key points, wherein the probability distributions are used to describe the probabilities of the corresponding key points at various positions in the image sample; The sampling unit is used to sample from the multiple probability distributions to obtain multiple samples; The fitting unit is configured to perform fitting based on the multiple samples using an initial normalized flow model to obtain a simulated probability distribution; The adjustment unit is used to adjust the model parameters of the initial normalized flow model and the model parameters of the initial feature extraction model according to the difference between the simulated probability distribution and the true probability distribution to obtain a feature extraction model corresponding to the initial feature extraction model, and the true probability distribution is determined based on the true position coordinates of the key points corresponding to each of the multiple samples.

10. An application device for a feature extraction model, the device comprising: Acquisition unit, feature extraction unit, conversion unit and recognition unit; The acquisition unit is used to acquire the image to be identified; The feature extraction unit is configured to perform feature extraction on the image to be identified using the feature extraction model to obtain features corresponding to a plurality of key points to be identified included in the image to be identified, wherein the feature extraction model is trained using the method according to any one of claims 1 to 7; The conversion unit is configured to determine the probability distribution corresponding to each of the plurality of key points to be identified based on the features corresponding to each of the plurality of key points to be identified; The recognition unit is configured to determine the position coordinates of the multiple key points to be recognized in the image to be recognized according to the probability distributions respectively corresponding to the multiple key points to be recognized.

11. A computer device comprising a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1 to 8 according to the computer program.

12. A computer-readable storage medium for storing a computer program for executing the method according to any one of claims 1 to 8.

13. A computer program product comprising a computer program, which, when run on a computer device, causes the computer device to perform the method according to any one of claims 1 to 8.

Images