Model training, image recognition method and device

Abstract

The application discloses a model training method and device, and relates to the technical field of image processing. A specific embodiment of the method comprises the following steps: in response to obtaining a sample pair set, for each sample pair, inputting the sample pair into a student network and a teacher network in an initial teacher-student network; based on a prediction value of the student network for a first image, a prediction value of the student network for a second image, a prediction value of an adversarial regressor for the first image, and a label of the first image and a pseudo label of the second image, constructing a target loss function to train the initial teacher-student network to obtain a first teacher-student network; based on the second image in the sample pair set, training a feature extractor and the adversarial regressor of the student network in the first teacher-student network respectively to obtain a second teacher-student network; and determining the student network in the second teacher-student network as a recognition model. The embodiment effectively improves the accuracy and robustness of the recognition model obtained through training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to the technical field of image processing, and especially to a model training method and device and an image recognition method and device. BACKGROUND

[0002] The high-precision model designed at present is heavily dependent on a large amount of labeled data, and once sufficient training data is lacking, the accuracy and robustness of the model will be greatly limited. However, data labeling in actual scenarios is not an easy task, and not only is time-consuming and laborious, but also may have labeling errors. At present, there are many synthetic data sets, which have a large number of pictures that are easy to obtain and accurately labeled, but there is a huge difference in texture and background between the pictures of the real-world data set (target domain) and the synthetic data set (source domain), which makes the model trained directly using the synthetic data set have poor generalization in actual application. Therefore, the unsupervised domain adaptation problem, that is, how to use the source domain data that is accurately labeled but has a data distribution difference to guide the training of the model on the target domain, has important practical value.

[0003] At present, adversarial training is one of the common solutions in the domain adaptation task. Through continuous adversarial game in the feature space, the data distribution difference between the source domain and the target domain is made as small as possible. In this way, the model trained based on the source domain data can be applied to the target domain data. SUMMARY

[0004] The embodiments of the present application provide a model training method and device, equipment and a storage medium.

[0005] According to a first aspect, the embodiments of the present application provide a model training method, which comprises: in response to obtaining a sample pair set, for each sample pair, inputting the sample pair into a student network and a teacher network in an initial master-student network; based on a prediction value of the first image, a prediction value of the second image, a prediction value of the first image of an adversarial regressor, and a label of the first image and a pseudo label of the second image, constructing a target loss function to train the initial master-student network to obtain a first master-student network; based on the second image in the sample pair set, training a feature extractor and the adversarial regressor of the student network in the first master-student network respectively to obtain a second master-student network; and determining the student network in the second master-student network as a recognition model.

[0006] According to a second aspect, the embodiments of the present application provide an image recognition method, which comprises: obtaining a to-be-recognized image containing a target object; and inputting the to-be-recognized image into a recognition model to obtain key point information of the target object, wherein the recognition model is a recognition model obtained by the method described in any of the implementations of the first aspect.

[0007] According to a third aspect, embodiments of the present application provide a model training apparatus, the apparatus comprising: an obtaining data module configured to, in response to obtaining a sample pair set, for each sample pair, input the sample pair into a student network and a teacher network in an initial teacher-student network; a first training module configured to, based on a prediction value of the first image by a main regressor in the student network, a prediction value of the second image by the main regressor, a prediction value of the first image by an adversarial regressor, and a label of the first image and a pseudo label of the second image, construct a target loss function to train the initial teacher-student network to obtain a first teacher-student network; a second training module configured to, based on the second image in the sample pair set, train a feature extractor and the adversarial regressor in the student network in the first teacher-student network respectively to obtain a second teacher-student network; and a model determining module configured to determine the student network in the second teacher-student network as the recognition model.

[0008] According to a fourth aspect, embodiments of the present application provide an image recognition apparatus, the apparatus comprising: an obtaining image module configured to obtain a to-be-recognized image containing a target object; and a recognizing image module configured to input the to-be-recognized image into a recognition model to obtain key point information of the target object, wherein the recognition model is obtained by the method according to any implementation manner of the first aspect.

[0009] According to a fifth aspect, embodiments of the present application provide an electronic device, the electronic device comprising one or more processors; and a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any implementation manner of the first aspect or the second aspect.

[0010] According to a sixth aspect, embodiments of the present application provide a computer readable medium having a computer program stored thereon, when the program is executed by a processor, the program implements the method according to any implementation manner of the first aspect or the second aspect.

[0011] The present application inputs the sample pair into the student network and the teacher network in the initial master-student network for each sample pair in response to obtaining the sample pair set; constructs a target loss function based on the predicted value of the main regressor in the student network for the first image, the predicted value of the main regressor for the second image, the predicted value of the adversarial regressor for the first image, and the label of the first image and the pseudo label of the second image to train the initial master-student network to obtain a first master-student network; trains the feature extractor and the adversarial regressor of the student network in the first master-student network based on the second image in the sample pair set to obtain a second master-student network; and determines the student network in the second master-student network as the recognition model, i.e., through self-training and adversarial training of the master-student network with multiple branches based on the sample pair set to reduce the difference between the actual image and the synthetic image, improve the accuracy of the pseudo label, and further improve the accuracy and robustness of the recognition model obtained by training.

[0012] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0014] Figure 2 is a flowchart of one embodiment of the model training method according to the present application;

[0015] Figure 3 is a schematic diagram of one application scenario of the model training method according to the present application;

[0016] Figure 4 is a flowchart of another embodiment of the model training method according to the present application;

[0017] Figure 5 is a flowchart of another embodiment of the image recognition method according to the present application;

[0018] Figure 6 is a schematic diagram of one embodiment of the model training device according to the present application;

[0019] Figure 7 is a schematic diagram of one embodiment of the image recognition device according to the present application;

[0020] Figure 8 is a structural schematic diagram of a computer system of a server suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are cited as illustrative examples. Specific details of the present embodiments are set forth for the purpose of providing a thorough understanding of the present application. It will be apparent, however, to one skilled in the art that the present application can be practiced without many of the specific details set forth herein. In other instances, well-known features are not described in detail in order to avoid obscuring the present application. Accordingly, those who work in the art will recognize that many modifications and adaptations to the methods and

[0022] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] Figure 1 An exemplary system architecture 100 to which embodiments of the model training method of the present application can be applied is shown.

[0024] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0025] The terminal devices 101, 102, 103 interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as image recognition applications, communication applications, etc.

[0026] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to mobile phones and laptops. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules (such as services for providing model training), or as a single software or software module. No specific limitation is made herein.

[0027] The server 105 can be a server providing various services, for example, in response to obtaining a set of sample pairs, inputting, for each sample pair, the sample pair into a student network and a teacher network in an initial master-student network; constructing a target loss function based on a predicted value of the first image by a main regressor in the student network, a predicted value of the second image, a predicted value of the first image by an adversarial regressor, and a label of the first image and a pseudo label of the second image, to train the initial master-student network to obtain a first master-student network; training the feature extractor and the adversarial regressor of the student network in the first master-student network based on the second image in the set of sample pairs to obtain a second master-student network; and determining the student network in the second master-student network as the identification model.

[0028] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, a service for providing a trained model), or as a single software or software module. No specific limitation is made herein.

[0029] It should be noted that the model training method provided by the embodiments of the present disclosure can be executed by the server 105, or by the terminal devices 101, 102, and 103, or by the server 105 and the terminal devices 101, 102, and 103 in cooperation with each other. Accordingly, the device for model training can include various parts (for example, units, sub-units, modules, sub-modules) which can be all arranged in the server 105, or all arranged in the terminal devices 101, 102, and 103, or arranged in the server 105 and the terminal devices 101, 102, and 103 respectively.

[0030] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0031] Figure 2 A flowchart 200 of an embodiment of the model training method applicable to the present application is shown. In this embodiment, the model training method includes the following steps:

[0032] In step 201, in response to obtaining a set of sample pairs, inputting, for each sample pair, the sample pair into a student network and a teacher network in an initial master-student network.

[0033] In this embodiment, the execution subject (such as the server 105) can be a terminal device, a network, or a server. Figure 1The server 105 or the terminal device 101, 102, or 103 shown in the middle can obtain a sample pair set from a local or remote server storing sample pairs, and after obtaining the sample pair set, for each sample pair, input the sample pair into the student network and the teacher network in the initial teacher-student network.

[0034] The sample pair includes a first image and a second image containing the same target object, the first image is a synthetic image, and the second image is an actually captured image, that is, an image obtained by image capturing of the target object by an image capturing device, the first image is labeled with a label, and the label is used to indicate key point information of the target object, and the second image is not labeled with a label.

[0035] Here, the same target object included in the first image and the second image can be any target object, for example, a hand, a face, etc., and the present application does not limit this.

[0036] It should be noted that if the target object included in the first image and the second image is a hand, the hand gestures of the first image and the second image can be the same or different, and the present application does not limit this.

[0037] The teacher-student network belongs to a kind of transfer learning, and for the teacher-student network, the teacher network is often a more complex network with very good performance and generalization ability, which can be used as a soft target to guide another simpler student network to learn, so that the student network with simpler structure and less parameter operation can also have similar performance to the teacher network.

[0038] Here, the student network and the teacher network have the same structure, and the student network includes a feature extractor, a main regressor, and an adversarial regressor.

[0039] In step 202, based on the predicted value of the main regressor in the student network for the first image, the predicted value of the main regressor for the second image, the predicted value of the adversarial regressor for the first image, the label of the first image, and the pseudo label of the second image, a target loss function is constructed to train the initial teacher-student network to obtain a first teacher-student network, and the pseudo label of the second image is provided by the teacher network.

[0040] In this embodiment, the execution subject can construct a target loss function according to the output result of the main regressor in the student network for the first image, the output result of the main regressor for the second image, that is, the predicted value, the output result of the adversarial regressor for the first image, that is, the predicted value, the label of the first image, and the pseudo label of the second image, and minimize the target loss function to train the initial teacher-student network to obtain a first teacher-student network.

[0041] The pseudo label of the second image is provided by the pre-trained teacher network.

[0042] Here, the execution subject can construct a target loss function according to the first loss function, the second loss function and the third loss function to train the initial teacher-student network to obtain the first teacher-student network. The first loss function can be determined based on the predicted value of the main regressor for the first image and the label of the first image, the second loss function can be determined based on the predicted value of the main regressor for the second image and the pseudo label of the second image, and the third loss function can be determined based on the predicted value of the adversarial regressor for the first image and the label of the first image.

[0043] It should be pointed out that if the student network further includes an output regressor, the third loss function can also be determined based on the predicted value of the output regressor for the first image and the label of the first image, which is not limited in the present application.

[0044] Further, the target loss function can further include a regularization loss function, a global loss function and other loss functions that can further improve the performance of the model.

[0045] In addition, it should be pointed out that during the training process, the model parameters of the student network are updated by the normal SGD (Stochastic Gradient Descent) algorithm, and the teacher network does not participate in the gradient back propagation. The model parameters θ' of the teacher network are updated by the EMAN (Exponential Moving Average Normalization) of the model parameters θ of the student network, which is specifically shown as follows:

[0046] θ' = mθ' + (1-m)θ

[0047] μ' = mμ' + (1-m)μ

[0048] σ' = mσ' + (1-m)σ 2 2 2

[0049] wherein μ, σ 2 are the mean and variance of the BN respectively.

[0050] Step 203, based on the second image in the sample pair set, the feature extractor and the adversarial regressor in the student network of the first teacher-student network are trained respectively to obtain the second teacher-student network.

[0051] ​​In the embodiment, the execution subject can first train the adversarial regressor of the student network in the first teacher-student network based on the second images in the sample pair set, keep the rest parameters of the student network unchanged, obtain an initial second teacher-student network, and then train the feature extractor in the initial second teacher-student network based on the second images in the sample pair set, keep the rest parameters of the student network unchanged, and obtain the second teacher-student network; or first train the feature extractor of the student network in the first teacher-student network based on the second images in the sample pair set, keep the rest parameters of the student network unchanged, obtain an initial second teacher-student network, and then train the adversarial regressor in the initial second teacher-student network based on the second images in the sample pair set, keep the rest parameters of the student network unchanged, and obtain the second teacher-student network, which is not limited in the present application.

[0052] In step 204, the student network in the second teacher-student network is determined as the identification model.

[0053] In the embodiment, after obtaining the second teacher-student network, the execution subject can determine the student network in the second teacher-student network as the identification model.

[0054] The identification model can be used for identification of an image containing a target object.

[0055] Continuing to refer to Figure 3 , Figure 3 is one of the application scenarios of the model training method according to the embodiment.

[0056] In the application scenario of Figure 3 , the execution subject 301 responds to the acquisition of the sample pair set 302, inputs the sample pair into the student network and the teacher network in the initial teacher-student network 303 for each sample pair, wherein the sample pair includes a first image and a second image containing the same target object, for example, the first image and the second image are both gesture images, and the gestures are the same, the first image is a synthetic image, and the second image is an actually collected image, the first image is labeled with a label, and the label is used to indicate the key point information of the target object, and the second image is not labeled, the student network includes a feature extractor, a main regressor and an adversarial regressor; based on the predicted value of the main regressor in the student network for the first image, the predicted value for the second image, the predicted value of the adversarial regressor for the first image, and the label of the first image and the pseudo label of the second image, a target loss function is constructed to train the initial teacher-student network 303 to obtain the first teacher-student network 304; based on the second images in the sample pair set, the feature extractor and the adversarial regressor of the student network in the first teacher-student network are trained respectively to obtain the second teacher-student network 305; and the student network in the second teacher-student network is determined as the identification model 306.

[0057] The model training method of the present disclosure comprises the following steps: in response to obtaining a sample pair set, inputting, for each sample pair, the sample pair into a student network and a teacher network in an initial teacher-student network; constructing a target loss function based on a prediction value of a main regressor in the student network for the first image, a prediction value of the main regressor for the second image, a prediction value of an adversarial regressor for the first image, a label of the first image, and a pseudo label of the second image, to train the initial teacher-student network, to obtain a first teacher-student network; training, based on the second image in the sample pair set, a feature extractor and the adversarial regressor in the student network in the first teacher-student network, to obtain a second teacher-student network; and determining the student network in the second teacher-student network as the recognition model, thereby effectively improving the accuracy and robustness of the recognition model obtained by training.

[0058] Further reference is made to Figure 4 which shows Figure 2 The flow 400 of still another embodiment of the model training method is shown. In this embodiment, the flow 400 can comprise the following steps:

[0059] In response to obtaining a sample pair set, inputting, for each sample pair, the sample pair into a student network and a teacher network in an initial teacher-student network.

[0060] In this embodiment, the implementation details and technical effects of step 401 can be referred to the description of step 201, which will not be repeated here.

[0061] Constructing a target loss function based on a prediction value of a main regressor in the student network for the first image, a prediction value of the main regressor for the second image, a prediction value of an adversarial regressor for the first image, an output prediction value of an output regressor for the first image, and a label of the first image and a pseudo label of the second image, to train the initial teacher-student network, to obtain a first teacher-student network.

[0062] In this embodiment, the student network further comprises an output regressor. The execution subject can construct a target loss function based on a prediction value of a main regressor in the student network for the first image, a prediction value of the main regressor for the second image, a prediction value of an adversarial regressor for the first image, an output prediction value of an output regressor for the first image, and a label of the first image and a pseudo label of the second image, and minimize the target loss function to train the initial teacher-student network, to obtain a first teacher-student network.

[0063] Here, the execution subject can construct a target loss function according to the first loss function, the second loss function, the third loss function and the fourth loss function to train the initial teacher-student network to obtain the first teacher-student network. The first loss function can be determined based on the predicted value of the main regressor for the first image and the label of the first image, the second loss function can be determined based on the predicted value of the main regressor for the second image and the pseudo label of the second image, the third loss function can be determined based on the predicted value of the output regressor for the first image and the predicted value of the adversarial regressor for the first image, and the fourth loss function can be determined based on the predicted value of the output regressor for the first image and the label of the first image.

[0064] In some optional manners, based on the predicted value of the main regressor in the student network for the first image, the predicted value for the second image, the predicted value of the adversarial regressor for the first image, the predicted value of the output regressor for the first image, and the label of the first image and the pseudo label of the second image, the target loss function is constructed to train the initial teacher-student network to obtain the first teacher-student network, including: constructing the target loss function according to the first loss function, the second loss function, the third loss function and the fourth loss function to train the initial teacher-student network to obtain the first teacher-student network.

[0065] In the implementation manner, the execution subject can construct a target loss function according to the first loss function, the second loss function, the third loss function and the fourth loss function to train the initial teacher-student network to obtain the first teacher-student network. The first loss function can be determined based on the predicted value of the main regressor for the first image and the label of the first image, the second loss function can be determined based on the predicted value of the main regressor for the second image and the pseudo label of the second image, the third loss function can be determined based on the predicted value of the output regressor for the first image and the predicted value of the adversarial regressor for the first image, and the fourth loss function can be determined based on the predicted value of the output regressor for the first image and the label of the first image.

[0066] Specifically, for the sample pair set, a set of the first images in the sample pair set can be taken as source domain samples The label is A set of the second images in the sample pair set can be taken as target domain samples For each sample pair, it is input into the student network in the initial teacher-student network, and the corresponding feature map (F s ,F t ) and heat map (H s ,H t ) can be obtained, which are specifically as follows:

[0067] F s = ψ(x s ), H s = f(Fs )

[0068] F t = ψ(x t ), H t = f(F t )

[0069] where ψ is a feature extractor and f is a regressor.

[0070] The target loss function can be represented by the following formula:

[0071]

[0072] where L T (f0(ψ(x s )), y s ) represents a first loss function, represents a second loss function, L T (f1(ψ(x s )), f2(ψ(x s ))) represents a third loss function, L T (f1(ψ(x s )), y s ) represents a fourth loss function; λ1, λ2 represent the weights of the loss; f0 indicates the main regressor, f1 indicates the output regressor, and f2 indicates the adversarial regressor.

[0073] Here, in order to subsequent adversarial training, the KL divergence can be used to calculate the heat map loss. First, define the spatial probability distribution P T (H k ), k∈{1,2,…,K}, which normalizes the heat map H k ∈R H×W for each key point k in the spatial dimension:

[0074]

[0075] Use σ to represent the spatial softmax function:

[0076]

[0077] Then use the KL divergence to calculate the loss value:

[0078]

[0079] where H s = f(ψ(x s ))∈R K×W×H , is the label y sThe heat map of each key point k. Since using the KL (Kullback-Leibler divergence) divergence does not cause numerical explosion, the KL is used instead of the MSE (Mean Squared Error) in subsequent calculations by default.

[0080] The loss between the predicted value and the label in the source domain is:

[0081] L s = L T (H s ,y s )

[0082] Since the labels of the target domain are not available, the pseudo labels of the target domain are used to calculate the loss of self-training:

[0083]

[0084] The implementation trains the initial teacher-student network by constructing a target loss function based on a first loss function, a second loss function, a third loss function, and a fourth loss function, obtains a first teacher-student network, and then determines an identification model based on the first teacher-student network, thereby further improving the robustness of the determined identification model.

[0085] In some optional manners, the feature extractor and the adversarial regressor in the student network in the first teacher-student network are trained based on the second image in the sample pair set, respectively, to obtain a second teacher-student network, including: keeping the parameters of the feature extractor, the main regressor, and the output regressor in the student network unchanged, training the first teacher-student network based on the second image in the sample pair set by minimizing a fifth loss function, to obtain an initial second teacher-student network; keeping the parameters of each regressor in the student network unchanged, training the initial second teacher-student network based on the second image in the sample pair set by minimizing a sixth loss function, to obtain the second teacher-student network.

[0086] In the implementation, the execution subject can first keep the parameters of the feature extractor, the main regressor, and the output regressor in the student network unchanged, train the first teacher-student network based on the second image in the sample pair set by minimizing a fifth loss function, to obtain an initial second teacher-student network. The fifth loss function is determined based on the predicted value of the first teacher-student network for the second image and the pseudo label.

[0087] Specifically, it can be represented by the following formula:

[0088]

[0089] ​Further, the execution subject keeps parameters of each regressor unchanged, and trains the initial second teacher-student network based on the second image in the sample pair set and the sixth loss function, to obtain the second teacher-student network, wherein the sixth loss function is determined based on a predicted value of the output regressor for the second image and a predicted value of the adversarial regressor for the second image.

[0090] Specifically, it can be represented by the following formula:

[0091]

[0092] Wherein, the output regressor and the adversarial regressor are used to realize the adversarial training, f1 is the output regressor, and f2 is the adversarial regressor. When the teacher-student network makes a prediction error on the target domain, the position of the prediction error is not uniformly distributed in the pixel space. For example, when the model predicts the hand posture, the key points are more likely to be located in the hand region, and the probability of appearing in the background is close to zero. Therefore, the error probability distribution is used to make the adversarial regressor pay more attention to those positions with high probability. Simply speaking, the sparsity in the output space probability is used to guide the optimization of the adversarial regressor, so that the regressor pays more attention to the positions with high probability.

[0093] In order to more accurately and completely express the error probability distribution of each key point, a new heat map is composed of multiple heat maps, and the error probability distribution is calculated by the pseudo label of the target domain to be regarded as ground falseprediction. First, define H F as:

[0094]

[0095] Wherein, is the output of the regressor for each key point k in the heat map.

[0096] Then we can obtain:

[0097]

[0098] In order to alleviate the optimization difficulty of the adversarial training, the maximization and minimization of the same target in the adversarial training are converted into the minimization of two opposite targets. The two opposite targets are designed for the feature extractor and the adversarial regressor respectively. The target of the adversarial regressor is to minimize the loss between the predicted value of the adversarial regressor f2 and the ground false prediction. The target of the feature extractor is to minimize the loss between f1 and f2. The calculation formula is:

[0099]

[0100]

[0101] The implementation trains the adversarial regressor in the student network by minimizing the fifth loss function based on the second images in the sample pair set to obtain an initial second teacher-student network; trains the feature extractor in the second teacher-student network based on the second images in the sample pair set to minimize the sixth loss function to obtain the second teacher-student network, and determines the recognition model based on the second teacher-student network, thereby further improving the accuracy of the pseudo label and further improving the robustness of the model.

[0102] In some optional manners, the fifth loss function is determined by: obtaining an initial value based on the sum of the pseudo label of the second image by the teacher network in the first teacher-student network and the predicted value of the second image by the output regressor in the student network; and determining the fifth loss function based on the difference between the initial value and the predicted value of the second image by the adversarial regressor in the student network.

[0103] In the implementation, the execution subject can obtain an initial value based on the sum of the pseudo label of the second image by the teacher network in the first teacher-student network and the predicted value of the second image by the output regressor in the student network in the first teacher-student network; and determine the fifth loss function based on the difference between the initial value and the predicted value of the second image by the adversarial regressor in the student network.

[0104] The implementation obtains an initial value based on the sum of the pseudo label of the second image by the teacher network in the first teacher-student network and the predicted value of the second image by the output regressor in the student network; and determines the fifth loss function based on the difference between the initial value and the predicted value of the second image by the adversarial regressor in the student network, which helps to improve the accuracy of the determined fifth loss function.

[0105] In some optional manners, the first teacher-student network is obtained by training the initial teacher-student network based on the target loss function constructed based on the first loss function, the second loss function, the third loss function and the fourth loss function, including: the first teacher-student network is obtained by training the initial teacher-student network based on the target loss function constructed based on the first loss function, the second loss function, the third loss function, the fourth loss function and the global loss function.

[0106] In the implementation, the execution subject can train the initial teacher-student network based on the target loss function constructed based on the first loss function, the second loss function, the third loss function, the fourth loss function and the global loss function to obtain the first teacher-student network, wherein the global loss function is determined based on the feature distribution of the first image and the second image in the sample pair set.

[0107] Specifically, the target loss function can be represented by the following formula:

[0108]

[0109] wherein λ1, λ2, λ3represent the weights of the losses.

[0110] The implementation trains the initial teacher-student network based on the first loss function, the second loss function, the third loss function, the fourth loss function, and the global loss function to obtain the first teacher-student network, and then determines the recognition model based on the first teacher-student network, which uses the global loss function to compensate for the gap between the target domain and the source domain to some extent and reduces the influence of noise on the pseudo label, thereby further improving the robustness of the determined recognition model.

[0111] In some optional manners, the first teacher-student network is obtained by training the initial teacher-student network based on the first loss function, the second loss function, the third loss function, and the fourth loss function, including: the first teacher-student network is obtained by training the initial teacher-student network based on the first loss function, the second loss function, the third loss function, the fourth loss function, the global loss function, and the regularization loss function.

[0112] In the implementation, the execution subject can train the initial teacher-student network based on the first loss function, the second loss function, the third loss function, the fourth loss function, the global loss function, and the regularization loss function to obtain the first teacher-student network.

[0113] Specifically, the target loss function can be represented by the following formula:

[0114]

[0115]

[0116] wherein λ1, λ2, λ3, λ4represent the weights of the losses, H(x t ; w) is the output of the main regressor.

[0117] The implementation trains the initial teacher-student network based on the first loss function, the second loss function, the third loss function, the fourth loss function, the global loss function, and the regularization loss function to obtain the first teacher-student network, and then determines the recognition model based on the first teacher-student network, which uses the regularization loss function to further reduce the influence of noise on the pseudo label, speeds up the convergence, further improves the robustness of the model, and improves the model training efficiency.

[0118] In step 403, the feature extractor and the adversarial regressor of the student network in the first teacher-student network are trained based on the second image in the sample pair set to obtain a second teacher-student network.

[0119] In the embodiment, details and technical effects of implementing step 403 can refer to the description of step 203, and will not be repeated here.

[0120] Step 404, determining the student network in the second teacher-student network as the identification model.

[0121] In the embodiment, details and technical effects of implementing step 404 can refer to the description of step 204, and will not be repeated here.

[0122] The above embodiments of the present application, compared with the corresponding embodiments, Figure 2 Compared with the corresponding embodiments, the model training method in the embodiment presents the process 400, which is based on the predicted value of the main regressor in the student network for the first image, the predicted value of the main regressor for the second image, the predicted value of the adversarial regressor for the first image, the predicted value of the output regressor for the first image, and the label of the first image and the pseudo label of the second image, to construct a target loss function to train the initial teacher-student network to obtain the first teacher-student network, and then determine the identification model based on the first teacher-student network, which helps to further improve the accuracy and robustness of the obtained identification model.

[0123] Continuing to refer to Figure 5 , the flow 500 of one embodiment of the image recognition method according to the present application is shown. The image recognition method comprises the following steps:

[0124] Step 501, obtaining an to-be-identified image containing a target object.

[0125] In the embodiment, the execution subject can obtain the to-be-identified object containing the target object through wired or wireless connection.

[0126] Among them, the wireless connection mode can include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection modes.

[0127] Step 502, inputting the to-be-identified image into the identification model to obtain key point information of the target object.

[0128] In the embodiment, after obtaining the to-be-identified image, the execution subject can input the to-be-identified image into the identification model to obtain the key point information of the target object corresponding to the to-be-identified image. Among them, the identification model is the identification model obtained by the method described in the corresponding embodiment, which will not be repeated here. Figure 2

[0129] ​The image recognition method provided in the embodiments of the present disclosure comprises the following steps: obtaining an image to be recognized; inputting the image to be recognized into a recognition model to obtain key point information of a target object corresponding to the image to be recognized, wherein the recognition model is a teacher-student network as shown in Figure 2 The recognition model obtained by the method described in the embodiments can help improve the accuracy of recognizing the image to be recognized.

[0130] Further referring to Figure 6 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a model training device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied in various electronic devices.

[0131] As shown in Figure 6 , the model training device 600 of the present embodiment comprises an obtaining data module 601, a first training module 602, a second training module 603 and a model determining module 604.

[0132] The obtaining data module 601 can be configured to, in response to obtaining a sample pair set, input, for each sample pair, the sample pair into a student network and a teacher network in an initial teacher-student network.

[0133] The first training module 602 can be configured to construct a target loss function based on a predicted value of the main regressor in the student network for the first image, a predicted value of the main regressor for the second image, a predicted value of the adversarial regressor for the first image, a label of the first image and a pseudo label of the second image, to train the initial teacher-student network to obtain a first teacher-student network.

[0134] The second training module 603 can be configured to train, based on the second image in the sample pair set, a feature extractor and the adversarial regressor of the student network in the first teacher-student network respectively to obtain a second teacher-student network.

[0135] The model determining module 604 can be configured to determine the student network in the second teacher-student network as the recognition model.

[0136] In some optional manners of the present embodiment, the first training module is further configured to construct a target loss function based on a predicted value of the main regressor in the student network for the first image, a predicted value of the main regressor for the second image, a predicted value of the adversarial regressor for the first image, a predicted value of the output regressor for the first image, and a label of the first image and a pseudo label of the second image, to train the initial teacher-student network to obtain the first teacher-student network.

[0137] In some optional manners of the present embodiment, the first training module is further configured to construct a target loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function to train the initial teacher-student network to obtain the first teacher-student network.

[0138] In some optional manners of the present embodiment, the first training module is further configured to construct a target loss function based on the first loss function, the second loss function, the third loss function, the fourth loss function and the global loss function to train the initial teacher-student network to obtain the first teacher-student network.

[0139] In some optional manners of the present embodiment, the first training module is further configured to construct a target loss function based on the first loss function, the second loss function, the third loss function, the fourth loss function, the global loss function and the regularization loss function to train the initial teacher-student network to obtain the first teacher-student network.

[0140] In some optional manners of the present embodiment, the second training module is further configured to keep the parameters of the feature extractor, the main regressor and the output regressor in the student network unchanged, train the first teacher-student network based on the second image in the sample pair set and the fifth loss function to obtain an initial second teacher-student network, and keep the parameters of each regressor in the student network unchanged, train the initial second teacher-student network based on the second image in the sample pair set and the sixth loss function to obtain the second teacher-student network.

[0141] In some optional manners of the present embodiment, the fifth loss function is determined by: obtaining an initial value based on the sum of the pseudo-label of the second image by the teacher network in the first teacher-student network and the predicted value of the second image by the output regressor in the student network; and determining the fifth loss function based on the difference between the initial value and the predicted value of the second image by the adversarial regressor in the student network.

[0142] Further referring to Figure 7 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of an image recognition device, which corresponds to the method embodiment shown in Figure 5 , and the device can be applied in various electronic devices.

[0143] As shown in Figure 7 , the image recognition device 700 of the present embodiment includes an image acquisition module 701 and an image recognition module 702.

[0144] The image acquisition module 701 can be configured to acquire an image to be recognized containing a target object.

[0145] The image recognition module 702 can be configured to input the image to be recognized into a recognition model to obtain key point information of a target object.

[0146] According to embodiments of the present application, the present application also provides an electronic device and a readable storage medium.

[0147] As Figure 8 shown is a block diagram of an electronic device for a model training method according to embodiments of the present application.

[0148] 800 is a block diagram of an electronic device for a model training method according to embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0149] As Figure 8 shown, the electronic device includes one or more processors 801, memory 802, and interfaces 803 for connecting the components to each other and to other devices. The components are interconnected by a system bus or beams, and can be mounted on a common main board or in other manners as appropriate. The processor can process instructions for execution within the electronic device, including instructions stored in the memory or on storage devices to display graphical information of a GUI on an external input / output device, such as a display device coupled to the interface. In other implementations, multiple processors and / or multiple buses can be used as appropriate, along with multiple memories and types of memory. Also, multiple electronic devices can be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Figure 8 The processor 801 is taken as an example in the figure.

[0150] The memory 802 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by at least one processor, so that the at least one processor executes the model training method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the model training method provided by the present application.

[0151] The memory 802 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the model training method in the embodiment of the present application (for example, the attached Figure 6 The processor 801 executes the non-transient software programs, instructions, and modules stored in the memory 802 to execute various functional applications and data processing of the server, that is, to implement the model training method in the above method embodiment.

[0152] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device for model training, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely located relative to the processor 801, and these remote memories may be connected to the electronic device for model training via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The electronic device of the model training method may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 may be connected via a bus or other means. Figure 8 The bus connection is taken as an example.

[0154] The input device 803 can receive input digital or character information, such as an input device such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, a pointer, one or more mouse buttons, a trackball, a joystick, etc. The output device 804 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0155] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0156] These computer programs (also known as programs, software, software applications or code) include machine instructions for the programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0157] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0158] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0159] The computer system can include clients and servers. The clients and the servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0160] According to the technical scheme of the embodiment of the present application, the accuracy and robustness of the recognition model obtained by training are effectively improved.

[0161] It should be understood that various forms of flow shown above can be used, with reordering, adding or removing steps. For example, the steps recited in the specification can be performed in parallel, sequentially, or in different order, unless otherwise specified, and are not limited to the order in which they are recited herein.

[0162] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A model training method, the method comprising: in response to obtaining a sample pair set, inputting, for each sample pair, the sample pair into a student network and a teacher network in an initial teacher-student network, wherein the sample pair comprises a first image and a second image containing a same target object, the first image is a synthetic image, the second image is an actually captured image, the first image is labeled with a label indicating key point information of the target object, and the second image is not labeled, the student network comprises a feature extractor, a main regressor, and an adversarial regressor; based on a predicted value of the main regressor in the student network for the first image, a predicted value of the main regressor for the second image, a predicted value of the adversarial regressor for the first image, and the label of the first image and a pseudo label of the second image provided by the teacher network, constructing a target loss function to train the initial teacher-student network to obtain a first teacher-student network; based on the second image in the sample pair set, training the feature extractor and the adversarial regressor in the student network in the first teacher-student network respectively to obtain a second teacher-student network; determining the student network in the second teacher-student network as a recognition model.

2. The method of claim 1, wherein, The student network further comprises an output regressor, and the constructing the target loss function to train the initial teacher-student network to obtain the first teacher-student network based on the predicted value of the main regressor in the student network for the first image, the predicted value of the main regressor for the second image, the predicted value of the adversarial regressor for the first image, and the label of the first image and the pseudo label of the second image comprises: constructing the target loss function based on the predicted value of the main regressor in the student network for the first image, the predicted value of the main regressor for the second image, the predicted value of the adversarial regressor for the first image, the predicted value of the output regressor for the first image, and the label of the first image and the pseudo label of the second image to train the initial teacher-student network to obtain the first teacher-student network.

3. The method of claim 2, wherein, The constructing the target loss function to train the initial teacher-student network to obtain the first teacher-student network based on the predicted value of the main regressor in the student network for the first image, the predicted value of the main regressor for the second image, the predicted value of the adversarial regressor for the first image, the predicted value of the output regressor for the first image, and the label of the first image and the pseudo label of the second image comprises: constructing the target loss function based on a first loss function, a second loss function, a third loss function, and a fourth loss function to train the initial teacher-student network to obtain the first teacher-student network, wherein the first loss function is determined based on the predicted value of the main regressor for the first image and the label of the first image, the second loss function is determined based on the predicted value of the main regressor for the second image and the pseudo label of the second image, the third loss function is determined based on the predicted value of the output regressor for the first image and the predicted value of the adversarial regressor for the first image, and the fourth loss function is determined based on the predicted value of the output regressor for the first image and the label of the first image.

4. The method of claim 3, wherein, The target loss function is constructed based on the first loss function, the second loss function, the third loss function and the fourth loss function to train the initial teacher-student network to obtain the first teacher-student network, including: The target loss function is constructed based on the first loss function, the second loss function, the third loss function, the fourth loss function and the global loss function to train the initial teacher-student network to obtain the first teacher-student network, wherein the global loss function is determined based on the feature distribution of the first image and the second image in the sample pair set.

5. The method of claim 4, wherein, The target loss function is constructed based on the first loss function, the second loss function, the third loss function, the fourth loss function and the global loss function to train the initial teacher-student network to obtain the first teacher-student network, including: The target loss function is constructed based on the first loss function, the second loss function, the third loss function, the fourth loss function, the global loss function and the regularization loss function to train the initial teacher-student network to obtain the first teacher-student network.

6. The method of claim 2, wherein, The feature extractor and the adversarial regressor in the student network of the first teacher-student network are trained based on the second image in the sample pair set to obtain the second teacher-student network, including: The parameters of the feature extractor, the main regressor and the output regressor in the student network are kept unchanged, and the first teacher-student network is trained based on the second image in the sample pair set to minimize the fifth loss function to obtain the initial second teacher-student network, wherein the fifth loss function is determined based on the predicted value of the first teacher-student network for the second image and the pseudo label; The parameters of each regressor in the student network are kept unchanged, and the initial second teacher-student network is trained based on the second image in the sample pair set to minimize the sixth loss function to obtain the second teacher-student network, wherein the sixth loss function is determined based on the predicted value of the output regressor for the second image and the predicted value of the adversarial regressor for the second image.

7. The method of claim 6, wherein, The fifth loss function is determined by: An initial value is obtained based on the sum of the pseudo label of the teacher network in the first teacher-student network for the second image and the predicted value of the output regressor in the student network for the second image; The fifth loss function is determined based on the difference between the initial value and the predicted value of the adversarial regressor in the student network for the second image.

8. A recognition method, the method comprising: obtaining a to-be-recognized image containing a target object; inputting the to-be-recognized image into a recognition model to obtain key point information of the target object, wherein the recognition model is obtained by the method of any one of claims 1-7.

9. A model training apparatus, comprising: an acquisition data module configured to, in response to acquiring a sample pair set, for each sample pair, input the sample pair into a student network and a teacher network in an initial teacher-student network, wherein the sample pair includes a first image and a second image containing the same target object, the first image is a synthetic image, the second image is an actually acquired image, the first image is labeled with a label, the label is used to indicate key point information of the target object, the second image is not labeled with a label, the student network includes a feature extractor, a main regressor and an adversarial regressor; The first training module is configured to construct a target loss function based on the predicted value of the main regressor in the student network for the first image, the predicted value of the main regressor for the second image, the predicted value of the adversarial regressor for the first image, the label of the first image and the pseudo label of the second image, to train the initial teacher-student network to obtain the first teacher-student network, and the pseudo label of the second image is provided by the teacher network. The second training module is configured to train the feature extractor and the adversarial regressor of the student network in the first teacher-student network based on the second image in the sample pair set, to obtain the second teacher-student network. The model determination module is configured to determine the student network in the second teacher-student network as the recognition model.

10. An image recognition device, comprising: an image acquisition module configured to acquire a to-be-recognized image containing a target object; an image recognition module configured to input the to-be-recognized image into a recognition model to obtain key point information of the target object, wherein the recognition model is obtained by the method in any one of claims 1-7.

11. An electronic device, comprising: comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores computer instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

12. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-8.

Images