Image recognition and model training method and device

Abstract

The invention discloses an image recognition and model training method and device, and relates to the field of computers. The image recognition method comprises the steps of obtaining a to-be-processed image, adjusting pixel values of pixel points in the to-be-processed image to obtain a pre-processed image, and inputting the pre-processed image into a recognition model to obtain a category to which the to-be-processed image belongs. The similarity between the features of the preprocessed image and the features of the source domain of the recognition model reaches a first threshold value, and then the preprocessed image is close to the source domain of the recognition model. And the identification model is used for the category to which the image belongs.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to image recognition and model training methods and devices. Background Art

[0002] Image recognition refers to identifying the features or categories of the content shown in an image. For image recognition, neural networks are often used for recognition. However, when using neural networks for recognition, it depends on the recognition ability of the neural network, that is, the training data used to train the neural network. If the scene of the training data is incomplete or unevenly distributed, the resulting neural network will only be able to recognize images in a certain scene. Therefore, when the data input to the neural network is different from the scene to which the aforementioned training data belongs, the neural network's recognition effect on the input data will be poor. Therefore, how to improve the model's recognition effect on images has become a problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides an image recognition and model training method and device to solve the problem that when the data to be processed belonging to multiple scenarios is input into the model, the model has poor recognition effect on the data to be processed due to the single training data scenario used by the training model.

[0004] In a first aspect, the present application provides an image recognition method. The image recognition method can be applied to an image recognition system or to a computing device that supports the image recognition system to implement the image recognition method. The computing device is a server or a terminal. The image recognition method includes: obtaining an image to be processed, adjusting the pixel values ​​of pixels in the image to be processed, obtaining a preprocessed image, and then inputting the preprocessed image into a recognition model to obtain the category to which the image to be processed belongs. The similarity between the features of the preprocessed image and the features of the source domain of the recognition model reaches a first threshold, and the recognition model is used to identify the category to which the image belongs.

[0005] In the present application, by adjusting the pixels of the image to be processed, the similarity between the features of the obtained pre-processed image and the features of the source domain of the recognition model reaches a first threshold, and then the pre-processed image is close to the source domain of the recognition model, thereby improving the recognition accuracy of the recognition model for the image to be processed and the robustness of the recognition model for image recognition. In addition, when the recognition model has a high recognition rate for images belonging to the source domain, since the pre-processed image is close to the source domain, the recognition model has a high recognition rate for the pre-processed image, thereby further improving the accuracy of the recognition model in obtaining the category to which the image to be processed in different fields belongs.

[0006] In a possible implementation, the above-mentioned adjusting the pixel values ​​of pixels in the image to be processed to obtain the preprocessed image includes: inputting the image to be processed into a preprocessing model to obtain the preprocessed image. The preprocessing model is used to adjust the pixel values ​​of pixels in the image, and the preprocessing model is trained with training data in the field to which the image to be processed belongs.

[0007] In the present application, since the preprocessing model is trained with training data of the field to which the image to be processed belongs, the preprocessing model learns the ability to adjust images of the field to which the image to be processed belongs. The preprocessing model is used to adjust the pixels of the model to be processed, and the features of the obtained preprocessed image have a similarity with the features of the above-mentioned source domain that reaches a first threshold, thereby improving the accuracy of the recognition model in obtaining the category to which the image to be processed belongs.

[0008] In one possible scenario, during the training process, the preprocessing model can be trained separately using the training data, or the preprocessing model can be trained in combination with the recognition model. When the preprocessing model is trained in combination with the recognition model, the recognition model will be frozen, that is, the parameters of the recognition model will not change during the training process.

[0009] Exemplarily, in the above training process, if the loss function of the preprocessing model converges, the training data is used up, or the number of iterations during training reaches a set threshold, the training of the preprocessing model is terminated.

[0010] In a possible implementation, the above-mentioned inputting the image to be processed into the preprocessing model to obtain the preprocessed image includes: inputting the image to be processed into the preprocessing model, and the preprocessing model identifies the field to which the image to be processed belongs, and then adopts an adjustment strategy that matches the field to which the image to be processed belongs to adjust the pixels in the image to be processed accordingly, thereby obtaining the preprocessed image.

[0011] In the present application, the preprocessing model adopts corresponding adjustment strategies for images in different fields to adjust the pixels of the image, thereby achieving adaptive adjustment according to the field to which the image to be processed belongs, so as to ensure that the similarity between the features of the adjusted preprocessed image and the features of the above-mentioned source domain reaches a first threshold, thereby improving the accuracy of the recognition model in determining the category to which the image to be processed belongs.

[0012] In a possible implementation, the preprocessing model includes a first sub-model and a second sub-model; the first sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain is lower than a second threshold, and the multiple pixels in the image to be processed correspond to a feature of the image to be processed; the second sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain reaches the second threshold.

[0013] In the present application, the first sub-model and the second sub-model are used to perform differential adjustments on the pixel points corresponding to different features to ensure that the feature similarity of the preprocessed image obtained by adjustment by the first sub-model and the second sub-model and the feature of the above-mentioned source domain reaches a first threshold, thereby improving the accuracy of the recognition model in determining the category to which the image to be processed belongs.

[0014] In one possible implementation, the loss function corresponding to the first sub-model is a cross-entropy loss function; the loss function corresponding to the second sub-model is a cross-entropy loss function and a steady-state regularization term, wherein the steady-state regularization term is used to indicate: if the domain to which the training data belongs changes during the training process, the parameters in the preprocessing model whose change range exceeds the third threshold are adjusted.

[0015] In the present application, different loss functions are set for the first sub-model and the second sub-model so that the first sub-model and the second sub-model can respectively learn the differences in the features of the image to be processed, the differences in the features of the source domain whose similarity is lower than the second threshold, and the differences in the features of the source domain whose similarity reaches the second threshold, thereby enabling the first sub-model and the second sub-model to perform differentiated adjustments on the pixel points corresponding to different features.

[0016] In the second aspect, the present application provides a model training method for image recognition. The model training method for image recognition can be applied to an image recognition system or to a computing device that supports the image recognition system to implement the model training method for image recognition. The computing device is, for example, a server or a terminal. The model training method for image recognition includes: obtaining training data from a variety of fields, and using the training data from a variety of fields to train a preprocessing model. The preprocessing model is used to adjust the pixel values ​​of pixels in the image, and the image to be processed is input into the trained preprocessing model to obtain a feature similarity between the features of the preprocessed image and the features of the source domain of the recognition model, and the recognition model is used to identify the category to which the image belongs.

[0017] In the present application, only the preprocessing model is trained using training data from multiple fields, while the recognition model is not trained, thereby avoiding the problem of degradation and catastrophic forgetting of the recognition model caused by training the recognition model during multi-domain adaptation, that is, the recognition accuracy of the recognition model for images belonging to the source domain will not change. The similarity between the features of the preprocessed image obtained by adjusting the trained preprocessing model and the features of the source domain reaches a first threshold, that is, the preprocessed image is close to the source domain of the recognition model, thereby improving the recognition accuracy of the recognition model for the preprocessed image.

[0018] Exemplarily, the preprocessing model can be trained separately, or the preprocessing model can be trained in combination with the recognition model. When the preprocessing model is trained in combination with the recognition model, the recognition model will be frozen, that is, the parameters of the recognition model will not change during the training process.

[0019] In a possible implementation, the preprocessing model includes a first sub-model and a second sub-model; the first sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain is lower than a second threshold, and the multiple pixels in the image to be processed correspond to a feature of the image to be processed; the second sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain reaches the second threshold.

[0020] In the present application, the first sub-model and the second sub-model are used to perform differential adjustments on the pixel points corresponding to different features to ensure that the feature similarity of the preprocessed image obtained by adjustment by the first sub-model and the second sub-model and the feature of the above-mentioned source domain reaches a first threshold, thereby improving the accuracy of the recognition model in determining the category to which the image to be processed belongs.

[0021] In one possible implementation, the loss function corresponding to the first sub-model is a cross-entropy loss function; the loss function corresponding to the second sub-model is a cross-entropy loss function and a steady-state regularization term, wherein the steady-state regularization term is used to indicate: if the domain to which the training data belongs changes during the training process, the parameters in the preprocessing model whose change range exceeds the third threshold are adjusted.

[0022] In the present application, different loss functions are set for the first sub-model and the second sub-model so that the first sub-model and the second sub-model can respectively learn the differences in the features of the image to be processed, the differences in the features of the source domain whose similarity is lower than the second threshold, and the differences in the features of the source domain whose similarity reaches the second threshold, thereby enabling the first sub-model and the second sub-model to perform differentiated adjustments on the pixel points corresponding to different features.

[0023] In a third aspect, the present application provides an image recognition device, which is applied to a computing device or a computing device that supports the recognition system to implement the image recognition method, and the image recognition device includes various modules for executing the positioning method in the first aspect or any optional implementation of the first aspect. For example, the image recognition device includes: an acquisition module, an adjustment module, and a recognition module. Among them,

[0024] The acquisition module is used to acquire the image to be processed.

[0025] The adjustment module is used to adjust the pixel values ​​of the pixels in the image to be processed to obtain a preprocessed image; the similarity between the features of the preprocessed image and the features of the source domain of the recognition model reaches a first threshold; the recognition model is used to identify the category to which the image belongs.

[0026] The recognition module is used to input the pre-processed image into the recognition model to obtain the category to which the image to be processed belongs.

[0027] In a possible implementation, the adjustment module is specifically used to input the image to be processed into a preprocessing model to obtain a preprocessed image; wherein the preprocessing model is used to adjust the pixel values ​​of pixels in the image, and the preprocessing model is trained with training data in the field to which the image to be processed belongs.

[0028] In a possible implementation, the above-mentioned adjustment module is also specifically used to input the image to be processed into a preprocessing model; and the preprocessing model identifies the field to which the image to be processed belongs, and then adopts an adjustment strategy that matches the field to which the image to be processed belongs, and adjusts the pixels in the image to be processed accordingly, thereby obtaining a preprocessed image.

[0029] In a possible implementation, the preprocessing model includes a first sub-model and a second sub-model; the first sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain is lower than a second threshold, and the multiple pixels in the image to be processed correspond to a feature of the image to be processed; the second sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain reaches the second threshold.

[0030] In one possible implementation, the loss function corresponding to the first sub-model is a cross-entropy loss function; the loss function corresponding to the second sub-model is a cross-entropy loss function and a steady-state regularization term, wherein the steady-state regularization term is used to indicate: if the domain to which the training data belongs changes during the training process, the parameters in the preprocessing model whose change range exceeds the third threshold are adjusted.

[0031] In a fourth aspect, the present application provides a model training device for image recognition. The model training device for image recognition is applied to a computing device or a computing device that supports the recognition system to implement a model training method for image recognition. The model training device for image recognition includes various modules for executing the model training method for image recognition in the first aspect or any optional implementation of the first aspect. For example, the model training device for image recognition includes: an acquisition module and a training module. Among them,

[0032] The acquisition module is used to acquire training data in various fields.

[0033] The training module is used to train a preprocessing model using training data from multiple fields; the preprocessing model is used to adjust the pixel values ​​of pixels in an image; wherein the image to be processed is input into the trained preprocessing model, and the similarity between the features of the obtained preprocessed image and the features of the source domain of the recognition model reaches a first threshold, and the recognition model is used to identify the category to which the image belongs.

[0034] In a possible implementation, the preprocessing model includes a first sub-model and a second sub-model; the first sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain is lower than a second threshold, and the multiple pixels in the image to be processed correspond to a feature of the image to be processed; the second sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain reaches the second threshold.

[0035] In one possible implementation, the loss function corresponding to the first sub-model is a cross-entropy loss function; the loss function corresponding to the second sub-model is a cross-entropy loss function and a steady-state regularization term, wherein the steady-state regularization term is used to indicate: if the domain to which the training data belongs changes during the training process, the parameters in the preprocessing model whose change range exceeds the third threshold are adjusted.

[0036] In a fifth aspect, the present application provides a chip comprising: an interface circuit and a power supply circuit; the interface circuit is used to obtain an analysis request, and the control circuit is used to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect, or the above-mentioned second aspect or any possible implementation of the second aspect.

[0037] In a sixth aspect, the present application provides a computing device. The computing device includes a memory and a processor, the memory is used to store computer instructions; when the processor executes the computer instructions, the method in the first aspect or any possible implementation of the first aspect is implemented, or the method in the second aspect or any possible implementation of the second aspect is implemented. The computing device may refer to a server or a personal computer.

[0038] In the seventh aspect, an image recognition system is provided, which includes a training device and at least one terminal, wherein the at least one terminal is connected to the training device, and the at least one terminal is used to execute the method in the first aspect or any possible implementation of the first aspect, and the training device is used to execute the method in the second aspect or any possible implementation of the second aspect.

[0039] In an eighth aspect, the present application provides a computer-readable storage medium storing a computer program or instructions. When the computer program or instructions are executed by a computing device, the method in the first aspect or any possible implementation of the first aspect is implemented, or the method in the second aspect or any possible implementation of the second aspect is implemented.

[0040] In a ninth aspect, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a computing device, it implements the method in the above-mentioned first aspect or any optional implementation of the first aspect, or the method in the second aspect or any possible implementation of the second aspect.

[0041] The beneficial effects of the third to ninth aspects above can refer to the description of the first aspect or any implementation of the first aspect, and will not be repeated here. Based on the implementations provided by the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the structure of a neural network provided for this application;

[0043] Figure 2 A schematic diagram of the structure of a convolutional neural network provided for this application;

[0044] Figure 3 A schematic diagram of a model domain adaptation method;

[0045] Figure 4 A schematic diagram of the architecture of an image recognition system provided in this application;

[0046] Figure 5 A flowchart of a model training method provided in this application;

[0047] Figure 6 A flowchart of an image recognition method provided in this application;

[0048] Figure 7 A schematic diagram of adjusting the image to be processed provided in this application;

[0049] Figure 8A schematic diagram of a model training device for image recognition provided in this application;

[0050] Fig. 9 A schematic diagram of an image recognition device provided in this application;

[0051] Fig.10 A schematic diagram of the structure of a computing device provided in this application. DETAILED DESCRIPTION

[0052] To facilitate understanding, the technical terms involved in this application are first introduced.

[0053] Domain adaptation: refers to the process of adapting a pre-trained model or algorithm to a new domain or task.

[0054] Continue test time adaptation (CoTTA): refers to a method used to solve the problem of adapting deep neural networks to continuously changing target domains.

[0055] Domain: refers to a set of related datasets or problems that share some specific features and properties. For example, the image domain may include a set of image datasets that may contain image data of the same content in different styles, such as images of horses in real environments, images of horses hand-painted, or images of abstract horses.

[0056] Error accumulation means that in neural network training, each layer may introduce some errors, which will be accumulated and amplified in subsequent layers of the network, eventually leading to model instability or wrong prediction results.

[0057] A neural network can be composed of neurons, where neurons can refer to s The output of this operation unit satisfies the following formula 1.

[0058]

[0059] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s, b is the bias of the neuron. f is the activation function of the neuron, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neuron into the output signal. The output signal of the activation function can be used as the input of the next layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neurons mentioned above, that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neurons. The weight characterizes the strength of the connection between different neurons. The weight determines the influence of the input on the output. A weight close to 0 means that changing the input does not change the output. A negative weight means that increasing the input reduces the output.

[0060] like Figure 1 As shown, Figure 1 A schematic diagram of the structure of a neural network provided for the present application. The neural network 100 includes N processing layers, where N is an integer greater than or equal to 3. The first layer of the neural network 100 is the input layer 110, which is responsible for receiving input signals, and the last layer of the neural network 100 is the output layer 130, which is responsible for outputting the processing results of the neural network. The other layers except the first and last layers are intermediate layers 140, and these intermediate layers 140 together constitute the hidden layer 120. Each intermediate layer 140 in the hidden layer 120 can both receive input signals and output signals. The hidden layer 120 is responsible for the processing of input signals. Each layer represents a logical level of signal processing. Through multiple layers, the data signal can be processed by multiple levels of logic.

[0061] In some feasible embodiments, the input signal of the neural network can be a signal in various forms such as a video signal, a voice signal, a text signal, an image signal or a temperature signal. The video signal or the image signal can be various sensor signals such as an image signal recorded or photographed by a camera (image sensor). The input signal of the neural network also includes various other computer-processable engineering signals, which are not listed here one by one. If the neural network is used to perform deep learning on the image signal, the quality of the image processed by the neural network can be improved.

[0062] Convolutional neural network (CNN) is a deep neural network with convolutional structure. Convolutional neural network contains a feature extractor composed of convolution layer and subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or feature map. Convolution layer refers to the neuron layer in convolutional neural network that performs convolution processing on the input signal. In the convolution layer of convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer can output several feature maps, which can refer to the intermediate results in the operation process of convolutional neural network. Neurons in the same feature map share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. That is, the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0063] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0064] For example, Figure 2 As shown, Figure 2 A schematic diagram of the structure of a convolutional neural network provided in the present application. The convolutional neural network 200 may include an input layer 210, a convolutional layer / pooling layer 220 (wherein the pooling layer is optional) and a multi-layer perceptron 230.

[0065] The convolution layer / pooling layer 220 may include, for example, layers 221 to 226. In one example, layer 221 may be, for example, a convolution layer, layer 222 may be, for example, a pooling layer, layer 223 may be, for example, a convolution layer, layer 224 may be, for example, a pooling layer, layer 225 may be, for example, a convolution layer, and layer 226 may be, for example, a pooling layer. In another example, layers 221 and 222 may be, for example, convolution layers, layer 223 may be, for example, a pooling layer, layers 224 and 225 may be, for example, convolution layers, and layer 226 may be, for example, a pooling layer. The output of a convolution layer may be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0066] The convolution layer 221 will be taken as an example to introduce the internal working principle of a convolution layer.

[0067] The convolution layer 221 may include a plurality of convolution operators, which may also be referred to as kernels. The role of the convolution operator in image processing is equivalent to a filter that extracts specific information from an input image matrix. The convolution operator may essentially be a weight matrix, which is usually predefined. The size of the weight matrix is ​​related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output of a single depth dimension, but in most cases a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), i.e., multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image. Different weight matrices can be used to extract different features in an image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of an image, and another weight matrix is ​​used to blur unwanted noise in an image. The multiple weight matrices have the same size (rows × columns), and the sizes of the feature maps extracted by the multiple weight matrices with the same size are also the same. The multiple feature maps with the same size extracted are then merged to form the output of the convolution operation.

[0068] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0069] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (e.g., layer 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (e.g., layer 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0070] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 2Each layer 221 to 226 of the example of the convolution layer / pooling layer 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the process of image processing, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value within the range as the result of maximum pooling within a specific range. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0071] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the multi-layer perceptron 230 to generate one or a group of outputs of the required number of classes. Therefore, the multi-layer perceptron 230 may include multiple hidden layers (such as Figure 2 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in the multiple hidden layers can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0072] After the multiple hidden layers in the multi-layer perceptron 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 2 The propagation from layer 210 to layer 240 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from layer 240 to layer 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0073] It should be noted that if Figure 2The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, such as U-Net, residual network (ResNet), etc.

[0074] Recurrent neural networks (RNN) are used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The same error back propagation algorithm is used, but there is a difference: if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above. And when using the gradient descent algorithm, the output of each step depends not only on the network at the current step, but also on the state of the network at the previous steps.

[0075] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0076] In specific applications, recurrent neural networks can exist in the form of various network models, such as long short-term memory networks (LSTM).

[0077] Generative adversarial networks (GAN) are a type of deep learning model. The model includes at least two modules: one is the generative model and the other is the discriminative model. These two modules learn from each other through competition to produce better output. Both the generative model and the discriminative model can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Taking the GAN that generates images as an example, assume that there are two networks, G (generator) and D (discriminator), where G is a network that generates images. It receives a random noise z and generates images through this noise, denoted as G(z); D is a discriminative network that is used to determine whether a picture is "real". Its input parameter is x, which represents a picture. The output D(x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if it is 0, it means that it cannot be a real picture. In the process of training the generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. In this way, G and D constitute a dynamic "game" process, which is also the "adversary" in the "generative adversarial network". The final result of the game is that, under ideal conditions, G can generate images G(z) that are "real" enough, while D has difficulty determining whether the images generated by G are real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0078] In specific applications, generative adversarial networks can exist in the form of various network models, such as an image synthesis model based on a generative adversarial network, or a Pix2Pix network using a U-Net network structure.

[0079] Loss function, in the process of training deep neural networks, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0080] Back propagation algorithm, convolutional neural network can use error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0081] When training a deep neural network, if the training data follows a consistent distribution, the trained deep neural network will have a stronger recognition ability. However, when there is a domain gap between the training data, the recognition performance of the trained deep neural network will be significantly reduced.

[0082] When the trained deep neural network is domain adapted again, the distribution of the training data used changes over time. For example, when the trained deep neural network (pre-trained model) is a traffic light recognition model, the traffic light recognition model is used to recognize changes in traffic lights on the street. The target domain of the traffic light recognition model is constantly changing over time, such as rain, fog, or snow. Therefore, the traffic light recognition model needs to adapt to the target domain to improve the recognition accuracy of the traffic light recognition model for traffic lights in rainy, foggy, or snowy days. To solve the aforementioned problems, CoTTA and test time adaptation (TTA) are introduced to fine-tune the model or change the output distribution of the model during adaptation.

[0083] If TTA is used to improve the recognition accuracy of the traffic light recognition model, in order to avoid the risk of modifying the structure of the traffic light recognition model and causing the traffic light recognition model to degenerate, only the Laplace transform is used to change the output of the traffic light recognition model, and then adaptive maximum likelihood estimation is performed to adjust the parameters of the traffic light recognition model. However, the aforementioned method requires that the target domain of the training data has a stable distribution, so it does not take into account the continuous change of the target domain, so the traffic light recognition model processed by the TTA method has a poor recognition effect on the changing target domain.

[0084] If CoTTA is used to improve the recognition accuracy of the traffic light recognition model, in order to avoid error accumulation in the traffic light recognition model, the pseudo-labels in the training process are fine-tuned by weights and data augmentation averages, and the traffic light is finally determined based on the pseudo-labels. In order to avoid catastrophic forgetting in the traffic light recognition model, the parameters of a small number of neurons in the traffic light recognition model trained in the current round are randomly restored to the parameters of the neurons in the traffic light recognition model trained in the previous round, so as to achieve long-term adaptation of all parameters in the traffic light recognition model. However, since the pseudo-labels obtained by the traffic light recognition model are still unreliable, they are limited in avoiding error accumulation. And when the domain gap between the target domains is large, the catastrophic forgetting of the model cannot be completely solved by random recovery. In summary, the use of CoTTA still cannot effectively improve the recognition accuracy of the traffic light recognition model for the changing target domain.

[0085] For example, the above CoTTA and TTA methods are adjustments to the pre-trained model. Figure 3 As shown, Figure 3 Schematic diagram of a model domain adaptation method. When the CoTTA and TTA methods are used to adapt the target domain using the pre-trained model, a new model will be obtained. The source domain of the pre-trained model is a horse image in a real environment.

[0086] For example, if the target domain is an oil painting style horse image, the pre-trained model is used to adapt to the target domain, and the model parameters or structure in the pre-trained model are changed to obtain the target model 1.

[0087] If the target domain is a cartoon-style horse image, the traffic light recognition model is used to adapt to the target domain to obtain target model 2.

[0088] If the target domain is a hand-painted horse image, the traffic light recognition model is used to adapt to the target domain to obtain target model 3.

[0089] In the above example, the model can only be trained on data from a single target domain to obtain the corresponding target model. Since the target domain has different features from the source domain used by the pre-trained model, the above-mentioned error accumulation and catastrophic forgetting problems will occur in the process of training the target model using the pre-trained model, resulting in the target model having poor recognition accuracy for images belonging to the source domain. That is, the target model learns the parameters of the new domain and forgets the parameters of the source domain. The target model obtained by the above method can only have high recognition accuracy for images in a single domain, but has low recognition accuracy for images belonging to different target domains.

[0090] In order to improve the recognition accuracy of the model for the target domain, the present application provides a model training method, which includes: obtaining training data from multiple fields, and using the training data from the multiple fields to train a preprocessing model. The preprocessing model is used to adjust the pixel values ​​of pixels in the image, and the image to be processed is input into the trained preprocessing model to obtain a feature similarity between the feature of the preprocessed image and the feature of the source domain of the recognition model to reach a first threshold, and the recognition model is used to identify the category to which the image belongs.

[0091] In the present application, only the preprocessing model is trained using training data from multiple fields, while the recognition model is not trained, thereby avoiding the problem of degradation and catastrophic forgetting of the recognition model caused by training the recognition model during multi-domain adaptation, that is, the recognition accuracy of the recognition model for images belonging to the source domain will not change. The similarity between the features of the preprocessed image obtained by adjusting the trained preprocessing model and the features of the source domain reaches a first threshold, that is, the preprocessed image is close to the source domain of the recognition model, thereby improving the recognition accuracy of the recognition model for the preprocessed image.

[0092] Based on a preprocessing model, the present application provides an image recognition method, which includes: obtaining an image to be processed, adjusting pixel values ​​of pixels in the image to be processed, obtaining a preprocessed image whose features have a similarity with features in a source domain of a recognition model that reaches a first threshold, and then inputting the preprocessed image into a recognition model for identifying the category to which the image belongs, thereby obtaining the category to which the image to be processed belongs.

[0093] In the present application, by adjusting the pixels of the image to be processed, the similarity between the features of the obtained pre-processed image and the features of the source domain of the recognition model reaches a first threshold, and then the pre-processed image is close to the source domain of the recognition model, thereby improving the recognition accuracy of the recognition model for the image to be processed and the robustness of the recognition model for image recognition. In addition, when the recognition model has a high recognition rate for images belonging to the source domain, since the pre-processed image is close to the source domain, the recognition model has a high recognition rate for the pre-processed image, thereby further improving the accuracy of the recognition model in obtaining the category to which the image to be processed in different fields belongs.

[0094] Next, the image recognition and model training methods provided in this application are introduced in detail in conjunction with the accompanying drawings.

[0095] First, see Figure 4 , Figure 4 This is a schematic diagram of the architecture of an image recognition system provided in this application. Figure 4 As shown, the image recognition system includes an execution device 410 , a training device 420 , a database 430 , a terminal device 440 , a data storage system 450 and a data acquisition device 460 .

[0096] The execution device 410 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera or a vehicle-mounted terminal, etc., and can also be an edge device (for example, a box carrying a chip with processing capability), etc.

[0097] The training device 420 may be a terminal or other computing device, such as a server or a cloud device.

[0098] As a possible embodiment, the execution device 410 and the training device 420 are different processors deployed on different physical devices (such as a server or a server in a cluster). For example, the execution device 410 can be a graphics processing unit (GPU), a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The training device 420 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0099] In another possible embodiment, the execution device 410 and the training device 420 are deployed on the same physical device, or the execution device 410 and the training device 420 are the same physical device.

[0100] The data acquisition device 460 is used to collect training data and store the training data in the database 430. The data acquisition device 460 can be the same as or different from the execution device 410 and the training device 420. The training data includes data in at least one form of image, voice and text. For example, the training data includes image data belonging to different fields.

[0101] The training device 420 is used to train the neural network using the training data until the loss function in the neural network converges, and the neural network training is completed when the loss function value is less than a specific threshold, so that the neural network reaches a certain accuracy. Alternatively, all the training data in the database 430 are used for training, and the neural network training is completed, so that the trained neural network has the function of identifying the category of the continuously changing target domain. Furthermore, the training device 420 configures the trained neural network 401 to the execution device 410. The execution device 410 is used to implement the function of processing input data according to the trained neural network 401.

[0102] In some embodiments, the execution device 410 and the training device 420 are the same computing device, and the computing device can configure the trained neural network 401 to itself, and use the trained neural network 401 to implement functions such as adjusting the pixel values ​​of pixel points of an image or identifying the categories of image data in various different fields.

[0103] In other embodiments, the training device 420 can configure the trained neural network 401 to multiple execution devices 410. Each execution device 410 uses the trained neural network 401 to adjust the pixel values ​​of the pixels of the image, or identify the categories of image data in multiple different fields.

[0104] In combination with the image recognition system, the model training method for image recognition provided in this embodiment can be applied to model adaptation scenarios. Specifically, the model training method for image recognition in the embodiment of the present application can be applied to scenarios such as cross-scene / domain recognition. The following is a brief introduction to the application of the aforementioned model training method for image recognition in cross-scene / domain recognition scenarios.

[0105] For example, the training device 420 receives training data input by a user or obtains training data from the database 430, and the training data includes image data of multiple different fields, and the image data of multiple different fields can be different expressions of unified content, for example, real horse images, hand-painted horse images, or cartoon-style horse images, etc. The training device 420 trains the preprocessing model according to the aforementioned image data of multiple different fields.

[0106] It should be noted that, in actual applications, the training data (e.g., image data in multiple fields) maintained in the database 430 may not all come from the data acquisition device 460, but may also be received from other devices. In addition, the training device 420 may not train the neural network entirely based on the training data maintained by the database 430, but may also obtain training data from the cloud or other places to train the neural network. The above description should not be used as a limitation on the embodiments of the present application.

[0107] Furthermore, according to the functions performed by the execution device 410, the execution device 410 can be further subdivided, such as Figure 4 As shown, the execution device 410 is configured with a calculation module 411 , an I / O interface 412 and a pre-processing module 413 .

[0108] The I / O interface 412 is used to exchange data with external devices. A user can input data to the I / O interface 412 through a terminal device 440. The input data may include images or videos. In addition, the input data may also come from a database 430.

[0109] The preprocessing module 413 is used to perform preprocessing according to the input data received by the I / O interface 412. In the embodiment of the present application, the preprocessing module 413 can be used to improve the clarity of the image data received from the I / O interface 412, etc.

[0110] When the execution device 410 preprocesses the input data, or when the computing module 411 of the execution device 410 performs calculations and other related processing, the execution device 410 can call the data, code, etc. in the data storage system 450 for corresponding processing, and can also store the data and instructions obtained from the corresponding processing in the data storage system 450.

[0111] For example, the first neural network stored in the execution device 410 can be applied to the execution device 410. After the execution device 410 obtains the image data, the calculation module 411 inputs the image data into the first neural network to obtain a processing result. Since the first neural network is trained by the training device 420 based on image data in multiple fields, the input image data is processed using the first neural network to meet the user's requirements for identifying image data in multiple fields.

[0112] Finally, the I / O interface 412 returns the processing result to the terminal device 440, thereby providing it to the user so that the user can view the processing result. It should be understood that the terminal device 440 and the execution device 410 can also be the same physical device.

[0113] exist Figure 4 In the case shown, the user can manually input data, which can be operated through the interface provided by the I / O interface 412. In another case, the terminal device 440 can automatically send input data to the I / O interface 412. If the terminal device 440 needs to obtain the user's authorization to automatically send the input data, the user can set the corresponding authority in the terminal device 440. The user can view the processing results output by the execution device 410 on the terminal device 440, and the specific presentation form can be display, sound, action and other specific methods. The terminal device 440 can also be used as a data acquisition terminal to collect data such as Figure 4 The input data of the input I / O interface 412 and the processing result of the output I / O interface 412 are taken as new sample data and stored in the database 430. Of course, the data may be collected without going through the terminal device 440, but the input data of the input I / O interface 412 and the processing result of the output I / O interface 412 are stored in the database 430 as new sample data by the I / O interface 412.

[0114] Figure 4 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 4The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 4 In the embodiment, the data storage system 450 is an external memory relative to the execution device 410. In other cases, the data storage system 450 may also be placed in the execution device 410.

[0115] This application shows a possible implementation of a model training method for image recognition by training device 420. The model training method can be applied to Figure 4 In the image recognition system shown in Figure 5 As shown, Figure 5 The flowchart of a model training method provided in the present application is illustrated by taking a computing device executing the model training method as an example. The following takes the computing device as a training device 420 as an example to illustrate the model training method provided in the present application, but does not constitute a specific limitation. Figure 5 As shown, the model training method may include the following steps S510 to S520.

[0116] S510. The training device 420 obtains training data in various fields.

[0117] The training data in various fields may be images of the same content in different scenes, such as images of traffic lights in rainy days, traffic lights in sunny days, or traffic lights in foggy days, or images of horses in real environments, images of horses in hand-painted styles, or images of horses in cartoon styles.

[0118] In a possible scenario, the training data may include data of a source domain of a recognition model. The recognition model is used to recognize the category to which an image belongs.

[0119] With respect to the training device 420 acquiring training data in various fields, two possible examples are provided below.

[0120] Example 1: The training device 420 obtains training data in various fields from the database 430.

[0121] Example 2: The training device 420 receives training data in various fields input by a user.

[0122] S520, the training device 420 trains the preprocessing model using training data from various fields.

[0123] Among them, the preprocessing model is used to adjust the pixel values ​​of pixels in the image.

[0124] For the trained preprocessing model, the image to be processed is input into the trained preprocessing model, and the similarity between the features of the obtained preprocessing image and the features of the source domain of the recognition model reaches a first threshold. The recognition model is used to identify the category to which the image belongs.

[0125] Exemplarily, the preprocessing model can adjust the image accordingly according to the field to which the image belongs. The network structure of the preprocessing model can be CNN or LeNet, etc. The LeNet includes two convolutional layers, two pooling (downsampling) layers and three fully connected layers. For the network structure of CNN, please refer to the above Figure 1 The content shown is not repeated here. The first threshold may be 70%, and the present application does not limit the first threshold, and it may also be 80%.

[0126] With respect to the training phase ① in which the training device 420 trains the preprocessing model using training data from various fields, two possible examples are provided below.

[0127] In a first possible example, the training device 420 trains the preprocessing model alone.

[0128] During the training process, the training device 420 will use the training data from various fields to input into the preprocessing model for supervised learning. The training data from various fields include images from various fields and their corresponding images with adjusted pixel values.

[0129] In a second possible example, the training device 420 combines the preprocessing model and the recognition model for training.

[0130] Since the preprocessing model is only used to adjust the pixel values ​​of pixels in the image, and the recognition model is used to identify the category of the image, the training device 420 combines the preprocessing model with the recognition model for training, and can directly adjust the parameters of the preprocessing model according to the recognition result. The training data in the above-mentioned multiple fields include images in multiple fields and the categories to which they belong.

[0131] When the training device 420 combines the preprocessing model with the recognition model for training, the recognition model is frozen, that is, during the training process and the subsequent testing process, the structure and parameters of the recognition model will not change.

[0132] Based on the above two possible examples, the training device 420 can use the teacher-student model as a framework to train the preprocessing model. The teacher-student model can be used to directly compare the parameter differences of the two models in two adjacent iterations. When the student network in the teacher-student model completes an iteration, the parameters before and after the change of the student network will be updated to the parameters of the teacher network through exponential moving average. During the training process, when the preprocessing model meets the exit condition of the training, the trained preprocessing model is obtained.

[0133] Among them, the exit condition may be that the loss function of the preprocessing model converges, the training data is used up, or the number of iterations during training reaches a set threshold, etc. This application does not limit the exit condition, and it may also be that the loss function is less than a set value, etc.

[0134] In one possible scenario, examples of two pre-processing models for image processing are shown below.

[0135] Example 1, the preprocessing model can adjust the pixel values ​​of the pixels in the image through a matrix. The size of the matrix can be determined according to the resolution of the training data. For example, if the resolution of the training data is 200x 100, the size of the matrix is ​​200x 100 or less than 200x 100. In addition, the values ​​in the matrix can be randomly generated. For example, the training device 420 can randomly generate values ​​in the matrix, and the values ​​in the matrix satisfy a uniform distribution, or one-hot encoding is used to obtain the values ​​in the matrix. The training device 420 updates the values ​​in the matrix by training the preprocessing model.

[0136] Example 2: The preprocessing model can adjust the pixel value of the pixel in the image through a function. In the preprocessing model, the constant of the function is randomly generated or user-defined. This application does not limit the function, which can be y=ax^3+b, or a more replicated or simple function.

[0137] In other embodiments of the present application, the preprocessing model may also process the image in other forms, which will not be described in detail here.

[0138] In a possible implementation, the preprocessing model includes a first sub-model and a second sub-model. The first sub-model is used to adjust one or more pixels corresponding to features of a specific domain in an image, and the first sub-model may also be called a domain specific prompt (DSP), and the second sub-model is used to adjust one or more pixels corresponding to features of a non-specific domain in an image, and the second sub-model may also be called a domain-agnostic prompt (DAP).

[0139] The first sub-model uses the first sub-parameter to adjust the pixel values ​​of one or more pixels corresponding to the second threshold portion of the feature of the image to be processed, whose similarity with the feature of the source domain of the recognition model is lower than that of the pixel values ​​of the one or more pixels in the image to be processed, and the multiple pixels in the image to be processed correspond to a feature in the image to be processed. The second sub-model uses the second sub-parameter to adjust the pixel values ​​of one or more pixels corresponding to the second threshold portion of the feature of the image to be processed, whose similarity with the feature of the source domain reaches that of the one or more pixels.

[0140] Among them, the features of the specific field, i.e., the features corresponding to the image to be processed, have a similarity with the features of the source domain of the recognition model lower than the second threshold. The features of the non-specific field, i.e., the features corresponding to the image to be processed, have a similarity with the features of the source domain of the recognition model reaching the second threshold.

[0141] Exemplarily, if the training data is horse images in various fields, such as hand-painted horse images and cartoon-style horse images. The source domain is a horse image in a real environment. The image of the training data has the following features: horse, horse outline, hair, horse face, horse eyes, horse color, painting style, etc. The source domain has the following features: horse, horse outline, hair, horse face, shooting style, etc. After matching, the similarity between each feature of the training data and each feature of the source domain can be obtained, and then according to the similarity and the second threshold, the features of the target domain are divided into features of a specific field and features of a non-specific field. Features with a similarity greater than or equal to the second threshold are used as features of a non-specific field: horse, horse outline, hair, horse face, and features with a similarity less than the second threshold are used as features of a specific field: horse eyes, horse color, painting style. The preprocessing model obtains a trained preprocessing model by learning the above-mentioned features of the specific field and features of the non-specific field, that is, the trained first sub-model and the trained second sub-model.

[0142] It is worth noting that the above features can be represented by vectors, and when calculating similarity, the distance between vectors is calculated, such as Euclidean distance, Minkowski distance, cosine similarity, etc. The above second threshold can be 0.6, and this application does not limit the second threshold, and the second threshold can also be 0.7 or 0.8, etc.

[0143] In the present application, the preprocessing model learns the above-mentioned features of the specific field and the features of the non-specific field respectively, so that the execution device 410 processes the pixel points corresponding to the features of the specific field according to the first sub-model, and processes the pixel points corresponding to the features of the non-specific field according to the second sub-model, so that the obtained preprocessed image is close to the source domain, thereby improving the accuracy of the recognition model in recognizing the preprocessed image.

[0144] On the basis of the above-mentioned training stage ①, in the use stage ② of the preprocessing model, the trained preprocessing model can make corresponding adjustments to the image according to the field to which the image belongs.

[0145] Exemplarily, the trained preprocessing model adjusts the pixels in the image by identifying the domain to which the image belongs and adopts an adjustment strategy that matches the domain to which the image belongs, thereby obtaining a preprocessed image. The adjustment strategy may be the matrix in Example 1 above.

[0146] The trained preprocessing model can adjust the image accordingly according to the field to which the image belongs. For example, the execution device 410 inputs the image to be processed into the preprocessing model to obtain the preprocessed image content, which will not be described in detail here.

[0147] In other embodiments of the present application, there is also a model testing process. In the model testing process, the pre-processing model after training can also be retrained. The training device 420 inputs the test data into the model that combines the pre-processing model after training and the recognition model to obtain the output result corresponding to the test data. The training device 420 readjusts the pre-processing model after training according to the test data and its corresponding output result.

[0148] The training device 420 uses training data from various fields to train the preprocessing model to obtain the preprocessed content after training, uses the preprocessing model to adjust the pixels of the image according to the matrix, and trains in a manner that combines the preprocessing model with the recognition model. A possible implementation method is provided below.

[0149] Since the recognition model is frozen during the training process of the preprocessing model combined with the recognition model, only the preprocessing model needs to be trained, and the recognition model only needs to obtain the corresponding category according to the preprocessed image output by the preprocessing model. In order to learn the features of specific fields and the features of non-specific fields in the training data of multiple fields, the training device 420 divides the preprocessing model into a first sub-model and a second sub-model to respectively learn the features of specific fields and the features of non-specific fields.

[0150] For example, the training device 420 uses the cross entropy loss function as the loss function of the first sub-model, and uses the cross entropy loss function and the steady-state regularization term as the loss function of the second sub-model.

[0151] The steady-state regularization term is used to adjust the parameters in the second sub-model that are sensitive to changes in the domain of the training data during the training process, that is, when the domain to which the training data belongs changes, the parameters in the second sub-model whose change range exceeds the third threshold are adjusted. Furthermore, the training device 420 trains the first sub-model and the second sub-model according to the above loss function to obtain the trained first sub-model and the trained second sub-model.

[0152] Exemplarily, the third threshold may be 0.1. The present application does not limit the third threshold, and the third threshold may also be 0.2 or 0.05, etc.

[0153] In the present application, the training device 420 sets different loss functions for the first sub-model and the second sub-model so that the first sub-model and the second sub-model can respectively learn the differences in the features of the image to be processed, the differences in the features of the source domain whose similarity is lower than the second threshold, and the differences in the features of the source domain whose similarity reaches the second threshold, thereby realizing that the first sub-model and the second sub-model can respectively make differentiated adjustments to the pixel points corresponding to different features.

[0154] In the process of performing multiple rounds of iterative training using the preprocessing model of the training data, the training data used in each round of training can be images of multiple continuously changing fields. For example, images of multiple continuously changing fields such as hand-painted style horse images, cartoon style horse images, or oil painting style horse images. In other words, each round of training needs to include images of multiple fields. Then, the preprocessing model learns the processing strategies corresponding to the images in each field.

[0155] The following is a possible example of the loss function for the preprocessing model.

[0156] The loss function of the first sub-model is as follows:

[0157]

[0158] Among them, the ω φ represents the first sub-model, represents the training data, represents randomly enhanced training data, Represents the moving average vector corresponding to the randomly enhanced training data, The aforementioned enhanced training data is an image obtained after the training device 420 performs image segmentation, offset or definition enhancement on the training data.

[0159] The loss function of the second sub-model is as follows:

[0160]

[0161] Among them, the ψ δ represents the second sub-model, represents the steady-state regularization term.

[0162] The above steady-state regularization term can be expressed as formula 4:

[0163]

[0164] The above α is used to control the contribution of the steady-state regularization term. If the α is small, it means that a parameter in the second sub-model contributes less. represents the steady-state factor, θ represents the parameter that changes in real time in the current training round, and θ* Parameters representing the last change in the current training round.

[0165] For It can be calculated by the following formula 5:

[0166]

[0167] in, It is used to represent the parameter differences of the model between domains v, that is, Should represents the importance weight, ξ represents a non-zero value, such as 0.01, which is used to avoid Approaching 0, In the case of infinity, τ refers to the current domain, and v refers to the domain before τ. When it is small, it means that the parameters of the model change slowly. When is larger, it means that the model parameters contribute more effectively to the loss. Smaller or When it is larger, it means that the parameters of the model are more sensitive to changes in the domain of the training data. That is, when the domain of the training data changes, the parameters in the second sub-model whose change range exceeds the threshold are adjusted. Therefore, Formula 4 is used to adjust the parameters that are sensitive to domain changes, and at the same time, the parameters in the second sub-model that are insensitive to the domain are stably updated.

[0168] For It can be calculated by the following formula 6:

[0169]

[0170] Among them, t0 and t1 represent two adjacent iterations in the training process, g(θ(t)) represents the gradient of the loss function in the t-th round of training, and θ'(t) represents the model parameters after the moving average of the t-th round of training.

[0171] In summary, the loss function of the preprocessing model is as follows:

[0172]

[0173] The training device 420 continuously trains the preprocessing model by using training data from various fields until the loss function shown in the above formula 7 converges, thereby obtaining a trained preprocessing model. Correspondingly, the first sub-model learns the training data using the above formula 2, and compared with the characteristics of the specific field of the source domain, the corresponding matrix 1 can be obtained, and the second sub-model learns the characteristics of the non-specific field between the training data and the source domain using the above formula 3, and the corresponding matrix 2 can be obtained.

[0174] In one possible scenario, to update the above The training device 420 needs to know the domain changes of the training data. To address this problem, the training device 420 predicts whether the current domain has changed by predicting the confidence change Conf(t) of the pseudo-label of the training data according to the preprocessing model during the training process.

[0175] Exemplarily, the training device 420 determines the confidence of the pseudo labels of two adjacent batches of data, compares the difference in the confidence of the pseudo labels of the two adjacent batches of data with a set threshold, and if the difference exceeds the set threshold, performs a reverse update using the aforementioned loss to adjust θ * and Update the preprocessing model to change the processing strategy according to the changes in the field of training data. In other words, the preprocessing model uses different matrices to process images according to the training data in different fields.

[0176] The following provides a comparison of the recognition effects of the above-mentioned trained preprocessing model and recognition model on two test sets.

[0177] Example 1: The recognition accuracy of the trained preprocessing model and recognition model in the CIFAR100->CIFAR100C test set is compared with the recognition accuracy of other methods. The results are shown in Tables 1-1 and 1-2.

[0178] Table 1-1

[0179] method Gaussion shot impulse defocus glass motion zoom snow frost source 73.0 68.0 39.4 29.3 54.1 30.8 28.8 39.5 45.8 BNStatsAdapt 42.1 40.7 42.7 27.6 41.9 29.7 27.9 34.9 35 pseudo-label 38.1 36.1 40.7 33.2 45.9 38.3 36.4 44 45.6 tent-continual 37.2 35.8 41.7 37.9 51.2 48.3 48.5 58.4 63.7 CoTTA 40.1 37.7 39.7 26.9 38.0 27.9 26.4 32.8 31.8 This application 29.5 25.8 31.9 2.8 30.5 7.7 5.7 14.8 14.8

[0180] Table 1-2

[0181] method fog brightness contrast elastic_trans pixelate jpeg source 50.3 29.5 55.1 37.2 74.7 41.2 BNStatsAdapt 41.5 26.5 30.3 35.7 32.9 41.2 pseudo-label 52.8 45.2 53.5 60.1 58.1 64.5 tent-continual 71.1 70.4 82.3 88.0 88.5 90.4 CoTTA 40.3 24.7 26.9 32.5 28.3 33.5 This application 24.2 1.8 6.8 18.5 9.1 28.0

[0182] Based on the contents of Table 1-1 and Table 1-2, the existing methods, such as the above-mentioned source, BN Stats Adapt, pseudo-label, tent-continual, and CoTTA, have average recognition errors of 46.4, 35.4, 46.2, 60.9, and 32.5, respectively, when recognizing the above-mentioned test set across multiple fields including continuous changes. When the preprocessing model and recognition model used in the present application process the test set, the average recognition error is 16.8. Therefore, the preprocessing model and recognition model of the present application have lower errors when recognizing cross-domain data, and thus have more stable and better recognition capabilities. Through the above Tables 1-1 and 1-2, it can also be determined that the pros and cons (gains) of the recognition samples of each method are 0, 11, 0.2, -14.5, and 13.9, respectively, while the gain of the present application is 29.6. The higher the gain is, the higher the recognition accuracy is. Therefore, the preprocessing model and recognition model of the present application have improved recognition accuracy compared to the aforementioned method.

[0183] Example 2: The recognition accuracy of the trained preprocessing model and recognition model in the CIFAR100->CIFAR100C test set is compared with the recognition accuracy of other methods. The results are shown in Tables 2-1 and 2-2. Example 2 is a retest based on Example 1, and the tent-online recognition method is added.

[0184] Table 2-1

[0185]

[0186]

[0187] Table 2-2

[0188] method fog brightness contrast elastic_trans pixelate jpeg source 26 9.3 46.7 26.6 58.5 30.3 BN Stats Adapt 15.3 8.4 12.6 23.8 19.7 27.3 pseudo-label 16.5 10.1 13.4 22.4 18.9 25.9 tent-online 13.7 7.9 12.1 22.0 17.3 24.2 tent-continual 18.6 12.2 20.3 25.7 20.8 24.9 baseline(CoTTA) 12.4 7.5 10.6 18.3 13.4 17.3 This application 9 1.5 5.6 18.5 12.8 18.5

[0189] Based on the contents of Table 2-1 and Table 2-2, the existing methods, such as the above-mentioned source, BN Stats Adapt, pseudo-label, tent-online, tent-continual, and CoTTA, have average recognition errors of 43.5, 20.4, 19.8, 18.6, 20.7, and 16.2, respectively, when identifying the above-mentioned test sets across multiple fields that include continuous changes. When the preprocessing model and recognition model used in this application process the test set, the average recognition error is 13.9. Thus, on the basis of Example 1, the error is further reduced, thereby improving the stability and recognition ability of the recognition ability. It can also be determined from Tables 2-1 and 2-2 that the pros and cons (gains) of the recognition samples of each method are 0, 23.1, 23.7, 24.9, 22.8, and 27.3, respectively, while the gain of this application is 29.6. Based on Example 1, the gain indicator of the present application has not increased, indicating that the preprocessing model of the present application can learn the characteristics of the test set in one test process, thereby ensuring that the accuracy remains high.

[0190] Example 3, the recognition accuracy of the preprocessing model and recognition model after training in the VLCS test set is compared with the recognition accuracy of other methods. The results are shown in Tables 3-1, 3-2 and 3-3. Each table shows the recognition effect of CoTTA and the preprocessing model and recognition model of this application on the test set after a round of training. Table 3-1 shows the results of the first round of training, Table 3-2 shows the results of the second round of training, and Table 3-3 shows the results of the third round of training.

[0191] Table 3-1

[0192]

[0193] Table 3-2

[0194]

[0195] Table 3-3

[0196]

[0197] In summary, Table 3-1, Table 3-2 and Table 3-3 show that in each round of testing, the preprocessing model and recognition model of the present application are combined to identify the data in the test set with a lower recognition error than the recognition error of CoTTA. Moreover, according to Table 3-1, Table 3-2 and Table 3-3 above, it can be determined that the average errors of CoTTA are 56.4, 57.5, and 40.1, and the average errors of the present application are 54.7, 41.6, and 38.7, and the average errors of the present application are also lower than the average error of CoTTA. Therefore, the preprocessing model and recognition model of the present application have lower errors when identifying cross-domain data, and thus have more stable and better recognition capabilities. The gains corresponding to the preprocessing model and recognition model are 1.7, 15.9, and 1.4, respectively, and the corresponding gain of CoTTA is 0. Therefore, the recognition accuracy of the preprocessing model and recognition model of the present application is improved compared to that of CoTTA.

[0198] In one possible scenario, the training device 420 may deploy the trained preprocessing model to a terminal device, such as the execution device 410 . The training device 420 and the execution device 410 may be the same computing device or different computing devices.

[0199] The present application also provides a possible implementation of an image recognition method, which can be applied to Figure 4 In the image recognition system shown in Figure 6 As shown, Figure 6 4 is a flowchart of an image recognition method provided by the present application. The image recognition method is described by taking a computing device as an example. The image recognition method provided by the present application is described by taking a computing device as an execution device 410 as an example, but it does not constitute a specific limitation. Figure 6 As shown, the image recognition method may include the following steps S610 to S630.

[0200] S610: The execution device 410 obtains an image to be processed.

[0201] The image to be processed belongs to any one of the multiple fields corresponding to the aforementioned training data.

[0202] With respect to the execution device 410 acquiring the image to be processed, two possible examples are provided below.

[0203] Example 1: The execution device 410 obtains an image to be processed from the database 430.

[0204] Example 2: The execution device 410 receives an image to be processed input by a user.

[0205] S620: The execution device 410 adjusts the pixel values ​​of the pixels in the image to be processed to obtain a preprocessed image.

[0206] Among them, the similarity between the features of the preprocessed image and the features of the source domain of the recognition model reaches a threshold, and the recognition model is used to identify the type to which the image belongs.

[0207] In a possible implementation, the execution device 410 inputs the image to be processed into a preprocessing model to obtain a preprocessed image. The preprocessing model is used to adjust the pixel values ​​of pixels in the image, and the preprocessing model is trained with training data of the field to which the image to be processed belongs.

[0208] For the preprocessing model and the training process of the preprocessing model, please refer to the above Figure 4 The process of preprocessing model training is not described here.

[0209] In the present application, since the preprocessing model is trained with training data of the field to which the image to be processed belongs, the preprocessing model learns the ability to adjust images of the field to which the image to be processed belongs. The execution device 410 uses the preprocessing model to adjust the pixels of the model to be processed, and the feature similarity of the obtained preprocessed image and the feature of the above-mentioned source domain reaches a first threshold, thereby improving the accuracy of the recognition model in obtaining the category to which the image to be processed belongs.

[0210] In another possible implementation, the execution device 420 may adjust the pixel values ​​of the pixels in the image to be processed by downsampling or convolution.

[0211] With respect to the execution device 410 inputting the image to be processed into the preprocessing model to obtain the content of the preprocessed image, a possible example is provided below.

[0212] The execution device 410 inputs the image to be processed into the preprocessing model, which identifies the field to which the image to be processed belongs, and uses an adjustment strategy that matches the field to which the image to be processed belongs to adjust the pixel values ​​of the pixels in the image to be processed to obtain a preprocessed image.

[0213] Taking the adjustment strategy as the above matrix as an example, the preprocessing model determines the field to which the image to be processed belongs according to the feature distribution of the image to be processed, and then determines the matrix corresponding to the field to which the image to be processed belongs, and directly adds the values ​​in the matrix to the corresponding area of ​​the image to be processed, so as to adjust the pixel values ​​of the pixel points in the corresponding area of ​​the image to be processed, thereby obtaining the preprocessed image.

[0214] Among them, the matrix corresponding to the field to which the image to be processed belongs includes matrix 1 and matrix 2 of the first sub-model and the second sub-model under the field to which the image to be processed belongs, and the values ​​of the aforementioned matrices are the sum of the values ​​of matrix 1 in the first sub-model and the values ​​of matrix 2 in the second sub-model at the corresponding positions.

[0215] For example, the image to be processed is a hand-painted horse image, and the feature distribution of the image to be processed may be the distribution or probability of the horse, the outline of the horse, the hair, the horse face, the horse eyes, etc. Alternatively, the feature distribution of the image to be processed may also be the confidence of the pseudo-label predicted by the preprocessing model for the image to be processed, such as multiple pseudo-labels and their corresponding confidences.

[0216] In the present application, the preprocessing model uses corresponding matrices for images in different fields to adjust the pixels of the image, thereby achieving adaptive adjustment according to the field to which the image to be processed belongs, so as to ensure that the similarity between the features of the adjusted preprocessed image and the features of the above-mentioned source domain reaches a first threshold, thereby improving the accuracy of the recognition model in determining the category to which the image to be processed belongs.

[0217] In one possible example, Figure 7 As shown, Figure 7 This is a schematic diagram of adjusting the image to be processed provided by this application. Figure 7 As shown in a, the resolution of the image to be processed is 3x3, and the matrix corresponding to the field to which the image to be processed in the preprocessing model belongs is also a 3x 3 matrix. The pixel values ​​of the image to be processed are added to the corresponding values ​​in the matrix to obtain the pixel values ​​of each pixel in the preprocessed image.

[0218] In another possible example, Figure 7 As shown in b, the resolution of the image to be processed is 3x3, the matrix in the preprocessing model is 2x2, and the preprocessing model also generates a position parameter, such as (1, 1). The position parameter represents the position of the matrix on the image to be processed. For example, (1, 1) represents the position coordinate of the upper left corner of the matrix on the image to be processed. The starting point of the coordinate system corresponding to the position coordinate is the upper left corner of the image to be processed.

[0219] The execution device 410 adjusts the pixel points corresponding to the features of the specific field and the features of the non-specific field in the image to be processed respectively, and the obtained preprocessed image can be recognized by the preprocessed image and is close to the source domain of the recognition model, thereby improving the accuracy of the recognition model in determining the category to which the image to be processed belongs.

[0220] S630 , the execution device 410 inputs the pre-processed image into the recognition model to obtain the category to which the image to be processed belongs.

[0221] In the present application, the pixels of the image to be processed are adjusted so that the feature similarity between the obtained preprocessed image and the feature of the source domain of the recognition model reaches a first threshold. Then, the preprocessed image is close to the source domain of the recognition model, thereby improving the recognition accuracy of the recognition model for the image to be processed and the robustness of the recognition model in image recognition.

[0222] For example, the execution device 410 inputs the pre-processed image into a recognition model using CNN, RNN, GAN, etc. to obtain the category to which the image to be processed belongs.

[0223] In a possible implementation, the execution device 410 may send the category to which the image to be processed belongs to the terminal device, so that the terminal device displays the category to which the image to be processed belongs to the user. Alternatively, the execution device 410 displays the category to which the image to be processed belongs on a display device connected to the execution device 410.

[0224] In a possible implementation, based on obtaining the category to which the to-be-processed image belongs, the category to which each to-be-processed image belongs is classified and sorted.

[0225] In one possible scenario, the above-mentioned preprocessing model and recognition model will be frozen during use, so as to avoid the problem that the preprocessing model and recognition model are polluted by input data during use, leading to model degradation and catastrophic forgetting.

[0226] It is understandable that in order to implement the functions in the above-mentioned embodiments, the execution device 410 and the training device 420 can be the same or different terminals, and the terminals include hardware structures and / or software modules corresponding to the execution of each function. It should be easy for those skilled in the art to realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0227] Combination of the above Figure 5 The model training method for image recognition provided by this embodiment is described in detail. Figure 8 , describing the model training device for image recognition provided according to this embodiment.

[0228] Figure 8 Schematic diagram of the image recognition model training device provided in the present application. The image recognition model training device can be used to implement the function of the training device in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In this embodiment, the image recognition model training device can be as follows Figure 4 The training device 420 shown may also be a module (such as a chip) applied to a server.

[0229] The model training device 800 for image recognition includes an acquisition module 810 and a training module 820. The model training device 800 for image recognition is used to implement the above Figure 5 The method embodiments shown in FIG. 1 are functions of a computing device.

[0230] The acquisition module 810 is used to acquire training data in various fields.

[0231] The training module 820 is used to train the preprocessing model using training data from various fields; the preprocessing model is used to adjust the pixel values ​​of the pixels in the image; wherein the image to be processed is input into the trained preprocessing model, and the features of the obtained preprocessed image have a similarity with the features of the source domain of the recognition model that reaches a threshold, and the recognition model is used to identify the category to which the image belongs.

[0232] In one possible scenario, the image recognition model training device 800 can deploy the trained preprocessing model to the terminal-side device.

[0233] Exemplarily, the preprocessing model and the recognition model may be deployed together on the end-side device, and then the image recognition method may be implemented on the end-side device.

[0234] Combination of the above Figure 6 and Figure 7 The image recognition method provided by this embodiment is described in detail. Fig. 9 , describing the image recognition device provided according to this embodiment.

[0235] Fig. 9 Schematic diagram of the image recognition device provided in the present application. The image recognition device can be used to implement the functions of the execution device in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In this embodiment, the image recognition device can be as follows Figure 4 The execution device 410 shown may also be a module (such as a chip) applied to a server.

[0236] The image recognition device 900 includes an acquisition module 910, an adjustment module 920 and a recognition module 930. The image recognition device 900 is used to implement the above Figure 6 or Figure 7 The method embodiments shown in FIG. 1 are functions of a computing device.

[0237] The acquisition module 910 is used to acquire the image to be processed.

[0238] The adjustment module 920 is used to adjust the pixel values ​​of the pixels in the image to be processed to obtain a preprocessed image; the similarity between the features of the preprocessed image and the features of the source domain of the recognition model reaches a threshold; the recognition model is used to identify the category to which the image belongs.

[0239] The recognition module 930 is used to input the pre-processed image into the recognition model to obtain the category to which the image to be processed belongs.

[0240] In a possible scenario, the image recognition device 900 may send the obtained category to which the image to be processed belongs to to the terminal device, or send the category to a display device corresponding to the computing device for display.

[0241] It should be understood that the image recognition model training device 800 and the image recognition device 900 of the embodiment of the present application can be implemented by GPU, NPU, ASIC, or programmable logic device (PLD), and the above PLD can be a complex programmable logical device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL) or any combination thereof. It can also be implemented by software Figures 5 to 7 In the method shown, the image recognition model training device 800 and the image recognition device 900 and their respective modules can also be software modules.

[0242] The image recognition model training device 800 and the image recognition device 900 according to the embodiment of the present application may correspond to the method described in the embodiment of the present application, and the above and other operations and / or functions of each unit in the image recognition model training device 800 and the image recognition device 900 are respectively to achieve Figures 5 to 7 For the sake of brevity, the corresponding process of the method shown in any one of the items will not be repeated here.

[0243] An embodiment of the present application also provides a chip system, which includes a control circuit and an interface circuit. The interface circuit is used to obtain training data or images to be processed in various fields, and the control circuit is used to implement the functions of the training device 420 or the execution device 410 in the above method.

[0244] In a possible design, the chip system further includes a memory for storing program instructions and / or data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0245] The present application also provides a computing device, please refer to Fig.10 , Fig.10 The present application provides a schematic diagram of a computing device. The computing device 1000 includes a memory 1001, a processor 1002, a communication interface 1003 and a bus 1004. The memory 1001, the processor 1002 and the communication interface 1003 are connected to each other through the bus 1004.

[0246] The memory 1001 may be a volatile memory, such as a random access memory (RAM). The memory 1001 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 1001 may store computer instructions, and when the computer instructions stored in the memory 1001 are executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute the model training method of image recognition of the software system and the steps in the image recognition method. The memory may also store a data set, for example: a portion of the storage resources in the memory 1001 is divided into an area for storing a program that implements the function of the preprocessing model or recognition model of the embodiment of the present application.

[0247] The processor 1002 may be a general-purpose CPU, an application-specific integrated circuit (ASIC), a GPU, or any combination thereof. The processor 1002 may include one or more chips. The processor 1002 may include an AI accelerator, such as an NPU.

[0248] The communication interface 1003 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the computing device 1000 and other devices or communication networks. For example, an iterative training request can be obtained through the communication interface 1003, and the neural network after iterative training can be fed back.

[0249] The bus 1004 may include a path for transmitting information between various components of the computing device 1000 (eg, the memory 1001 , the processor 1002 , and the communication interface 1003 ).

[0250] The computing device 1000 may be a computer (eg, a server) in a cloud data center, or a computer in an edge data center, or a terminal.

[0251] The functions of the training device 420 may be deployed on each computing device 1000. For example, a GPU is used to implement the functions of the training device 420.

[0252] For the functions of the training device 420 and the functions of the execution device 410 deployed in the same computing device 1000 , the training device 420 can communicate with the execution device 410 through the bus 1004 .

[0253] For the functions of the training device 420 and the functions of the execution device 410 deployed in different computing devices 1000, the training device 420 can communicate with the execution device 410 through a communication network.

[0254] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the above-mentioned image recognition method or image recognition model training method.

[0255] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute an image recognition method or a model training method for image recognition.

[0256] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0257] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image recognition method, characterized in that: The method comprises: Get the image to be processed; Adjusting the pixel values ​​of the pixels in the image to be processed to obtain a preprocessed image; the similarity between the features of the preprocessed image and the features of the source domain of the recognition model reaches a first threshold; the recognition model is used to identify the category to which the image belongs; The pre-processed image is input into the recognition model to obtain the category to which the image to be processed belongs.

2. The method according to claim 1, characterized in that The step of adjusting the pixel values ​​of the pixels in the image to be processed to obtain a preprocessed image includes: The image to be processed is input into a preprocessing model to obtain the preprocessed image; wherein the preprocessing model is used to adjust the pixel values ​​of the pixels in the image, and the preprocessing model is trained with training data of the field to which the image to be processed belongs.

3. The method according to claim 2, characterized in that The step of inputting the image to be processed into a preprocessing model to obtain the preprocessed image comprises: Inputting the image to be processed into the preprocessing model; The preprocessing model identifies the domain to which the image to be processed belongs, and an adjustment strategy matching the domain to which the image to be processed belongs is adopted to adjust the pixels in the image to be processed accordingly to obtain the preprocessed image.

4. The method according to claim 2 or 3, characterized in that: The preprocessing model includes a first sub-model and a second sub-model; The first sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to a portion of the feature of the image to be processed whose similarity with the feature of the source domain is lower than a second threshold, and the multiple pixels in the image to be processed correspond to a feature of the image to be processed; The second sub-model is used to adjust the pixel values ​​of one or more pixel points corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain reaches a second threshold.

5. The method according to claim 4, characterized in that The loss function corresponding to the first sub-model is the cross entropy loss function; The loss function corresponding to the second sub-model is a cross entropy loss function and a steady-state regularization term, and the steady-state regularization term is used to indicate: if the domain to which the training data belongs changes during the training process, adjust the parameters in the preprocessing model whose change range exceeds the third threshold.

6. A model training method for image recognition, characterized in that: The method comprises: Obtain training data in a variety of fields; The preprocessing model is trained using the training data of the multiple fields; the preprocessing model is used to adjust the pixel values ​​of the pixels in the image; Among them, the image to be processed is input into the trained preprocessing model, and the similarity between the features of the obtained preprocessed image and the features of the source domain of the recognition model reaches a first threshold, and the recognition model is used to identify the category to which the image belongs.

7. The method according to claim 6, characterized in that The preprocessing model includes a first sub-model and a second sub-model; The first sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to a portion of the feature of the image to be processed whose similarity with the feature of the source domain is lower than a second threshold, and the multiple pixels in the image to be processed correspond to a feature of the image to be processed; The second sub-model is used to adjust the pixel values ​​of one or more pixel points corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain reaches a second threshold.

8. The method according to claim 7, characterized in that The loss function corresponding to the first sub-model is the cross entropy loss function; The loss function corresponding to the second sub-model is a cross entropy loss function and a steady-state regularization term, and the steady-state regularization term is used to indicate: if the domain to which the training data belongs changes during the training process, adjust the parameters in the preprocessing model whose change range exceeds the third threshold.

9. An image recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed; An adjustment module, used for adjusting the pixel values ​​of the pixels in the image to be processed to obtain a preprocessed image; the similarity between the features of the preprocessed image and the features of the source domain of the recognition model reaches a threshold; the recognition model is used for identifying the category to which the image belongs; The recognition module is used to input the pre-processed image into the recognition model to obtain the category to which the image to be processed belongs.

10. The device according to claim 9, characterized in that The adjustment module is specifically used to input the image to be processed into a preprocessing model to obtain the preprocessed image; wherein the preprocessing model is used to adjust the pixel values ​​of the pixels in the image, and the preprocessing model is trained with training data in the field to which the image to be processed belongs.

11. The device according to claim 10, characterized in that The adjustment module is also specifically used to input the image to be processed into the preprocessing model; and the preprocessing model identifies the field to which the image to be processed belongs, adopts an adjustment strategy that matches the field to which the image to be processed belongs, and adjusts the pixels in the image to be processed accordingly to obtain the preprocessed image.

12. The device according to claim 10 or 11, characterized in that The preprocessing model includes a first sub-model and a second sub-model; The first sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to a portion of the feature of the image to be processed whose similarity with the feature of the source domain is lower than a second threshold, and the multiple pixels in the image to be processed correspond to a feature of the image to be processed; The second sub-model is used to adjust the pixel values ​​of one or more pixel points corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain reaches a second threshold.

13. The device according to claim 12, characterized in that The loss function corresponding to the first sub-model is the cross entropy loss function; The loss function corresponding to the second sub-model is a cross entropy loss function and a steady-state regularization term, and the steady-state regularization term is used to indicate: if the domain to which the training data belongs changes during the training process, adjust the parameters in the preprocessing model whose change range exceeds the third threshold.

14. A model training device for image recognition, characterized in that: The device comprises: Acquisition module, used to obtain training data in various fields; A training module, used to train a preprocessing model using the training data in the various fields; the preprocessing model is used to adjust the pixel values ​​of pixels in the image; Among them, the image to be processed is input into the trained preprocessing model, and the similarity between the features of the obtained preprocessed image and the features of the source domain of the recognition model reaches a threshold. The recognition model is used to identify the category to which the image belongs.

15. The device according to claim 14, characterized in that The preprocessing model includes a first sub-model and a second sub-model; The first sub-model is used to adjust the pixel values ​​of one or more pixels corresponding to a portion of the feature of the image to be processed whose similarity with the feature of the source domain is lower than a second threshold, and the multiple pixels in the image to be processed correspond to a feature of the image to be processed; The second sub-model is used to adjust the pixel values ​​of one or more pixel points corresponding to the part of the feature of the image to be processed whose similarity with the feature of the source domain reaches a second threshold.

16. The device according to claim 15, characterized in that The loss function corresponding to the first sub-model is the cross entropy loss function; The loss function corresponding to the second sub-model is a cross entropy loss function and a steady-state regularization term, and the steady-state regularization term is used to indicate: if the domain to which the training data belongs changes during the training process, adjust the parameters in the preprocessing model whose change range exceeds the third threshold.

17. A computing device, characterized in that include: Processor and memory; The memory is used to store computer instructions. When the processor executes the instructions, the computing device executes the image recognition method as described in any one of claims 1 to 5, or executes the model training method for image recognition as described in any one of claims 6 to 8.

18. An image recognition system, characterized in that: include: A training device and a terminal, wherein the terminal is used to execute the image recognition method as described in any one of claims 1 to 5; and the training device is used to execute the model training method for image recognition as described in any one of claims 6 to 8.

19. A computer-readable storage medium, characterized in that: It includes computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the image recognition method as described in any one of claims 1 to 5, or executes the model training method for image recognition as described in any one of claims 6 to 8.

Images