Training Method of Image Recognition Model, Image Recognition Method, Device and Equipment

Abstract

The present disclosure provides a training method, an image recognition method, an apparatus and a device for an image recognition model, relating to the field of artificial intelligence technologies, specifically to the fields of deep learning, image processing, and computer vision technologies. The method includes: grayscaling a first sample image in an auxiliary domain to obtain a first auxiliary image; inputting the first auxiliary image into an auxiliary model, and performing color restoration on the first auxiliary image through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain a second auxiliary image; pre-training the auxiliary encoder and the auxiliary decoder according to the first auxiliary image and the second auxiliary image; fine-tuning a target model by using a second sample image in a target domain, and using the fine-tuned target model as an image recognition model in the target domain; and initializing a target encoder in the target model by using the pre-trained auxiliary encoder. Through the above technical solution, the image recognition effect can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the fields of deep learning, image processing, and computer vision technology. Specifically, it relates to a method for training an image recognition model, an image recognition method, a device, and a device. Background Art

[0002] With the rapid development of deep learning (DL) technology, deep learning has been widely applied in fields such as computer vision technology, speech recognition technology, natural language processing technology, deep learning, and big data processing technology.

[0003] The Mask Auto Encoder (MAE) model has achieved great success in many natural image recognition tasks. However, when it is transferred to image recognition tasks in some fields, affected by the small amount of data, the recognition effect of images is not ideal. Summary of the Invention

[0004] The present disclosure provides a method for training an image recognition model, an image recognition method, a device, and a device.

[0005] According to one aspect of the present disclosure, there is provided a method for training an image recognition model, including:

[0006] Grayscale the first sample image in the auxiliary field to obtain a first auxiliary image;

[0007] Input the first auxiliary image into an auxiliary model, and perform color restoration on the first auxiliary image through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain a second auxiliary image;

[0008] Pre-train the auxiliary encoder and the auxiliary decoder according to the first auxiliary image and the second auxiliary image;

[0009] Fine-tune the target model with the second sample image in the target field, and use the fine-tuned target model as the image recognition model in the target field; the target encoder in the target model is initialized with the pre-trained auxiliary encoder.

[0010] According to another aspect of the present disclosure, there is provided an image recognition method, including:

[0011] Obtain a target image to be recognized in the target field;

[0012] Input the target image into the image recognition model in the target field to obtain the recognition result of the target image;

[0013] Among them, the image recognition model in the target domain is trained by using the training method of the image recognition model disclosed in any embodiment of the present disclosure.

[0014] According to another aspect of the present disclosure, there is provided a training device for an image recognition model, including:

[0015] A grayscale module, configured to perform grayscaling on a first sample image in an auxiliary domain to obtain a first auxiliary image;

[0016] A color restoration module, configured to input the first auxiliary image into an auxiliary model, and perform color restoration on the first auxiliary image through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain a second auxiliary image;

[0017] A pre-training module, configured to pre-train the auxiliary encoder and the auxiliary decoder according to the first auxiliary image and the second auxiliary image;

[0018] A model fine-tuning module, configured to fine-tune a target model by using a second sample image in a target domain, and use the fine-tuned target model as an image recognition model in the target domain; the target encoder in the target model is initialized by using the pre-trained auxiliary encoder.

[0019] According to another aspect of the present disclosure, there is provided an image recognition device, including:

[0020] A target image module, configured to obtain a target image to be recognized in a target domain;

[0021] An image recognition module, configured to input the target image into an image recognition model in the target domain to obtain a recognition result of the target image;

[0022] Among them, the image recognition model in the target domain is trained by the training device of the image recognition model disclosed in any embodiment of the present disclosure.

[0023] According to another aspect of the present disclosure, there is provided an electronic device, which includes:

[0024] At least one processor; and

[0025] A memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in any embodiment of the present disclosure.

[0027] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method provided in any embodiment of the present disclosure.

[0028] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0030] Figure 1a is a flowchart of a method for training an image recognition model provided according to an embodiment of the present disclosure;

[0031] Figure 1b is a schematic diagram of the training principle of an image recognition model provided according to an embodiment of the present disclosure;

[0032] Figure 2 is a flowchart of another method for training an image recognition model provided according to an embodiment of the present disclosure;

[0033] Figure 3a is a flowchart of yet another method for training an image recognition model provided according to an embodiment of the present disclosure;

[0034] Figure 3b is a schematic diagram of the training process of an auxiliary model and a target model provided according to an embodiment of the present disclosure;

[0035] Figure 4 is a flowchart of an image recognition method provided according to an embodiment of the present disclosure;

[0036] Figure 5 is a schematic structural diagram of a device for training an image recognition model provided according to an embodiment of the present disclosure;

[0037] Figure 6 is a schematic structural diagram of an image recognition device provided according to an embodiment of the present disclosure;

[0038] Figure 7 is a block diagram of an electronic device for implementing the method for training an image recognition model or the image recognition method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] Figure 1aIt is a flowchart of a method for training an image recognition model according to an embodiment of the present disclosure. This method is applicable to training an image recognition model in a target field. This method can be executed by a training device of the image recognition model, and the device can be implemented in a software and / or hardware manner and integrated into an electronic device. As Figure 1a shown, the method for training the image recognition model in this embodiment may include:

[0040] S101, grayscale the first sample images in the auxiliary field to obtain first auxiliary images;

[0041] S102, input the first auxiliary images into an auxiliary model, and perform color restoration on the first auxiliary images through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain second auxiliary images;

[0042] S103, pre-train the auxiliary encoder and the auxiliary decoder according to the first auxiliary images and the second auxiliary images;

[0043] S104, fine-tune the target model using the second sample images in the target field, and use the fine-tuned target model as the image recognition model in the target field.

[0044] In the embodiments of the present disclosure, the target field is the field that needs to perform image recognition. However, the number of labeled samples in the target field is very limited. If directly using the labeled samples in the target field for training, it is easy to cause overfitting, resulting in a very poor generalization of the trained model. The auxiliary field is other fields except the target field, that is, the auxiliary field is different from the target field. The auxiliary field has rich labeled samples. The auxiliary field can use an image dataset of natural scenes, such as the visualization dataset ImageNet. The ImageNet dataset has 1,281,167 images and labels, a total of 1000 categories, approximately 1300 images per category, the validation set has 50,000 images, 50 data per category, and the test set has 100,000 images, 100 data per category, which can provide sufficient first sample images.

[0045] Among them, the auxiliary model is a deep learning model in the auxiliary field, and the target model is a deep learning model in the target field. The auxiliary encoder in the auxiliary model and the target encoder in the target field can both be constructed based on the Mask Auto Encoder (MAE) structure. The MAE model has achieved great success in many natural image recognition tasks, but directly migrating it to the image recognition tasks in the target field has unsatisfactory effects. The embodiments of the present disclosure propose a brand-new MAE pre-training task. Through grayscale image color restoration (Colorization), the encoder can utilize the existing large natural scene dataset to learn features more suitable for the images in the target field, thereby improving the image recognition effect in the target field.

[0046] Figure 1b is a schematic diagram of the training principle of an image recognition model provided according to the embodiments of the present disclosure. Refer to Figure 1b , the auxiliary model is constructed based on the MAE structure, including an auxiliary encoder and an auxiliary decoder. The auxiliary encoder can adopt the Encoder structure in ViT (Vision Transformer), and the auxiliary decoder can adopt the MAE decoder structure; the target model includes a target encoder and a target output layer, and the target encoder has the same structure as the auxiliary decoder. The structure of the target output layer is determined by the type of the target recognition task, and the target recognition task can be an image classification task or an image segmentation task. The first sample image is an RGB three-channel image in the auxiliary field, and the second sample image can be a grayscale image in the target field, such as a CT image in the medical field, etc. The number of the first sample images is greater than the number of the second sample images.

[0047] Refer to Figure 1b , in the pre-training stage, the first sample image of the auxiliary type is grayscaled to obtain the first auxiliary image, and the first auxiliary image is input into the auxiliary model. The first auxiliary image is color-restored through the auxiliary encoder and the auxiliary decoder in the auxiliary model to obtain the second auxiliary image; the auxiliary encoder and the auxiliary decoder are pre-trained according to the first auxiliary image and the second auxiliary image to obtain the pre-trained auxiliary encoder and auxiliary decoder. In the pre-training stage, through the color restoration task, the auxiliary encoder fully learns the image semantic features in the auxiliary field, thereby learning representations with good generalization performance. By migrating the auxiliary encoder to the target field, the target encoder also has good generalization and can learn the image features in the target field with high quality, thereby improving the image recognition performance in the target field.

[0048] In the fine-tuning stage, a pre-trained auxiliary encoder is used as the initial target encoder, that is, the network parameters in the target encoder are initialized with the network parameters in the pre-trained auxiliary encoder, and the second sample image in the target domain is input into the target model. The second sample image is subjected to image recognition through the target encoder and the target output layer in the target model, and the target encoder and the target output layer are fine-tuned according to the image recognition result. The fine-tuned target model is used as the image recognition model in the target domain for recognizing images in the target domain.

[0049] Through the color restoration task proposed based on the MAE model structure, the auxiliary encoder can better learn image semantic features in the pre-training stage. By using the pre-trained auxiliary encoder to initialize the target encoder in the target model, the target encoder can also learn the image features in the target domain with high quality, thereby improving the image recognition performance in the target domain.

[0050] The technical solution provided in the embodiments of the present disclosure obtains a second auxiliary image by performing color restoration on the first auxiliary image through an auxiliary model, and initializes the target encoder in the target model with a pre-trained auxiliary encoder, so that the target encoder can learn more suitable image features for the target domain by using the dataset in the auxiliary domain, thereby improving the image recognition performance in the target domain.

[0051] In an alternative embodiment, the target domain is the medical field; the target model is used for processing images in the medical field.

[0052] When the target domain is the medical field, the image in the target domain can be a grayscale image such as a CT image. Through the color restoration process of the grayscale image in the pre-training stage, the model can be more easily migrated to the target domain and achieve better recognition results.

[0053] Figure 2 It is a flowchart of another method for training an image recognition model provided according to an embodiment of the present disclosure. Refer to Figure 2 , the method for training the image recognition model in this embodiment may include:

[0054] S201, grayscale the first sample image in the auxiliary domain to obtain a first auxiliary image;

[0055] S202, perform mask occlusion on the first auxiliary image to obtain an occluded first auxiliary image;

[0056] S203, input the occluded first auxiliary image into the auxiliary model, and perform color restoration and occlusion position reconstruction on the first auxiliary image through the auxiliary encoder and the auxiliary decoder in the auxiliary model to obtain a second auxiliary image;

[0057] S204. Pre-train the auxiliary encoder and the auxiliary decoder according to the first auxiliary image and the second auxiliary image.

[0058] S205. Fine-tune the target model using the second sample images in the target domain, and use the fine-tuned target model as the image recognition model for the target domain; the target encoder in the target model is initialized with the pre-trained auxiliary encoder.

[0059] In the pre-training stage, not only is the first sample image in the auxiliary domain grayscaled to obtain the first auxiliary image, but also the first auxiliary image is masked and occluded to obtain the occluded first auxiliary image. The occluded first auxiliary image is input into the auxiliary model. Through the auxiliary model, not only is the color of the first auxiliary image restored, but also the occluded positions are reconstructed based on the unoccluded positions to obtain the second auxiliary image. The auxiliary model not only performs the color restoration task but also the MIM (Mask Image Modeling) task, which can further improve the image feature extraction ability of the encoder.

[0060] The embodiments of the present disclosure do not specifically limit the grayscaling method of the first sample image. For example, grayscale transformation can be used: Y = 0.2126×R + 0.7152×G + 0.0722×B, where RGB respectively represent the values of the red, green, and blue color channels in the first sample image, and Y is the grayscale value of a single channel. Before grayscaling the first sample image, the first sample image can also be randomly scaled and cropped to a resolution of 224×224. Both the second auxiliary image and the first sample image are RGB three-channel color images.

[0061] Specifically, in the pre-training stage, the first sample image can be processed such as scaled and cropped, and then grayscaled using the grayscale transformation: Y = 0.2126×R + 0.7152×G + 0.0722×B to obtain the first auxiliary image, and a mask is used to occlude some regions in the first auxiliary image to obtain the occluded first auxiliary image. The occluded first auxiliary image is input into the auxiliary model, and the auxiliary encoder and the auxiliary decoder in the auxiliary model perform the color restoration task and the MIM task to obtain the second auxiliary image. By taking into account the color restoration task and the MIM task, the auxiliary encoder can better learn image features, thereby further improving the image recognition performance transferred to the target domain.

[0062] In an alternative embodiment, the step of masking and occluding the first auxiliary image to obtain the occluded first auxiliary image includes: using a preset occlusion ratio to mask and occlude the first auxiliary image to obtain the occluded first auxiliary image; the occlusion ratio is less than the occlusion ratio threshold.

[0063] Among them, the preset occlusion ratio and the occlusion ratio threshold can both be empirical values. For example, the occlusion ratio threshold can be 75%, and the preset occlusion ratio can be 15%. Since the color restoration task is equivalent to classifying each pixel point into 256×256×256 categories, the computational complexity is high. By controlling the MIM task to adopt a lower occlusion ratio, compared with the high occlusion ratio using the occlusion ratio threshold, it can make the auxiliary model converge better.

[0064] In an alternative embodiment, the pre-training of the auxiliary encoder and the auxiliary decoder according to the first auxiliary image and the second auxiliary image includes: determining a pre-training loss function according to the pixel value of each pixel point in the first auxiliary image and the pixel value of the corresponding pixel point in the second auxiliary image; using the pre-training loss function to pre-train the auxiliary encoder and the auxiliary decoder.

[0065] The loss function in the pre-training stage can adopt the pixel-level mean square error (MSE) loss, also known as the L2 loss. Specifically, for each pixel point I1 in the first auxiliary image, the corresponding pixel point I2 in the second auxiliary image can be determined, and the pre-training loss value can be obtained according to the Euclidean distance between the pixel value of the pixel point I1 in the first auxiliary image and the pixel value of the corresponding pixel point I2 in the second auxiliary image, and the network parameters in the auxiliary encoder and the auxiliary decoder are updated using the pre-training loss value. In the pre-training stage, through the basic L2 loss function of pixels, the update of the network parameters in the auxiliary encoder and the auxiliary decoder is realized, so that the auxiliary encoder has good image feature extraction ability. It should be noted that the trained auxiliary model can also be used for the color restoration function of black and white pictures.

[0066] The technical solution provided by the embodiments of the present disclosure, in the pre-training stage, the auxiliary model introduces the MIM task in addition to the color restoration task, and controls the mask occlusion ratio in the MIM task, which can not only further improve the image feature extraction ability, but also make the auxiliary model converge better, thereby further improving the image recognition performance of the auxiliary model migrated to the target domain.

[0067] Figure 3a It is a flowchart of another method for training an image recognition model according to an embodiment of the present disclosure. This embodiment is an alternative solution based on the above embodiment. See Figure 3a , the method for training the image recognition model of this embodiment may include:

[0068] S301, grayscale the first sample image in the auxiliary domain to obtain the first auxiliary image;

[0069] S302. Input the first auxiliary image into the auxiliary model, and perform color restoration on the first auxiliary image through the auxiliary encoder and the auxiliary decoder in the auxiliary model to obtain a second auxiliary image;

[0070] S303. Pre-train the auxiliary encoder and the auxiliary decoder according to the first auxiliary image and the second auxiliary image;

[0071] S304. Input the second sample image in the target domain into the target encoder in the target model for feature extraction to obtain a second target feature;

[0072] S305. Input the second target feature into the target output layer in the target model to obtain prediction information of the second sample image;

[0073] S306. Fine-tune the target encoder and the target output layer by using the annotation information and the prediction information of the second sample image, and use the fine-tuned target model as the image recognition model in the target domain.

[0074] The target model includes a target encoder and a target output layer, and the target encoder is initialized with the pre-trained auxiliary encoder. The second sample image has annotation information (ground truth), and the category of the annotation information is determined by the image recognition task in the target domain. In the case where the image recognition task is an image segmentation task, the annotation information of the second sample image is the image segmentation result annotated by the second sample image; in the case where the image recognition task is an image classification task, the annotation information of the second sample image is the image classification result annotated by the second sample image.

[0075] In the fine-tuning stage, input the second sample image in the target domain into the target encoder, and perform feature extraction on the second sample image through the target encoder to obtain a second target feature; and input the second target feature into the target output layer for image recognition to obtain prediction information of the second sample image; update the network parameters in the target encoder and the target output layer according to the prediction information and the annotation information of the second sample image, for example, update according to the Logits (logarithmic correction value) score between the prediction information and the annotation information of the second sample image, and use the fine-tuned target model as the image recognition model in the target domain.

[0076] In the fine-tuning stage, perform image recognition on the second sample image through the target model to obtain prediction information of the second sample image, and fine-tune the target model according to the annotation information and the prediction information of the second sample image to obtain the image recognition model in the target domain. Since the target encoder in the target model is initialized with the pre-trained auxiliary encoder, the ability of image feature extraction can be learned by borrowing the first sample image in the auxiliary domain, reducing the dependence on the labeled data in the target domain.

[0077] In an alternative embodiment, the fine-tuning of the target encoder and the target output layer by using the annotation information of the second sample image and the prediction information includes: determining a cross-entropy loss according to the difference between the annotation information of the second sample image and the prediction information; for the target encoding parameters in the target encoder, determining the corresponding auxiliary encoding parameters of the target encoding parameters in the auxiliary encoder, and determining a regularization loss according to the target encoding parameters and the auxiliary encoding parameters; and fine-tuning the target encoder and the target output layer according to the cross-entropy loss and the regularization loss.

[0078] In the fine-tuning stage, not only the cross-entropy loss is considered, but also the regularization loss between the target encoding parameters and the auxiliary encoding parameters can be introduced. By introducing the regularization loss, the update of the target encoder is based on the parameters of the auxiliary encoder, so that the target encoder can reduce overfitting in the fine-tuning stage.

[0079] Specifically, for the target encoding parameters in the target encoder, according to the initialization correspondence between the target encoder and the auxiliary encoder, the corresponding auxiliary encoding parameters of the target encoding parameters in the auxiliary encoder are determined, and the regularization loss is determined according to the target encoding parameters and the corresponding auxiliary encoding parameters; the network parameters in the target encoder and the target output layer are fine-tuned by combining the cross-entropy loss and the regularization loss.

[0080] In an alternative embodiment, the target encoder uses the encoder in the vision transformer; when the target model is for an image segmentation task, the target output layer is a decoder for semantic segmentation; when the target model is for an image classification task, the target output layer is a linear classifier.

[0081] In the embodiments of the present disclosure, the target encoder and the auxiliary encoder have the same structure, and both can use the ViT encoder in MAE. The target output layer discards the lightweight MAE decoder network. Refer to Figure 3b , when the target model is for an image segmentation task, the target output layer can be a decoder for semantic segmentation. For example, the Upernet decoder is used for 2D image segmentation, or the UNETR decoder is used for 3D image segmentation.

[0082] Refer to Figure 3b, in the case where the target model is for an image segmentation task, the target output layer can be a linear layer. The output corresponding to the learnable class CLS in the target encoder can be used as the representative of the entire image. An external linear layer is connected, and through the linear layer, the output is mapped into logits scores with the same number as the number of classes. Since many data in medical image classification are multi-label, that is, an image has multiple class labels, the Sigmoid function can be used to activate it, and the loss function can be used to optimize the entire network of the target model. By adapting the network structure for the target output layer according to the image recognition task, image recognition in the target domain can be achieved.

[0083] In the technical solution provided by the embodiments of the present disclosure, in the fine-tuning stage, by using a pre-trained auxiliary encoder to initialize the target encoder, the image feature extraction ability can be learned from the first sample images in the auxiliary domain; by fine-tuning the network parameters in the target encoder and the target output layer according to the cross-entropy loss and the regularization loss, the target encoder can avoid overfitting in the fine-tuning stage; and by providing an adapted network structure for the target output layer for the target recognition task, image recognition in the target domain can be achieved.

[0084] Figure 4 is a flowchart of an image recognition method provided by an embodiment of the present disclosure. This method is applicable to performing an image recognition task in a target domain. This method can be executed by an image recognition device, and the device can be implemented in a software and / or hardware manner and can be integrated into an electronic device. As Figure 4 shown, the image recognition method of this embodiment can include:

[0085] S401, obtain a target image to be recognized in the target domain;

[0086] S402, input the target image into the image recognition model in the target domain to obtain the recognition result of the target image;

[0087] Among them, the image recognition model in the target domain is trained by using the training method of the image recognition model disclosed in any embodiment of the present disclosure.

[0088] The target domain is the domain that needs to perform image recognition, but the number of labeled samples in the target domain is very limited. The target domain can be the medical image domain. The cost of medical image annotation is high, and generally the size of the dataset is limited. In the case where the target image is a grayscale image, such as a CT image, the target image can be directly input into the image recognition model in the target domain for image recognition to obtain the recognition result of the target image. In the case where the target image is a color image, it can be grayscaled first and then input into the image recognition model.

[0089] Since the image recognition model in the target domain provided by the embodiments of the present disclosure is constructed by using the training method of the image recognition model provided in any embodiment of the present disclosure, it has good image recognition performance.

[0090] The technical solution provided by the embodiments of the present disclosure is pre-trained by adding a MAE model with color restoration, and the encoder in the target domain is initialized by using the pre-trained auxiliary encoder, so that the target encoder can better learn the image features of the target domain, and thus has good image recognition performance for the target domain.

[0091] Figure 5 It is a schematic structural diagram of a training device for an image recognition model provided by an embodiment of the present disclosure. This embodiment is applicable to training an image recognition model in a target domain. The device can be implemented in a software and / or hardware manner. As Figure 5 shown, the training device 500 for the image recognition model in this embodiment may include:

[0092] A grayscale module 510, configured to grayscale the first sample image in the auxiliary domain to obtain a first auxiliary image;

[0093] A color restoration module 520, configured to input the first auxiliary image into an auxiliary model, and perform color restoration on the first auxiliary image through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain a second auxiliary image;

[0094] A pre-training module 530, configured to pre-train the auxiliary encoder and the auxiliary decoder according to the first auxiliary image and the second auxiliary image;

[0095] A model fine-tuning module 540, configured to fine-tune a target model by using a second sample image in a target domain, and use the fine-tuned target model as an image recognition model in the target domain; the target encoder in the target model is initialized by using the pre-trained auxiliary encoder.

[0096] In an alternative embodiment, the color restoration module 520 includes:

[0097] A masking unit, configured to perform masking occlusion on the first auxiliary image to obtain an occluded first auxiliary image;

[0098] An auxiliary model unit, configured to input the occluded first auxiliary image into an auxiliary model, and perform color restoration and occlusion position reconstruction on the first auxiliary image through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain a second auxiliary image.

[0099] In an alternative embodiment, the masking unit is specifically configured to:

[0100] Using a preset occlusion ratio, perform mask occlusion on the first auxiliary image to obtain an occluded first auxiliary image; the occlusion ratio is less than the occlusion ratio threshold.

[0101] In an alternative embodiment, the pre-training module 530 includes:

[0102] A training loss unit for determining a pre-training loss function based on the pixel values of each pixel point in the first auxiliary image and the pixel values of the corresponding pixel points in the second auxiliary image;

[0103] A pre-training unit for pre-training the auxiliary encoder and the auxiliary decoder using the pre-training loss function.

[0104] In an alternative embodiment, the model fine-tuning module 540 includes:

[0105] A feature extraction unit for inputting a second sample image in the target domain into a target encoder in the target model to perform feature extraction and obtain second target features;

[0106] A prediction information unit for inputting the second target features into a target output layer in the target model to obtain prediction information of the second sample image;

[0107] A model fine-tuning unit for fine-tuning the target encoder and the target output layer using the annotation information and the prediction information of the second sample image.

[0108] In an alternative embodiment, the model fine-tuning unit includes:

[0109] A cross-entropy loss sub-unit for determining cross-entropy loss based on the difference between the annotation information and the prediction information of the second sample image;

[0110] A regularization loss sub-unit for determining the corresponding auxiliary coding parameters of the target coding parameters in the auxiliary encoder for the target coding parameters in the target encoder, and determining regularization loss based on the target coding parameters and the auxiliary coding parameters;

[0111] A model fine-tuning sub-unit for fine-tuning the target encoder and the target output layer based on the cross-entropy loss and the regularization loss.

[0112] In an alternative embodiment, the target encoder uses the encoder in a vision transformer;

[0113] In the case where the target model is an image segmentation task, the target output layer is a decoder for semantic segmentation; in the case where the target model is an image classification task, the target output layer is a linear classifier.

[0114] In an alternative embodiment, the target field is the medical field; the target model is used to process images in the medical field.

[0115] In the technical solution of the embodiments of the present disclosure, the auxiliary model performs color restoration on the first auxiliary image to obtain the second auxiliary image, enabling the auxiliary encoder in the auxiliary model to learn image semantic features with high quality; and, by using the pre-trained auxiliary encoder to initialize the target encoder in the target model and fine-tuning the target model with the second sample images in the target field, the image recognition performance in the target field can be improved.

[0116] Figure 6 It is a schematic structural diagram of an image recognition device provided according to an embodiment of the present disclosure. This embodiment is applicable to performing image recognition tasks in the target field. The device can be implemented in software and / or hardware. As Figure 6 shown, the image recognition device 600 of this embodiment may include:

[0117] A target image module 610, configured to obtain a target image to be recognized in the target field;

[0118] An image recognition module 620, configured to input the target image into an image recognition model in the target field to obtain a recognition result of the target image;

[0119] Wherein, the image recognition model in the target field is trained by a training device of the image recognition model provided in any embodiment of the present disclosure.

[0120] In the technical solution of the embodiments of the present disclosure, pre-training is performed through a MAE model with color restoration, and the target encoder in the target field is initialized by using the pre-trained auxiliary encoder, enabling the target encoder to better learn the image features in the target field, and thus having good image recognition performance in the target field.

[0121] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0122] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0123] Figure 7 It is a block diagram of an electronic device for implementing the training method or image recognition method of the image recognition model of the embodiments of the present disclosure.

[0124] Figure 7FIG. 0 shows a schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, for example, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementations of the present disclosure described and / or claimed herein.

[0125] As Figure 7 shown, the electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0126] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as, for example, a keyboard, a mouse, etc.; an output unit 707, such as, for example, various types of displays, speakers, etc.; a storage unit 708, such as, for example, a magnetic disk, an optical disk, etc.; and a communication unit 709, such as, for example, a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0127] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as the training method or the image recognition method of an image recognition model. For example, in some embodiments, the training method or the image recognition method of an image recognition model can be implemented as a computer software program, which is tangibly incorporated in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the training method or the image recognition method of the image recognition model described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to execute the training method or the image recognition method of the image recognition model in any other suitable way (e.g., by means of firmware).

[0128] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), systems-on-a-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, and the programmable processor can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or a controller of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the processor or the controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, executed partially on the machine and partially on a remote machine as an independent software package, or executed entirely on a remote machine or a server.

[0130] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] For providing interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0134] Artificial intelligence is a discipline that studies the simulation of certain human thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) by computers, and it has technologies at both the hardware and software levels. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include several major directions such as computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, and knowledge graph technology.

[0135] Cloud computing refers to a technical system that accesses an elastic and scalable shared physical or virtual resource pool through a network. The resources can include servers, operating systems, networks, software, applications, and storage devices, etc., and the resources can be deployed and managed in a on-demand and self-service manner. Through cloud computing technology, it can provide efficient and powerful data processing capabilities for the application and model training of technologies such as artificial intelligence and blockchain.

[0136] It should be understood that various forms of processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps recorded in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this is not limited herein.

[0137] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A training method for an image recognition model, comprising: Grayscale the first sample image in the auxiliary domain to obtain a first auxiliary image; Input the first auxiliary image into an auxiliary model, and perform color restoration on the first auxiliary image through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain a second auxiliary image; Determine a pre-training loss function according to the pixel value of each pixel point in the first auxiliary image and the pixel value of the corresponding pixel point in the second auxiliary image; Use the pre-training loss function to pre-train the auxiliary encoder and the auxiliary decoder; Input the second sample image in the target domain into a target encoder in the target model for feature extraction to obtain a second target feature; Input the second target feature into a target output layer in the target model to obtain prediction information of the second sample image; Use the annotation information and the prediction information of the second sample image to fine-tune the target encoder and the target output layer, and use the fine-tuned target model as an image recognition model in the target domain; The target encoder in the target model is initialized with the pre-trained auxiliary encoder.

2. The method according to claim 1, wherein, The step of inputting the first auxiliary image into the auxiliary model and performing color restoration on the first auxiliary image through the auxiliary encoder and the auxiliary decoder in the auxiliary model to obtain a second auxiliary image includes: Perform mask occlusion on the first auxiliary image to obtain an occluded first auxiliary image; Input the occluded first auxiliary image into the auxiliary model, and perform color restoration and occlusion position reconstruction on the first auxiliary image through the auxiliary encoder and the auxiliary decoder in the auxiliary model to obtain a second auxiliary image.

3. The method according to claim 2, wherein, The step of performing mask occlusion on the first auxiliary image to obtain an occluded first auxiliary image includes: Perform mask occlusion on the first auxiliary image with a preset occlusion ratio to obtain an occluded first auxiliary image; the occlusion ratio is less than an occlusion ratio threshold.

4. The method according to claim 1, wherein The step of using the annotation information and the prediction information of the second sample image to fine-tune the target encoder and the target output layer includes: Determine a cross-entropy loss according to the difference between the annotation information and the prediction information of the second sample image; For the target encoding parameters in the target encoder, determine the corresponding auxiliary encoding parameters in the auxiliary encoder, and determine a regularization loss according to the target encoding parameters and the auxiliary encoding parameters; Fine-tune the target encoder and the target output layer according to the cross-entropy loss and the regularization loss.

5. The method according to claim 1, wherein the target encoder uses an encoder in a vision transformer; When the target model is for an image segmentation task, the target output layer is a decoder for semantic segmentation; when the target model is for an image classification task, the target output layer is a linear classifier.

6. The method according to any one of claims 1 to 3, wherein The target domain is the medical field; the target model is used to process images in the medical field.

7. An image recognition method, comprising: Obtain a target image to be recognized in the target domain; Input the target image into an image recognition model in the target domain to obtain the recognition result of the target image; Among them, the image recognition model in the target domain is trained by using the method described in any one of claims 1-6.

8. An apparatus for training an image recognition model, comprising: A grayscale module, configured to grayscale a first sample image in an auxiliary domain to obtain a first auxiliary image; A color restoration module, configured to input the first auxiliary image into an auxiliary model, and perform color restoration on the first auxiliary image through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain a second auxiliary image; A pre-training module, where the pre-training module includes: A training loss unit, configured to determine a pre-training loss function according to the pixel value of each pixel point in the first auxiliary image and the pixel value of the corresponding pixel point in the second auxiliary image; A pre-training unit, configured to pre-train the auxiliary encoder and the auxiliary decoder by using the pre-training loss function; And a model fine-tuning module, where the model fine-tuning module includes: A feature extraction unit, configured to input a second sample image in the target domain into a target encoder in a target model to extract features and obtain second target features; A prediction information unit, configured to input the second target features into a target output layer in the target model to obtain prediction information of the second sample image; A model fine-tuning unit, configured to fine-tune the target encoder and the target output layer by using the annotation information and the prediction information of the second sample image, and use the fine-tuned target model as an image recognition model in the target domain; the target encoder in the target model is initialized by using the pre-trained auxiliary encoder.

9. The apparatus according to claim 8, wherein, The color restoration module includes: A mask unit, configured to perform mask occlusion on the first auxiliary image to obtain an occluded first auxiliary image; An auxiliary model unit, configured to input the occluded first auxiliary image into an auxiliary model, and perform color restoration and occlusion position reconstruction on the first auxiliary image through an auxiliary encoder and an auxiliary decoder in the auxiliary model to obtain a second auxiliary image.

10. The device according to claim 9, wherein, The mask unit is specifically configured to: Perform mask occlusion on the first auxiliary image by using a preset occlusion ratio to obtain an occluded first auxiliary image; the occlusion ratio is less than an occlusion ratio threshold.

11. The apparatus according to claim 8, wherein, The model fine-tuning unit includes: A cross-entropy loss sub-unit, configured to determine a cross-entropy loss according to the difference between the annotation information and the prediction information of the second sample image; A regularization loss sub-unit, configured to determine, for the target coding parameters in the target encoder, the corresponding auxiliary coding parameters in the auxiliary encoder, and determine a regularization loss according to the target coding parameters and the auxiliary coding parameters; A model fine-tuning sub-unit, configured to fine-tune the target encoder and the target output layer according to the cross-entropy loss and the regularization loss.

12. According to the apparatus described in claim 8, the target encoder adopts an encoder in a vision transformer; When the target model is for an image segmentation task, the target output layer is a decoder for semantic segmentation; when the target model is for an image classification task, the target output layer is a linear classifier.

13. The apparatus according to any one of claims 8 - 12, wherein, The target field is the medical field; the target model is used to process images in the medical field.

14. An image recognition device, comprising: A target image module, configured to obtain a target image to be recognized in the target field; An image recognition module, configured to input the target image into an image recognition model in the target field to obtain a recognition result of the target image; wherein, the image recognition model in the target field is trained by the device according to any one of claims 8-13.

15. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause a computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.

Images