Pre-training method and device of industrial image processing model and electronic equipment

Abstract

The invention discloses a pre-training method and device for an industrial image processing model and electronic equipment, and belongs to the technical field of image processing. The method comprises the steps of performing mask processing on an unmarked industrial image sample set to obtain a first sample set and a second sample set, obtaining a first industrial visual feature output by a first training branch, and obtaining a second industrial visual feature output by a second training branch; and based on the first industrial visual feature and the second industrial visual feature, carrying out comparative learning on the first training branch and the second training branch to obtain a target loss function between the first training branch and the second training branch, and updating model parameters of the first training branch and the second training branch, and generating pre-training model parameters of the industrial image processing model. According to the method, diversified visual features of industrial data can be fully captured, and the generalization ability and expression ability of the model are effectively improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a pre-training method, device and electronic equipment for an industrial image processing model. Background Art

[0002] With the development of computer vision technology, more and more image processing methods are applied in the industrial field, playing an important role in tasks such as automated quality inspection, anomaly detection and defect detection, ensuring the product quality of industrial production and improving production efficiency.

[0003] Existing pre-training methods for large visual models mainly include weakly supervised pre-training and self-supervised pre-training. Since large-scale industrial data lacks comprehensive data annotation and complete description, it is impossible to use the weakly supervised paradigm for pre-training. In addition, since industrial images often have simple backgrounds, the effect of self-supervised pre-training is not good.

[0004] Existing pre-training methods are unable to fully capture the diverse visual features of industrial data, limiting the generalization and expression capabilities of the model. Summary of the invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a pre-training method, device and electronic device for an industrial image processing model, which can fully capture the diverse visual features of industrial data and effectively improve the generalization and expression capabilities of the model.

[0006] In a first aspect, the present application provides a pre-training method for an industrial image processing model, the method comprising:

[0007] Performing mask processing on the unlabeled industrial image sample set to obtain a first sample set and a second sample set, wherein the mask processing processes corresponding to the first sample set and the second sample set are different;

[0008] Inputting the first sample set into a first training branch of an industrial image processing model to obtain a first industrial visual feature output by the first training branch, and inputting the second sample set into a second training branch of the industrial image processing model to obtain a second industrial visual feature output by the second training branch;

[0009] Based on the first industrial visual feature and the second industrial visual feature, performing comparative learning on the first training branch and the second training branch to obtain a target loss function between the first training branch and the second training branch;

[0010] Based on the target loss function, the model parameters of the first training branch and the second training branch are updated to generate pre-trained model parameters of the industrial image processing model.

[0011] According to the pre-training method of the industrial image processing model of the present application, a first sample set and a second sample set are obtained by performing mask processing on an unlabeled industrial image sample set, and the first sample set and the second sample set are respectively input into a first training branch and a second training branch of the industrial image processing model, and comparative learning is performed based on the first industrial visual features and the second industrial visual features to generate pre-trained model parameters of the industrial image processing model, which can fully capture the diverse visual features of industrial data and effectively improve the generalization and expression capabilities of the model.

[0012] According to an embodiment of the present application, performing mask processing on the unlabeled industrial image sample set to obtain the first sample set and the second sample set includes:

[0013] Performing random mask processing on the industrial image sample set to obtain the first sample set;

[0014] The industrial image sample set is subjected to random mask processing and non-mask processing to obtain the second sample set.

[0015] According to one embodiment of the present application, the first training branch includes a first encoder and a plurality of first linear layers connected to the first encoder, and the second training branch includes a second encoder and a plurality of second linear layers connected to the second encoder;

[0016] The step of inputting the first sample set into a first training branch of an industrial image processing model to obtain a first industrial visual feature output by the first training branch includes:

[0017] Inputting the first sample set into the first training branch, and processing the first encoder and the first linear layer to obtain a first category word and a first image block word;

[0018] The step of inputting the second sample set into the second training branch of the industrial image processing model to obtain the second industrial visual features output by the second training branch includes:

[0019] The second sample set is input into the second training branch, and is processed by the second encoder and the second linear layer to obtain a second category word-unit and a second image block word-unit.

[0020] According to an embodiment of the present application, the comparative learning of the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature includes:

[0021] Based on the first category word-gram and the second category word-gram, comparative learning is performed on the first training branch and the second training branch.

[0022] According to an embodiment of the present application, the comparative learning of the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature includes:

[0023] In a case where the mask processing process corresponding to the first sample set is random mask processing, the first image block word-unit is reconstructed based on the second image block word-unit corresponding to the second sample set whose mask processing process is random mask processing.

[0024] According to one embodiment of the present application, after obtaining the first industrial visual feature output by the first training branch and before obtaining the target loss function between the first training branch and the second training branch, the method further includes:

[0025] The first industrial visual feature is subjected to center sharpening or clustering processing.

[0026] According to one embodiment of the present application, the model parameters of the second training branch are updated based on the gradient values ​​of back propagation.

[0027] According to one embodiment of the present application, the model parameters of the first training branch are updated based on a sliding average of the model parameters of the second training branch.

[0028] In a second aspect, the present application provides a pre-training device for an industrial image processing model, the device comprising:

[0029] A first processing module is used to perform mask processing on the unlabeled industrial image sample set to obtain a first sample set and a second sample set, wherein the mask processing processes corresponding to the first sample set and the second sample set are different;

[0030] A second processing module is used to input the first sample set into a first training branch of an industrial image processing model to obtain a first industrial visual feature output by the first training branch, and input the second sample set into a second training branch of the industrial image processing model to obtain a second industrial visual feature output by the second training branch;

[0031] A third processing module, configured to perform comparative learning on the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature, so as to obtain a target loss function between the first training branch and the second training branch;

[0032] The fourth processing module is used to update the model parameters of the first training branch and the second training branch based on the target loss function, and generate pre-trained model parameters of the industrial image processing model.

[0033] According to the pre-training device of the industrial image processing model of the present application, a first sample set and a second sample set are obtained by performing mask processing on an unlabeled industrial image sample set, and the first sample set and the second sample set are respectively input into a first training branch and a second training branch of the industrial image processing model, and comparative learning is performed based on the first industrial visual features and the second industrial visual features to generate pre-trained model parameters of the industrial image processing model, which can fully capture the diverse visual features of industrial data and effectively improve the generalization and expression capabilities of the model.

[0034] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the pre-training method for the industrial image processing model as described in the first aspect above is implemented.

[0035] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pre-training method for the industrial image processing model as described in the first aspect above.

[0036] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the pre-training method of the industrial image processing model as described in the first aspect above.

[0037] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0039] Figure 1 It is a flowchart of a pre-training method for an industrial image processing model provided in an embodiment of the present application;

[0040] Figure 2 is a structural diagram of an industrial image processing model provided in an embodiment of the present application;

[0041] Figure 3 It is a structural schematic diagram of a pre-training device for an industrial image processing model provided in an embodiment of the present application;

[0042] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0044] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0045] In related technologies, there are two ways to pre-train large visual models:

[0046] First, weakly supervised pre-training.

[0047] There are two main types of weakly supervised pre-training: one is to use a large-scale dataset with noisy category labels for pre-training similar to the supervised paradigm; the other is to use a large-scale dataset with text-image pairs for text-image alignment.

[0048] Both of the above weakly supervised pre-training methods require the provision of weak annotation information corresponding to the image, either noisy category annotations or corresponding text descriptions.

[0049] Second, self-supervised pre-training.

[0050] There are three main types of self-supervised pre-training: the first is autoregressive prediction, which treats the image as a sequence to predict the next word. The task shows good scalability in natural language processing (NLP), but images are not natural semantic sequences and do not show the same advantages as NLP in visual processing; the second is to randomly mask the input image and then reconstruct the high-level semantic features of the masked part; the third is a method based on contrastive learning, which shortens the feature distance between different forms in the same image and increases the feature distance between different images.

[0051] Since large-scale industrial data lacks comprehensive data annotation and complete description, it is impossible to use weakly supervised paradigm for pre-training. In addition, since industrial images often have simple backgrounds, mask images will reduce the difficulty of proxy tasks, lack high-quality reconstruction targets, and the effect of self-supervised pre-training is not good.

[0052] Existing pre-training methods usually rely on a single objective function (such as mask image modeling or contrastive learning), which makes it difficult to fully capture diverse visual features, resulting in limited generalization and expression capabilities of the model. At present, these two methods have their own advantages and disadvantages, but a single application often cannot take into account both global structure and local discriminative features.

[0053] Existing pre-training methods are unable to fully capture the diverse visual features of industrial data, limiting the generalization and expression capabilities of the model.

[0054] The embodiment of the present application provides a pre-training method for an industrial image processing model, which can fully capture the diverse visual features of industrial data and effectively improve the generalization and expression capabilities of the model.

[0055] In the following, in combination with the accompanying drawings, the pre-training method of the industrial image processing model, the pre-training device of the industrial image processing model, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0056] The pre-training method of the industrial image processing model provided in the embodiment of the present application, the execution subject of the pre-training method of the industrial image processing model can be an electronic device or a functional module or functional entity in the electronic device that can implement the pre-training method of the industrial image processing model. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras and wearable devices, etc. The pre-training method of the industrial image processing model provided in the embodiment of the present application is explained below using the electronic device as an execution subject as an example.

[0057] Among them, the industrial image processing model refers to a model that uses computer vision technology to analyze and process images generated in the industrial production process. The industrial image processing model can be used to perform tasks such as automated quality inspection, anomaly detection, and defect detection, to realize the automation of the industrial production process, improve production efficiency, and ensure production quality.

[0058] like Figure 1 As shown, the pre-training method of the industrial image processing model includes: step 110, step 120 and step 130.

[0059] Step 110: mask the unlabeled industrial image sample set to obtain a first sample set and a second sample set.

[0060] The mask processing processes corresponding to the first sample set and the second sample set are different.

[0061] The image data generated in the industrial production process is obtained without marking or classifying these image data, and an unlabeled industrial image sample set is obtained.

[0062] In this step, mask processing is performed on the unlabeled industrial image sample set through different mask processing processes to obtain a first sample set and a second sample set. The mask processing processes are different, and the image data of the first sample set is different from the image data of the second sample set.

[0063] For example, a first mask processing is performed on an unlabeled industrial image sample set to obtain a first sample set, and a second mask processing is performed on the unlabeled industrial image sample set to obtain a second sample set. The first mask processing is different from the second mask processing, so that the image data of the first sample set is different from the image data of the second sample set.

[0064] It can be understood that the first sample set and the second sample set are obtained by performing mask processing on the unlabeled industrial image sample set, and the first sample set and the second sample set are also unlabeled.

[0065] Step 120: input the first sample set into the first training branch of the industrial image processing model to obtain the first industrial visual features output by the first training branch, and input the second sample set into the second training branch of the industrial image processing model to obtain the second industrial visual features output by the second training branch.

[0066] In this embodiment, the industrial image processing model includes a first training branch and a second training branch. The two training branches may include similar network structures so that the first training branch and the second training branch can perform comparative learning.

[0067] In actual implementation, the first training branch can be regarded as a teacher branch, and the second training branch can be regarded as a student branch, which is applicable to the unlabeled first sample set and the second sample set.

[0068] In this step, the first training branch of the industrial image processing model processes the first sample set to obtain the first industrial visual features corresponding to the first sample set, and the second training branch of the industrial image processing model processes the second sample set to obtain the second industrial visual features corresponding to the second sample set.

[0069] It can be understood that industrial visual features are extracted from industrial images and can characterize the quantitative features of image content, such as shape features, key point features, semantic features, texture features, etc. Industrial visual features can be used for visual tasks such as quality inspection, anomaly detection and defect detection.

[0070] It should be noted that the mask processing processes corresponding to the first sample set and the second sample set are different. The first training branch processes the first sample set to obtain the first industrial visual features, and the second training branch processes the second sample set to obtain the second industrial visual features.

[0071] In actual implementation, the industrial image processing model can use Transformer as the basic architecture, which has strong scalability. By scaling the depth, width, number of attention heads, etc. of the basic architecture, the industrial image processing model can be built relatively simply.

[0072] Step 130: Based on the first industrial visual feature and the second industrial visual feature, comparative learning is performed on the first training branch and the second training branch to obtain a target loss function between the first training branch and the second training branch.

[0073] The goal of contrastive learning is to learn an effective representation of the data such that similar samples are closer in the representation space and dissimilar samples are farther away.

[0074] In contrastive learning, positive sample pairs and negative sample pairs can be constructed. Positive sample pairs refer to similar samples, and negative sample pairs refer to dissimilar samples. Positive sample pairs are used to shorten the distance of feature representation, and negative sample pairs are used to extend the distance of feature representation.

[0075] In this embodiment, comparative learning is performed based on the first industrial visual feature and the second industrial visual feature, the similarity between the representations of the first industrial visual feature and the second industrial visual feature is calculated, and the target loss function between the first training branch and the second training branch is obtained. The target loss function is used to pre-train the industrial image processing model so that the feature representations of the positive sample pairs are closer, while the feature representations of the negative sample pairs are farther apart.

[0076] Step 140: Based on the target loss function, update the model parameters of the first training branch and the second training branch to generate pre-trained model parameters of the industrial image processing model.

[0077] It can be understood that the target loss function is used to measure the difference (i.e., loss) between the predicted value and the true value output by the industrial image processing model. Different types of target loss functions can be obtained according to the different types of tasks performed by the industrial image processing model, such as cross entropy loss, log-likelihood loss, etc.

[0078] In this step, according to the target loss function, the model parameters of the first training branch and the second training branch are updated, the process of acquiring industrial visual features is repeated, and the industrial image processing model is iteratively trained until the model performance is no longer significantly improved or the preset number of iterations is reached. The pre-training of the industrial image processing model is completed, the model parameters are saved, and the pre-trained model parameters of the industrial image processing model are obtained.

[0079] In an embodiment of the present application, the unlabeled industrial image sample set is masked through different mask processing processes to obtain a first sample set and a second sample set, so as to fully explore the diverse visual features of the industrial image sample set, and the first sample set and the second sample set are respectively input into the first training branch and the second training branch of the industrial image processing model, and comparative learning is performed based on the first industrial visual features and the second industrial visual features to generate pre-trained model parameters of the industrial image processing model, thereby effectively improving the generalization and expression capabilities of the model.

[0080] Based on the unlabeled industrial image sample set, the industrial image processing model is pre-trained. By combining contrastive learning and mask image modeling, the industrial data can be fully mined to obtain high-quality industrial visual features. It has excellent feature extraction and generalization capabilities for industrial scenarios, and alleviates the problem in related technologies that a single objective function cannot fully capture diverse image features, resulting in limited model representation capabilities. For visual tasks such as quality inspection, anomaly detection, and defect detection, the model detection capability can be effectively improved.

[0081] According to the pre-training method of the industrial image processing model provided in the embodiment of the present application, by performing mask processing on the unlabeled industrial image sample set, a first sample set and a second sample set are obtained, and the first sample set and the second sample set are respectively input into the first training branch and the second training branch of the industrial image processing model, and comparative learning is performed based on the first industrial visual features and the second industrial visual features to generate pre-trained model parameters of the industrial image processing model, which can fully capture the diverse visual features of industrial data and effectively improve the generalization and expression capabilities of the model.

[0082] In some embodiments, step 120, performing mask processing on the unlabeled industrial image sample set to obtain the first sample set and the second sample set, may include:

[0083] Performing random mask processing on the industrial image sample set to obtain a first sample set;

[0084] The industrial image sample set is subjected to random masking and non-masking processing to obtain a second sample set.

[0085] Stochastic Masks refers to randomly covering or shielding part of the image data. Through stochastic masks, industrial image processing models can learn global features and no longer simply rely on local information, which helps to improve the performance and generalization ability of the model.

[0086] Non-Masked is the complement of the mask operation, which refers to the data parts that are not covered by the random mask. The non-masked part can help the model identify which samples are similar and help contrastive learning to construct positive and negative sample pairs.

[0087] It should be noted that the non-masked part plays a bridging role when combining contrastive learning and masked image modeling. Through contrastive learning, the industrial image processing model can learn the relationship between the non-masked part and the masked part, improve the model's overall understanding of the image, and take into account both the global structure and local discriminant features.

[0088] In this embodiment, the industrial image sample set is randomly masked to obtain a first sample set, and the first sample set is input into a first training branch. The industrial image sample set is randomly masked and unmasked to obtain a second sample set, and the second sample set is input into a second training branch.

[0089] The following takes the first training branch as the teacher branch and the second training branch as the student branch as an example.

[0090] The unlabeled industrial image sample set is randomly masked to obtain the first sample set global crops; the industrial image sample set is randomly masked and unmasked to obtain the second sample set, which includes the randomly masked global crops and the unmasked local crops.

[0091] In this embodiment, the unlabeled industrial image sample set is subjected to operations such as masks of different sizes and numbers and data augmentation to obtain the input of the teacher branch and the student branch. The introduction of different mask processing can compare different image regions, so that the industrial image processing model can effectively capture the relationship between different image regions and enhance semantic understanding.

[0092] Among them, the input of the teacher branch is the first sample set global crops, and the input of the student branch is the second sample set, including global crops and local crops.

[0093] In some embodiments, the first training branch includes a first encoder and a plurality of first linear layers connected to the first encoder, and the second training branch includes a second encoder and a plurality of second linear layers connected to the second encoder.

[0094] In this embodiment, the first sample set is input into the first training branch of the industrial image processing model to obtain the first industrial visual feature output by the first training branch, including:

[0095] Inputting the first sample set into the first training branch, and processing it through the first encoder and the first linear layer to obtain a first category word and a first image block word;

[0096] Inputting the second sample set into the second training branch of the industrial image processing model to obtain the second industrial visual features output by the second training branch, including:

[0097] The second sample set is input into the second training branch, and is processed by the second encoder and the second linear layer to obtain a second category word unit and a second image block word unit.

[0098] It can be understood that the first training branch processes the first sample set, and the obtained first industrial visual features include first category words and first image block words, and the second training branch processes the second sample set, and the obtained second industrial visual features include second category words and second image block words.

[0099] It should be noted that the category token (cls token) can be used to represent the category of the input sequence, and the image block token (patch token) refers to mapping the masked image block to a vector representation, which can be regarded as an abstract representation of the image.

[0100] In this embodiment, the first encoder encodes the image blocks visible in the first sample set, extracts features, and inputs the extracted features into the subsequent multiple linear layers for processing to obtain first category words and first image block words; the second encoder encodes the image blocks visible in the second sample set, extracts features, and inputs the extracted features into the subsequent multiple linear layers for processing to obtain second category words and second image block words.

[0101] In actual implementation, the first training branch is also provided with multiple first activation layers, and the first linear layer and the first activation layer are used alternately. The second training branch is also provided with multiple second activation layers, and the second linear layer and the second activation layer are used alternately. The linear layer is responsible for performing the linear combination of weights and biases, and the activation layer introduces nonlinearity, which can capture complex patterns in the data.

[0102] In some embodiments, the industrial image sample set is randomly masked to obtain a first sample set, and the first sample set is input into a first training branch to obtain first category words and first image block words corresponding to the first sample set; the industrial image sample set is randomly masked and unmasked to obtain a second sample set, and the second sample set is input into a second training branch to obtain second category words and second image block words corresponding to the second sample set.

[0103] In some embodiments, based on the first industrial visual feature and the second industrial visual feature, performing comparative learning on the first training branch and the second training branch includes:

[0104] Based on the first category word-grams and the second category word-grams, comparative learning is performed on the first training branch and the second training branch.

[0105] In this embodiment, a first sample set and a second sample set are obtained by masking an unlabeled industrial image sample set, and first category words obtained by processing the first sample set and second category words obtained by processing the second sample set are used for comparative learning of the first training branch and the second training branch, to construct positive and negative sample pairs, and to compare the similarities or differences between the data samples to drive the industrial image processing model to learn good features or representations.

[0106] In some embodiments, based on the first industrial visual feature and the second industrial visual feature, performing comparative learning on the first training branch and the second training branch includes:

[0107] When the mask processing process corresponding to the first sample set is random mask processing, the first image block word-unit is reconstructed based on the second image block word-unit corresponding to the second sample set whose mask processing process is random mask processing.

[0108] It should be noted that the mask processing processes corresponding to the first sample set and the second sample set are different. The mask processing process corresponding to the first sample set is random mask processing, and the mask processing process corresponding to the second sample set includes random mask processing and at least one other mask processing, that is, the second sample set includes images processed by random mask and images processed by other mask.

[0109] The second image block word-unit corresponding to the second sample set subjected to random mask processing in the mask processing process refers to the second image block word-unit obtained by image processing subjected to random mask processing in the second sample set, and the second image block word-unit output by the second training branch, including the second image block word-unit corresponding to the image subjected to random mask processing and the second image block word-unit corresponding to other mask processing images.

[0110] In this embodiment, the first image block word-unit output by the first training branch is reconstructed based on the second image block word-unit corresponding to the randomly masked image output by the second training branch.

[0111] The following takes the first training branch as the teacher branch and the second training branch as the student branch as an example.

[0112] The unlabeled industrial image sample set is randomly masked to obtain the first sample set global crops1; the industrial image sample set is randomly masked and unmasked to obtain the second sample set, which includes the randomly masked global crops2 and the unmasked local crops2.

[0113] Among them, the input of the teacher branch is the first sample set global crops1, and the input of the student branch is the second sample set, including global crops2 and local crops2.

[0114] After the encoder of the teacher branch and multiple linear layers, the first category word cls token and the first image block word patch token corresponding to the first sample set are obtained. After the encoder of the student branch and multiple linear layers, the second category word cls token and the second image block word patch token corresponding to the second sample set are obtained.

[0115] The cls token corresponding to global crops1, global crops2 and local crops2 aggregates the global information of the image and can be used for comparative learning. The patch token corresponding to global crops1 and global crops2 can be used for mask image modeling and reconstruction. The reconstruction target is the patch token output by the teacher branch.

[0116] In some embodiments, after obtaining the first industrial visual feature output by the first training branch and before obtaining the target loss function between the first training branch and the second training branch, the pre-training method of the industrial image processing model may further include:

[0117] The first industrial visual feature is subjected to center sharpening or clustering processing.

[0118] In this embodiment, before calculating the target loss function, the first industrial visual feature output by the first training branch is subjected to center sharpening or clustering processing, which can prevent mode collapse during model training and help the industrial image processing model to fully capture all features of the data.

[0119] In some embodiments, the model parameters of the second training branch are updated based on the back-propagated gradient values.

[0120] In this embodiment, the gradient of the loss value of the target loss function with respect to the model parameters is calculated through the back-propagation algorithm, and the model parameters of the second training branch are updated according to the calculated gradient value. Back-propagation can effectively optimize the model parameters and improve the performance and generalization ability of the model.

[0121] In some embodiments, the model parameters of the first training branch are updated based on a sliding average of the model parameters of the second training branch.

[0122] In this embodiment, a sliding average of the model parameters of the second training branch is obtained, and the model parameters of the first training branch are updated according to the sliding average, which helps the model to converge more stably during the training process and improves the generalization ability of the model.

[0123] A specific embodiment is described below.

[0124] like Figure 2 As shown, the right side is the first training branch, and the left side is the second training branch, wherein the first training branch is the teacher branch, and the second training branch is the student branch.

[0125] The unlabeled industrial image sample set is randomly masked to obtain the first sample set global crops1; the industrial image sample set is randomly masked and unmasked to obtain the second sample set, which includes the randomly masked global crops2 and the unmasked local crops2.

[0126] Among them, the input of the teacher branch is the first sample set global crops1, and the input of the student branch is the second sample set, including global crops2 and local crops2.

[0127] In this embodiment, the student branch includes an encoder, three linear layers and two GeLU activation layers of the ViT large model, and the structure of the teacher branch is similar to that of the student branch, including an encoder, three linear layers and two GeLU activation layers.

[0128] Among them, the input of the teacher branch is the first sample set global crops1, and the input of the student branch is the second sample set, including global crops2 and local crops2.

[0129] After the encoder of the teacher branch and multiple linear layers, the first category word cls token and the first image block word patch token corresponding to the first sample set are obtained. After the encoder of the student branch and multiple linear layers, the second category word cls token and the second image block word patch token corresponding to the second sample set are obtained.

[0130] In actual implementation, the output of the teacher branch is center-sharpened or clustered to prevent mode collapse in industrial image processing models during pre-training.

[0131] The cls token corresponding to global crops1, global crops2 and local crops2 aggregates the global information of the image and can be used for comparative learning. The patch token corresponding to global crops1 and global crops2 can be used for mask image modeling and reconstruction. The reconstruction target is the patch token output by the teacher branch.

[0132] For the patch token, the cross entropy loss of the same crop is calculated, and for the cls token, the cross entropy loss of different crops is calculated. The parameters of the student branch are updated according to the gradient value of back propagation, and the parameters of the teacher branch are updated according to the sliding average of the parameters of the student branch.

[0133] In this embodiment, by combining contrastive learning and mask image modeling, the feature advantages of both are aggregated to obtain high-quality industrial visual features. The encoder of the ViT large model is used to carry out self-supervised learning on a large-scale and unlabeled industrial image sample set to complete the pre-training of the industrial image processing model. It has excellent feature extraction and generalization capabilities for industrial scenarios, and effectively improves the detection capabilities of the industrial image processing model to perform various tasks.

[0134] The pre-training method of the industrial image processing model provided in the embodiment of the present application can be performed by a pre-training device for the industrial image processing model. In the embodiment of the present application, the pre-training device for the industrial image processing model performs the pre-training method of the industrial image processing model as an example to illustrate the pre-training device for the industrial image processing model provided in the embodiment of the present application.

[0135] An embodiment of the present application also provides a pre-training device for an industrial image processing model.

[0136] like Figure 3 As shown, the pre-training device of the industrial image processing model includes:

[0137] A first processing module 310 is used to perform mask processing on the unlabeled industrial image sample set to obtain a first sample set and a second sample set, wherein the mask processing processes corresponding to the first sample set and the second sample set are different;

[0138] The second processing module 320 is used to input the first sample set into the first training branch of the industrial image processing model to obtain the first industrial visual features output by the first training branch, and input the second sample set into the second training branch of the industrial image processing model to obtain the second industrial visual features output by the second training branch;

[0139] A third processing module 330 is used to perform comparative learning on the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature to obtain a target loss function between the first training branch and the second training branch;

[0140] The fourth processing module 340 is used to update the model parameters of the first training branch and the second training branch based on the target loss function, and generate pre-trained model parameters of the industrial image processing model.

[0141] According to the pre-training device of the industrial image processing model provided in the embodiment of the present application, a first sample set and a second sample set are obtained by performing mask processing on an unlabeled industrial image sample set, and the first sample set and the second sample set are respectively input into a first training branch and a second training branch of the industrial image processing model, and comparative learning is performed based on the first industrial visual features and the second industrial visual features to generate pre-trained model parameters of the industrial image processing model, which can fully capture the diverse visual features of industrial data and effectively improve the generalization and expression capabilities of the model.

[0142] In some embodiments, the first processing module 310 is used to perform mask processing on the unlabeled industrial image sample set to obtain the first sample set and the second sample set, including:

[0143] Performing random mask processing on the industrial image sample set to obtain a first sample set;

[0144] The industrial image sample set is subjected to random masking and non-masking processing to obtain a second sample set.

[0145] In some embodiments, the first training branch includes a first encoder and a plurality of first linear layers connected to the first encoder, and the second training branch includes a second encoder and a plurality of second linear layers connected to the second encoder;

[0146] The second processing module 320 is used to input the first sample set into the first training branch of the industrial image processing model to obtain the first industrial visual feature output by the first training branch, including:

[0147] Inputting the first sample set into the first training branch, and processing it through the first encoder and the first linear layer to obtain a first category word and a first image block word;

[0148] The second processing module 320 is used to input the second sample set into the second training branch of the industrial image processing model to obtain the second industrial visual features output by the second training branch, including:

[0149] The second sample set is input into the second training branch, and is processed by the second encoder and the second linear layer to obtain a second category word unit and a second image block word unit.

[0150] In some embodiments, the third processing module 330 is used to perform comparative learning on the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature, including:

[0151] Based on the first category word-grams and the second category word-grams, comparative learning is performed on the first training branch and the second training branch.

[0152] In some embodiments, the third processing module 330 is used to perform comparative learning on the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature, including:

[0153] When the mask processing process corresponding to the first sample set is random mask processing, the first image block word-unit is reconstructed based on the second image block word-unit corresponding to the second sample set whose mask processing process is random mask processing.

[0154] In some embodiments, after obtaining the first industrial visual feature output by the first training branch and before obtaining the target loss function between the first training branch and the second training branch, the second processing module 320 is further used to perform center sharpening or clustering processing on the first industrial visual feature.

[0155] In some embodiments, the model parameters of the second training branch are updated based on the back-propagated gradient values.

[0156] In some embodiments, the model parameters of the first training branch are updated based on a sliding average of the model parameters of the second training branch.

[0157] The pre-training device of the industrial image processing model in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0158] The pre-training device of the industrial image processing model in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0159] The pre-training device for the industrial image processing model provided in the embodiment of the present application can implement the various processes implemented in the above-mentioned pre-training method embodiment for the industrial image processing model. To avoid repetition, they will not be described here.

[0160] In some embodiments, Figure 4 As shown, an embodiment of the present application also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the embodiment of the pre-training method of the above-mentioned industrial image processing model is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0161] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0162] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the pre-training method embodiment of the above-mentioned industrial image processing model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0163] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0164] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the pre-training method of the above-mentioned industrial image processing model.

[0165] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0166] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned industrial image processing model pre-training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0167] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0168] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0170] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0171] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0172] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A pre-training method for an industrial image processing model, characterized in that: include: Performing mask processing on the unlabeled industrial image sample set to obtain a first sample set and a second sample set, wherein the mask processing processes corresponding to the first sample set and the second sample set are different; Inputting the first sample set into a first training branch of an industrial image processing model to obtain a first industrial visual feature output by the first training branch, and inputting the second sample set into a second training branch of the industrial image processing model to obtain a second industrial visual feature output by the second training branch; Based on the first industrial visual feature and the second industrial visual feature, performing comparative learning on the first training branch and the second training branch to obtain a target loss function between the first training branch and the second training branch; Based on the target loss function, the model parameters of the first training branch and the second training branch are updated to generate pre-trained model parameters of the industrial image processing model.

2. The pre-training method of the industrial image processing model according to claim 1, characterized in that: The step of performing mask processing on the unlabeled industrial image sample set to obtain a first sample set and a second sample set includes: Performing random mask processing on the industrial image sample set to obtain the first sample set; The industrial image sample set is subjected to random mask processing and non-mask processing to obtain the second sample set.

3. The pre-training method of the industrial image processing model according to claim 1, characterized in that: The first training branch includes a first encoder and a plurality of first linear layers connected to the first encoder, and the second training branch includes a second encoder and a plurality of second linear layers connected to the second encoder; The step of inputting the first sample set into a first training branch of an industrial image processing model to obtain a first industrial visual feature output by the first training branch includes: Inputting the first sample set into the first training branch, and processing the first encoder and the first linear layer to obtain a first category word and a first image block word; The step of inputting the second sample set into the second training branch of the industrial image processing model to obtain the second industrial visual features output by the second training branch includes: The second sample set is input into the second training branch, and is processed by the second encoder and the second linear layer to obtain a second category word-unit and a second image block word-unit.

4. The pre-training method of the industrial image processing model according to claim 3, characterized in that: The performing comparative learning on the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature includes: Based on the first category word-gram and the second category word-gram, comparative learning is performed on the first training branch and the second training branch.

5. The pre-training method of the industrial image processing model according to claim 3, characterized in that: The performing comparative learning on the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature includes: In a case where the mask processing process corresponding to the first sample set is random mask processing, the first image block word-unit is reconstructed based on the second image block word-unit corresponding to the second sample set whose mask processing process is random mask processing.

6. The pre-training method of the industrial image processing model according to any one of claims 1 to 5, characterized in that: After obtaining the first industrial visual feature output by the first training branch and before obtaining the target loss function between the first training branch and the second training branch, the method further includes: The first industrial visual feature is subjected to center sharpening or clustering processing.

7. The pre-training method of the industrial image processing model according to any one of claims 1 to 5, characterized in that: The model parameters of the second training branch are updated based on the gradient values ​​of back propagation.

8. The pre-training method of the industrial image processing model according to any one of claims 1 to 5, characterized in that: The model parameters of the first training branch are updated based on a sliding average of the model parameters of the second training branch.

9. A pre-training device for an industrial image processing model, characterized in that: include: A first processing module is used to perform mask processing on the unlabeled industrial image sample set to obtain a first sample set and a second sample set, wherein the mask processing processes corresponding to the first sample set and the second sample set are different; A second processing module is used to input the first sample set into a first training branch of an industrial image processing model to obtain a first industrial visual feature output by the first training branch, and input the second sample set into a second training branch of the industrial image processing model to obtain a second industrial visual feature output by the second training branch; A third processing module, configured to perform comparative learning on the first training branch and the second training branch based on the first industrial visual feature and the second industrial visual feature, so as to obtain a target loss function between the first training branch and the second training branch; The fourth processing module is used to update the model parameters of the first training branch and the second training branch based on the target loss function, and generate pre-trained model parameters of the industrial image processing model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the pre-training method of the industrial image processing model according to any one of claims 1 to 8 is implemented.

Images