A model pre-training method

Abstract

The model pre-training method provided in the application obtains high-dimensional features of the mask image by taking a preset training model as an encoder and image block features in the mask image as input; obtains predicted pixel values of mask image blocks in the mask image by asymmetric decoding of the high-dimensional features of the mask image; and finally optimizes and trains the preset training model in a self-supervised manner based on an image mask pixel prediction task and an image high-frequency restoration task. Therefore, the model pre-training method provided in the application combines the pre-training tasks of mask image pixel prediction and high-frequency information restoration, so that the pre-training model can learn knowledge in the industrial field from large-scale unlabeled industrial image data, and then it is convenient to obtain a pre-training model that is more suitable for the industrial field based on unlabeled industrial field data, so that the performance of the obtained pre-training model on industrial downstream tasks is more excellent, achieving higher precision and faster convergence speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a model pre-training method. BACKGROUND

[0002] In recent years, artificial intelligence is increasingly applied in the industrial field, and in the industrial field, data collection is difficult, data annotation is difficult, and sample differences are large. Currently, a pre-training model of an ImageNet data set is usually used as an initialization parameter, a small amount of labeled data in an industrial scene is fine-tuned, and finally model parameters specific to the task are obtained. However, the ImageNet data set contains a large number of natural scene images from the network, and the pre-training model based on the ImageNet data set has good performance in object recognition and target detection in the natural scene. However, when the ImageNet pre-training model is applied to the industrial field, due to the significant domain difference between the natural scene images and the industrial field images, the performance of the pre-training model will decrease, and it is difficult to quickly converge to high precision, which cannot meet the needs of the actual task in the industrial field, such as inaccurate detection and identification. SUMMARY

[0003] The embodiments of the present application provide a model pre-training method to obtain a pre-training model more suitable for the industrial field based on unlabeled industrial field data.

[0004] The model pre-training method provided by the present application comprises:

[0005] Dividing an input image into a plurality of image blocks according to a preset size and obtaining actual pixel values of the plurality of image blocks;

[0006] Randomly masking the plurality of image blocks according to a preset probability to obtain unmasked image blocks and masked image blocks;

[0007] Extracting features of all the unmasked image blocks and replacing features of all the masked image blocks with parameters of the same dimension as the unmasked image blocks;

[0008] Using a preset training model as an encoder and using the features of the masked image blocks and the unmasked image blocks as inputs to obtain high-dimensional features of the masked image;

[0009] Passing the high-dimensional features of the masked image through asymmetric decoding to obtain predicted pixel values of the masked image blocks in the masked image;

[0010] The preset training model is self-supervised optimized and trained based on an image mask pixel prediction task and an image high-frequency restoration task; wherein, the image mask pixel prediction task is to calculate the difference between the predicted pixel value and the actual pixel value of all the mask image blocks; and the image high-frequency restoration task is to calculate the difference between the predicted high-frequency information image pixel value and the true high-frequency information image pixel value of all the mask image blocks.

[0011] Optionally, in the above model pre-training method, the feature of all the unmasked image blocks is extracted, including:

[0012] The three-dimensional feature map of the unmasked image block is obtained through convolution processing of all the unmasked image blocks.

[0013] The feature vector of a preset dimension is obtained through linear transformation of the three-dimensional feature map of the unmasked image block.

[0014] Optionally, before the input image is divided into a plurality of image blocks according to the preset size in the above model pre-training method, the method further includes:

[0015] According to the image in the industrial training set, an input image of a target size is obtained through a preset processing, and the preset processing includes one or a combination of scaling, cropping or flipping.

[0016] Optionally, in the above model pre-training method, the input image is divided into a plurality of image blocks according to the preset size, including:

[0017] The input image is divided into a plurality of image blocks according to the size of 16x16.

[0018] The plurality of image blocks are randomly masked according to a preset probability to obtain unmasked image blocks and masked image blocks, including:

[0019] The plurality of image blocks are masked according to a preset probability of 70% to obtain unmasked image blocks and masked image blocks.

[0020] Optionally, in the above model pre-training method, the image mask pixel prediction task is to calculate the difference between the predicted pixel value and the actual pixel value of the mask image block in the mask image, including:

[0021] The image mask pixel prediction task is calculated according to The difference between the predicted pixel value and the actual pixel value of all the mask image blocks is calculated, wherein Loss1 is a mask pixel preset loss, N is the number of the mask image blocks, The predicted pixel value of the mask image block is y ij The actual pixel value of the mask image block is y.

[0022] Optionally, in the model pre-training method, the image high-frequency restoration task is to calculate a difference between predicted high-frequency information image pixel values of all the mask image blocks and real high-frequency information image pixel values, and the difference includes:

[0023] The image high-frequency restoration task is to calculate a difference between predicted high-frequency information image pixel values of all the mask image blocks and real high-frequency information image pixel values, and the difference includes: The image high-frequency restoration task is to calculate a difference between predicted high-frequency information image pixel values of all the mask image blocks and real high-frequency information image pixel values, and the difference includes: The image high-frequency restoration task is to calculate a difference between predicted high-frequency information image pixel values of all the mask image blocks and real high-frequency information image pixel values, and the difference includes: ij The image high-frequency restoration task is to calculate a difference between predicted high-frequency information image pixel values of all the mask image blocks and real high-frequency information image pixel values, and the difference includes:

[0024] Optionally, in the model pre-training method, the preset training model is self-supervised optimized based on the image mask pixel prediction task and the image high-frequency restoration task, and the self-supervised optimization includes:

[0025] The preset training model is self-supervised optimized according to Loss3=Loss1+aLoss2, where Loss3 is a pre-training model loss function, Loss1 is a mask pixel preset loss, Loss2 is a high-frequency information restoration loss, and a is a high-frequency information restoration loss weight factor.

[0026] Optionally, in the model pre-training method, the high-dimensional features of the mask image are transformed through an asymmetric decoding to obtain predicted pixel values of the mask image blocks in the mask image.

[0027] The high-dimensional features of the mask image are transformed through two stacked linear layers to deform and restore a feature vector into a 3x16x16 image block, so as to obtain the predicted pixel values of the mask image blocks in the mask image.

[0028] Optionally, in the model pre-training method, the input image of the target size is obtained through a preset processing according to the images in the industrial training set.

[0029] The input image of the target size is obtained through a preset processing according to the images in the industrial training set.

[0030] Optionally, in the model pre-training method, the a is 0.5.

[0031] In the model training method provided in the application, the high-dimensional features of the mask image are obtained by taking the preset training model as an encoder and the image block features in the mask image as input; the predicted pixel values of the mask image blocks in the mask image are obtained by passing the high-dimensional features of the mask image through asymmetric decoding; and finally the preset training model is self-supervised and optimized based on the image mask pixel prediction task and the image high-frequency restoration task. Therefore, the model training method provided in the application not only realizes the prediction of the pixel values of the input image mask from the high-dimensional image features extracted from the pre-training model, but also introduces the high-frequency information restoration task of the mask image, which can learn the detailed information of the input image. The pre-training task of combining the mask image pixel prediction and the high-frequency information restoration enables the pre-training model to learn the knowledge in the industrial field from large-scale unlabeled industrial image data, and thus facilitates obtaining a pre-training model more suitable for the industrial field based on unlabeled industrial field data. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present disclosure, the drawings needed to be used in some embodiments of the present disclosure will be briefly introduced as follows. Obviously, the drawings in the following description are only some of the drawings of the present disclosure, and other drawings can also be obtained by those skilled in the art according to these drawings. In addition, the drawings in the following description can be regarded as schematic diagrams, and are not limited to the actual size, actual process, actual timing of signals, etc. of the products involved in the embodiments of the present disclosure.

[0033] Figure 1 A flowchart of a model pre-training method according to some embodiments;

[0034] Figure 2 An extraction process diagram of predicted pixel values of a mask image block according to some embodiments;

[0035] Figure 3 A pre-training model loss function change curve diagram in a self-supervised optimization training process of a preset training model according to some embodiments. DETAILED DESCRIPTION

[0036] The technical solutions in some embodiments of the present disclosure will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments provided in the present disclosure, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present disclosure.

[0037] Machine vision technology is an important branch of computer science, involving computer, image processing, pattern recognition, artificial intelligence, signal processing, mechatronics and other technical fields, and is one of the foundations of artificial intelligence. The machine vision system applying machine vision technology is a system using machines to replace human eyes to make various measurements, recognition judgments and detections. The model pre-training is indispensable to machine vision technology, and the model training is the basis for guaranteeing the application of machine vision. How to ensure that the model parameters obtained after model training are suitable for the characteristics of the industrial field, the present application provides a model pre-training method for model training for industrial field images, so that the performance of the obtained pre-training model on the industrial downstream task is more excellent, achieving higher precision and faster convergence speed, such as more accurate image-based detection.

[0038] The model pre-training method provided by the embodiments of the present application is used to train a preset training model to obtain a pre-training model more suitable for the industrial field based on unannotated industrial field data, avoiding the limitations of difficult and costly annotation of industrial field data. In some embodiments, the preset training model can be a classic network model such as Vision Transformer (ViT).

[0039] Figure 1 A flowchart of a model pre-training method according to some embodiments is shown in FIG. Figure 1 As shown in FIG.

[0040] S100: dividing an input image into a plurality of image blocks according to a preset size and obtaining actual pixel values of the plurality of image blocks.

[0041] In the embodiments of the present application, in order to facilitate model pre-training and improve model pre-training, the input image for model pre-training is divided into a plurality of blocks according to a preset size. For example, the input image is divided into 9 (e.g., 3x3) blocks according to the actual size of the input image. Of course, in the embodiments of the present application, the input image can also be divided into a plurality of image blocks of a preset size, for example, the input image is divided into a plurality of image blocks of pixel size 16x16. For example, an input image of pixel size 224x224 is divided into 196 blocks of pixel size 16x16. After dividing the input image into a plurality of image blocks according to the preset size, the actual pixel values of each image block are obtained.

[0042] In the embodiments of the present application, the input image is divided into a plurality of image blocks, which can compress the information in the input image according to the image blocks, facilitating subsequent feature fusion and feature decoding to obtain high-dimensional features corresponding to the input image.

[0043] In some embodiments of the present application, the industrial training set includes a plurality of images, and the industrial training set is usually constructed based on public and private industrial scene data, covering multiple industrial scenes such as 3C and printing. In order to ensure the use effect of the images in the image training set, low-resolution image data needs to be filtered out during the construction of the industrial training set, and the images can also be classified and arranged based on the scene and the task. In some embodiments, before the input image is divided into a plurality of image blocks according to the preset size, the images in the industrial training set need to be preprocessed, such as one or more of scaling, cropping or flipping, to obtain input images with relatively regular sizes. For example, input images with a pixel size of 224x224 are obtained from the industrial training set.

[0044] S200: performing random mask processing on the plurality of image blocks according to a preset probability to obtain unmasked image blocks and masked image blocks.

[0045] This step is used to realize the mask processing of the image blocks in the input image. Specifically, the image blocks in the input image are randomly masked, that is, the plurality of image blocks divided in the input image are randomly masked, so that some unmasked image blocks and some masked image blocks in the input image are obtained, thereby obtaining a masked image including masked image blocks and unmasked image blocks. In some embodiments of the present application, the image blocks in the input image are masked according to a preset probability, that is, each image block in the input image has the same probability of being masked. The preset probability is between 60% and 90%, such as 60%, 70%, etc. For example, the image blocks in the input image are masked according to a preset probability of 70%, that is, each image block in the input image has a probability of 70% of being masked.

[0046] S300: extracting features of all the unmasked image blocks and replacing features of all the masked image blocks with parameters of the same dimension as the unmasked image blocks.

[0047] In some embodiments of the present application, the unmasked image blocks in the masked image are passed through a shared convolutional layer one by one to extract the features of the unmasked image blocks. For example, the input channel number of the shared convolutional layer is 3, the output channel number is 768, the convolution kernel size is 16, and the step is 16, so as to output a feature map of 768x1x1, and then linearly transform it into a feature vector of 768 dimensions, and then extract the features of the unmasked image blocks in the masked image.

[0048] In some embodiments of the present application, the features of the masked image blocks in the masked image are replaced with parameters of the same dimension as the unmasked image blocks, so that the masked image blocks in the masked image obtain learnable parameters of the same dimension as the unmasked image blocks.

[0049] S400: taking the preset training model as an encoder and taking the features of the masked image blocks and the unmasked image blocks as inputs, obtaining high-dimensional features of the masked image.

[0050] The preset training model is taken as an encoder, and the features of the unmasked image blocks and the features of the masked image blocks are taken as inputs to realize feature encoding of the masked image. The preset training model interacts with the features of the unmasked image blocks and the features of the masked image blocks to obtain high-dimensional features of the masked image. In the preset training model, different image block features are transformed and fused through a multi-head attention mechanism, and finally high-dimensional features of the masked image are output.

[0051] S500: obtaining predicted pixel values of the masked image blocks in the masked image by asymmetric decoding of the high-dimensional features of the masked image.

[0052] In the embodiments of the present application, an asymmetric decoder model is used to predict the masked image from the high-dimensional features of the masked image to obtain a predicted image, and then the predicted pixel values of the masked image blocks in the masked image are obtained, realizing feature decoding of the high-dimensional features of the masked image. In some embodiments of the present application, two stacked linear layers are used as the decoder to obtain the predicted pixel values of the corresponding masked image blocks by decoding the high-dimensional features of the masked image.

[0053] In some embodiments of the present application, after the feature encoding of the masked image, Mx768-dimensional features are obtained, where M is the number of image blocks. When decoding the features, each 768-dimensional image feature vector is transformed through two stacked linear layers, and the output feature vector has a size of 768. The final feature vector is deformed and restored to a 3x16x16 image block, which corresponds to the input image block, i.e., the predicted pixel values of the corresponding masked image block are obtained.

[0054] In the embodiments of the present application, the predicted pixel values of the masked image blocks in the masked image are obtained by asymmetric decoding, which facilitates obtaining relatively continuous information in the image decoding process, so that the predicted pixel values can be relatively smooth.

[0055] S600: based on an image mask pixel prediction task and an image high-frequency restoration task, the preset training model is self-supervised and optimized. Wherein: the image mask pixel prediction task is to calculate the difference between the predicted pixel values and the actual pixel values of all the masked image blocks; the image high-frequency restoration task is to calculate the difference between the predicted high-frequency information image pixel values and the true high-frequency information image pixel values of all the masked image blocks.

[0056] In some embodiments of the present application, the preset training model is trained by self-supervised optimization in combination with the image mask pixel prediction task and the image high-frequency restoration task, so that the overall pixel distribution of the input image can be focused on during the training of the preset training model, and the high-frequency details of the input image can also be learned, to facilitate the extraction of more effective image features.

[0057] The image mask pixel prediction task is used to compare the predicted pixel value of the mask image block in the mask image with the actual pixel value of the mask image block, which is obtained in step S100, so as to constrain the preset training model from the preset pixel value of the mask image block by calculating the average pixel difference between the predicted pixel value of the mask image block and the corresponding real pixel value. In some embodiments of the present application, according to The difference between the predicted pixel value of the mask image block in the mask image and the actual pixel value is calculated; wherein Loss1 is the mask pixel preset loss, N is the number of mask image blocks in the mask image, The predicted pixel value of the mask image block is y ij The actual pixel value of the mask image block is y.

[0058] The image high-frequency restoration task is used to compare the predicted high-frequency information image pixel value of the mask image block in the mask image with the real high-frequency information image pixel value. The real high-frequency information image pixel value of the mask image block in the mask image is obtained by performing fast Fourier transform on the grayscale of the input image to realize the conversion from time domain to frequency domain, and obtaining the frequency spectrum of the input image; in the frequency domain, the frequency spectrum corresponding to the input image is high-pass filtered to allow only the high-frequency part to pass, and then the filtered frequency spectrum is converted into a time domain image by inverse Fourier transform, so that the obtained time domain image only contains the high-frequency information in the input image, such as edge, contour and other high-frequency information. The predicted high-frequency information image pixel value of the mask image block is obtained by performing fast Fourier transform on the grayscale of the predicted image of the mask image block to realize the conversion from time domain to frequency domain, and obtaining the frequency spectrum of the preset image; in the frequency domain, the predicted image of the mask image block is high-pass filtered to allow only the high-frequency part to pass, and then the filtered frequency spectrum is converted into a time domain image by inverse Fourier transform, so that the obtained time domain image only contains the high-frequency information in the predicted image, such as edge, contour and other high-frequency information.

[0059] In some embodiments of the present application, according to The difference between the predicted high-frequency information image pixel value of the mask image and the input image high-frequency information image pixel value is calculated; wherein Loss2 is the high-frequency information restoration loss, N is the number of mask image blocks in the mask image, The predicted high-frequency information image pixel value of the mask image block is h iito mask the real high frequency information of the image block.

[0060] In the embodiments of the present application, the mask image prediction will generally cause the preset training model to tend to learn the low frequency information of the image, such as the background and continuous area, during the training process, and the high frequency information is crucial in the image of the industrial training set. If only the image mask pixel task is involved in the training process of the preset training model, it will cause the predicted image to lack key details and the predicted image to be relatively blurred. Therefore, in the embodiments of the present application, the image high frequency restoration task is introduced during the self-supervised optimization training of the preset training model, which can enhance the learning of the high frequency information of the input image by the preset training model.

[0061] In some embodiments of the present application, in order to coordinate the influence of the image mask pixel prediction task and the image high frequency restoration task on the training process of the preset training model, the loss function of the self-supervised optimization training of the preset training model is the weighted sum of the mask pixel preset loss and the high frequency information restoration loss. For example, the loss function Loss3 of the self-supervised optimization training of the preset training model is Loss1+αLoss2, where Loss3 is the loss function of the pre-training model, Loss1 is the mask pixel preset loss, Loss2 is the high frequency information restoration loss, and a is the high frequency information restoration loss weight factor.

[0062] In some embodiments, a is 0.4-0.6, such as 0.5. Further, in the pre-training model loss function, the image mask pixel prediction task is the main task and the image high frequency restoration task is the auxiliary task in the self-supervised optimization training of the preset training model, so as to balance the mask pixel preset loss and the high frequency information restoration loss. In the self-supervised optimization training process of the preset training model, the pre-training model loss function is reduced, and the pre-training model loss function tends to be flat or almost not to decline. Generally, in the self-supervised optimization training process of the preset training model, the training is not less than 400 epochs, and each epoch contains multiple iterations, so the number of model iterations is generally not less than 100k.

[0063] Figure 2 A process diagram for extracting the predicted pixel value of the mask image block according to some embodiments is provided. As shown in FIG. 6, the process includes the following steps: Figure 2As shown, the preprocessed input image is divided into 9 image blocks, and the actual pixel values of the 9 image blocks are obtained and recorded; the 9 image blocks of the input image are subjected to mask processing according to a preset probability of 70%, and the obtained mask image is masked in the 2nd, 3rd, 4th, 6th and 8th image blocks, and the 1st, 5th, 7th and 9th image blocks are not masked; the unmasked image blocks in the mask image are respectively subjected to a shared convolutional layer to extract the features of all unmasked image blocks in the mask image, and the features of all masked image blocks in the mask image are replaced with learnable parameters of the same dimension as the unmasked image blocks to obtain the features of the unmasked image blocks and the masked image blocks in the mask image; the preset training model is used as an encoder, and the unmasked image blocks and the masked image blocks in the mask image are sequentially input into the encoder according to the features of the image blocks, and the high-dimensional features of the mask image are output through the transformation and fusion of the preset training model; and an asymmetric decoder model is used to predict the mask image from the high-dimensional features of the mask image to obtain the predicted pixel values of the masked image blocks in the mask image.

[0064] Figure 3 A pre-training model loss function change curve graph in a self-supervised optimization training process of a preset training model is provided according to some embodiments. As shown in Figure 3 The vertical axis is the function value of the loss function, and the horizontal axis is the number of model iterations. When the number of model iterations is 100k or more, the pre-training model loss function change tends to be flat or almost no longer decreases.

[0065] The model pre-training method provided by the embodiments of the present application obtains the high-dimensional features of the mask image by taking the preset training model as an encoder and the image block features in the mask image as input; the predicted pixel values of the masked image blocks in the mask image are obtained by passing the high-dimensional features of the mask image through an asymmetric decoder; and finally, the preset training model is self-supervised and optimized based on the image mask pixel prediction task and the image high-frequency restoration task. Therefore, the model training method provided by the present application not only realizes the prediction of the pixel values of the input image mask from the high-dimensional image features extracted from the pre-training model, but also introduces the high-frequency information restoration task of the mask image, which can learn the detailed information of the input image. The combination of the pre-training tasks of the mask image pixel prediction and the high-frequency information restoration enables the pre-training model to learn the knowledge in the industrial field from large-scale unlabeled industrial image data, and thus facilitates obtaining a pre-training model more suitable for the industrial field based on unlabeled industrial field data, avoiding the limitations of difficult and costly data labeling in the industrial field.

[0066] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model pre-training method, characterized in that, The method comprises: dividing the input image into a plurality of image blocks according to a preset size and obtaining actual pixel values of the plurality of image blocks; randomly masking the plurality of image blocks according to a preset probability to obtain unmasked image blocks and masked image blocks; extracting features of all the unmasked image blocks and replacing features of all the masked image blocks with parameters of the same dimension as the unmasked image blocks; using a preset training model as an encoder and using the features of the masked image blocks and the unmasked image blocks as inputs to obtain high-dimensional features of the masked image; obtaining predicted pixel values of the masked image blocks in the masked image by asymmetric decoding of the high-dimensional features of the masked image; based on an image mask pixel prediction task and an image high-frequency restoration task, self-supervised optimization training of the preset training model; wherein: the image mask pixel prediction task is to calculate the difference between the predicted pixel values and the actual pixel values of all the masked image blocks; and the image high-frequency restoration task is to calculate the difference between the predicted high-frequency information image pixel values and the true high-frequency information image pixel values of all the masked image blocks.

2. The model pre-training method according to claim 1, wherein, extracting features of all the unmasked image blocks, comprising: obtaining a three-dimensional feature map of the unmasked image blocks through convolution processing of all the unmasked image blocks; obtaining a feature vector of a preset dimension through linear transformation of the three-dimensional feature map of the unmasked image blocks.

3. The model pre-training method according to claim 1, wherein, Before dividing the input image into a plurality of image blocks according to a preset size, the method further comprises: obtaining an input image of a target size through a preset processing according to images in an industrial training set, wherein the preset processing comprises one or a combination of scaling, cropping or flipping.

4. The model pre-training method according to claim 1, characterized in that, dividing the input image into a plurality of image blocks according to a preset size, comprising: dividing the input image into a plurality of image blocks according to a size of 16x16; randomly masking the plurality of image blocks according to a preset probability to obtain unmasked image blocks and masked image blocks, comprising: masking the plurality of image blocks according to a preset probability of 70% to obtain unmasked image blocks and masked image blocks.

5. The model pre-training method according to claim 4, characterized in that, The image mask pixel prediction task is to calculate the difference between the predicted pixel values and the actual pixel values of the masked image blocks in the masked image, comprising: The image mask pixel prediction task is according to Calculate the gap between the predicted pixel value and the actual pixel value of all the mask image blocks, wherein Loss1 is the mask pixel preset loss, N is the number of the mask image blocks, For the predicted pixel value of the mask image block, yij is the actual pixel value of the mask image block.

6. The model pre-training method according to claim 5, characterized in that, The image high-frequency restoration task is to calculate the difference between the predicted high-frequency information image pixel values and the true high-frequency information image pixel values of all the masked image blocks, comprising: The image high-frequency restoration task is according to a difference between a pixel value of a predicted high-frequency information image corresponding to the mask image and a pixel value of the input image high-frequency information image is calculated, wherein Loss2 is a high-frequency information restoration loss, and N is a number of the mask image blocks, a predicted high-frequency information image pixel value of the mask image block, h ii a true high-frequency information image pixel value of the mask image block.

7. The model pre-training method according to claim 6, characterized in that, based on an image mask pixel prediction task and an image high-frequency restoration task, self-supervised optimization training of the preset training model, comprising: self-supervised optimization training of the preset training model according to Loss3=Loss1+αLoss2, wherein Loss3 is a pre-training model loss function, Loss1 is a mask pixel preset loss, Loss2 is a high-frequency information restoration loss, and α is a high-frequency information restoration loss weight factor.

8. The model pre-training method according to claim 4, characterized in that, obtaining predicted pixel values of the masked image blocks in the masked image by asymmetric decoding of the high-dimensional features of the masked image, comprising: The high-dimensional features of the mask image are transformed through two stacked linear layers, and the output feature vector is deformed and restored to a 3x16x16 image block to obtain the predicted pixel value of the mask image block in the mask image.

9. The model pre-training method according to claim 3, characterized in that, According to the images in the industrial training set, input images of a target size are obtained through preset processing, including: According to the images in the industrial training set, input images of a target size are obtained through preset processing, including:

10. The model pre-training method according to claim 7, characterized in that, The alpha is 0.5.

Images