Image block self-supervised learning training method, device, electronic device and storage medium

Abstract

The present invention provides an image block self-supervised learning training method, device, electronic device and computer-readable storage medium. The image block self-supervised learning training method provided by the present invention includes: obtaining an original image to be learned, dividing the original image to be learned into multiple image blocks; masking the peripheral pixel values ​​of the multiple image blocks, and scrambling the order of the multiple image blocks according to a preset scrambling ratio; using preset pixel variables to replace the masked peripheral pixel values ​​of the multiple image blocks to obtain image blocks after pixel value replacement; inputting the image blocks after pixel value replacement into a preset neural network, performing self-supervised learning training on the preset neural network, updating the peripheral pixel values, and obtaining a neural network model after self-supervised learning training. The image block self-supervised learning training method provided by the present invention can improve the training effect of the model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic device and computer-readable storage medium for self-supervised learning of image blocks. Background Art

[0002] Self-supervised learning primarily uses auxiliary tasks to extract self-supervised information from large amounts of unsupervised data. This constructed supervised information is then used to train the network, thereby learning representations that are valuable for downstream tasks. Currently popular computer vision self-supervised learning models, such as MAE and SimMIM, randomly mask portions of an image and feed the unmasked portions into a neural network. The model is trained using the masked portions as supervision. The encoder from the trained model is then used for downstream tasks to improve accuracy. These self-supervised learning models have some drawbacks. Some information in some images cannot be inferred from other parts of the image, such as personal information on invoices. If a portion of the image is masked, the unmasked portion is used to infer the masked portion. However, the masked portion contains unique information that cannot be inferred from other parts of the image, resulting in poor model training results. Summary of the Invention

[0003] The purpose of the present invention is to provide a method, device, electronic device and computer-readable storage medium for self-supervised learning of image blocks to solve the technical problem of poor training effect of the model in the prior art.

[0004] The technical solution of the present invention is as follows: providing an image block self-supervised learning training method, comprising: obtaining an original image to be learned, and dividing the original image to be learned into a plurality of image blocks;

[0005] Masking peripheral pixel values ​​of the plurality of image blocks and disrupting the order of the plurality of image blocks according to a preset disruption ratio;

[0006] Using preset pixel variables to replace peripheral pixel values ​​of the masked image blocks to obtain image blocks after pixel value replacement;

[0007] The image block after the pixel value replacement is input into a preset neural network, the preset neural network is trained through self-supervised learning, the peripheral pixel values ​​are updated, and a neural network model after self-supervised learning training is obtained.

[0008] Furthermore, the image block self-supervised learning training method further includes, after obtaining the neural network model after self-supervised learning training,

[0009] Increasing the preset scrambling ratio, scrambling the order of the multiple image blocks according to the current preset scrambling ratio, inputting the image blocks after pixel values ​​are replaced at the current preset scrambling ratio into the neural network model trained by self-supervised learning, performing self-supervised learning training on the preset neural network, updating the peripheral pixel values, and re-obtaining the neural network model trained by self-supervised learning;

[0010] The preset scrambling ratio is increased again, and the above steps are repeated until the preset scrambling ratio is increased to the set number of times to obtain the final neural network model.

[0011] Furthermore, the image block self-supervised learning training method further includes, before or after scrambling the order of the plurality of image blocks according to the currently preset scrambling ratio, masking peripheral pixel values ​​of the plurality of image blocks and replacing the masked peripheral pixel values ​​of the plurality of image blocks with currently preset pixel variables.

[0012] Furthermore, the preset neural network is subjected to self-supervised learning training, including:

[0013] The pixel values ​​of the plurality of image blocks of the original image to be learned are used as training targets, and the numerical distance between the image blocks replaced with the pixel values ​​and the corresponding values ​​on the original image to be learned is used as the loss function to perform gradient descent training.

[0014] Furthermore, the image block self-supervised learning training method also includes, after dividing the image to be learned into multiple image blocks, and before scrambling the order of the multiple image blocks according to a preset scrambling ratio, determining the position codes of the multiple image blocks; after scrambling the order of the multiple image blocks according to the preset scrambling ratio, masking the position codes of the image blocks in the scrambled order, and replacing the masked position codes with preset coding variables.

[0015] Furthermore, the image block self-supervised learning training method also includes, after obtaining the neural network model after self-supervised learning training or the final neural network model, using the encoder network and its weights in the neural network model, in conjunction with a preset decoder or a preset output layer, to perform computer vision processing tasks.

[0016] Furthermore, the encoder of the preset neural network is a visual Transformer, and the decoder of the preset neural network is a Transformer or a fully connected network.

[0017] Another technical solution of the present invention is as follows: providing an image block self-supervised learning and training device, comprising an image block division module, an image block scrambling module, a pixel replacement module and a model training module;

[0018] The image block division module is used to obtain the original image to be learned and divide the original image to be learned into multiple image blocks;

[0019] The image block scrambling module is configured to mask peripheral pixel values ​​of the plurality of image blocks and scramble the order of the plurality of image blocks according to a preset scrambling ratio;

[0020] The pixel replacement module is configured to replace peripheral pixel values ​​of the masked image blocks with preset pixel variables to obtain image blocks after pixel value replacement.

[0021] The model training module is used to input the image block after the pixel value is replaced into a preset neural network, perform self-supervised learning training on the preset neural network, update the peripheral pixel values, and obtain a neural network model after self-supervised learning training.

[0022] Another technical solution of the present invention is as follows: an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor; the electronic device being characterized in that when the processor executes the computer program, the image block self-supervised learning training method as described in any one of the above technical solutions is implemented.

[0023] Another technical solution of the present invention is as follows: a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image block self-supervised learning training method as described in any of the above technical solutions is implemented.

[0024] The beneficial effects of the present invention are: obtaining an original image to be learned, dividing the original image to be learned into multiple image blocks; masking the peripheral pixel values ​​of the multiple image blocks, and scrambling the order of the multiple image blocks according to a preset scrambling ratio; using preset pixel variables to replace the peripheral pixel values ​​of the multiple masked image blocks to obtain image blocks after pixel value replacement; inputting the image blocks after pixel value replacement into a preset neural network, performing self-supervised learning training on the preset neural network, updating the peripheral pixel values, and obtaining a neural network model after self-supervised learning training; through the above method, the model's dependence on the peripheral pixels of the image blocks is reduced, and the training effect of the model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the process of the image block self-supervised learning training method according to the first embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a self-supervised learning training neural network model according to the first embodiment of the present invention;

[0027] Figure 3Schematic diagram of the structure of the image block self-supervised learning training device according to the second embodiment of the present invention;

[0028] Figure 4 FIG. 4 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0030] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] Figure 1 FIG is a flow chart of the image block self-supervised learning training method of the first embodiment of the present invention. It should be noted that the image block self-supervised learning training method of the present invention does not necessarily use the same method as the first embodiment if the results are substantially the same. Figure 1 The process sequence shown is limited. Figure 1 As shown in FIG, the image block self-supervised learning training method mainly includes the following steps:

[0032] S101, obtaining an original image to be learned, and dividing the original image to be learned into a plurality of image blocks; wherein the division method may be random division, and the area of ​​each image block may be no more than half of the original image to be learned and no less than one-tenth of the original image to be learned, and the number of image blocks may be greater than or equal to 3;

[0033] S102, masking peripheral pixel values ​​of the plurality of image blocks, and shuffling the order of the plurality of image blocks according to a preset shuffling ratio;

[0034] Here, masking the peripheral pixel values ​​of the plurality of image blocks refers to uniformly changing the peripheral pixel values ​​to set pixel values. For example, the pixel value of a certain peripheral pixel point is 20, and the pixel value of another peripheral pixel point is 25. The purpose of masking is to uniformly change all pixel values ​​to a set pixel value, such as 100. The peripheral pixel values ​​are the pixel values ​​of the peripheral pixel points of the image block. Within the range of the peripheral pixel points of the image block, the distance from the outermost pixel point to the innermost pixel point is a preset multiple of the distance from the outermost pixel point to the center of the image block. The preset multiple can be 1 / 5 to 1 / 10. If the image block has too few pixels, the range of the outermost pixel points can be one layer of pixels. For example, if an image block is a rectangular image block of 10*10 pixels, within the range of peripheral pixels of the image block, the distance from the outermost pixel to the innermost pixel is 1 / 5 of the distance from the outermost pixel to the center of the image block. Then the range of peripheral pixels of the image block is the remaining pixel range after removing the 6*6 pixels in the middle part of the matrix image block from the 10*10 pixel rectangular image block.

[0035] S103, using preset pixel variables to replace the peripheral pixel values ​​of the plurality of masked image blocks to obtain image blocks after pixel value replacement; wherein the preset pixel variables can be any natural number, preferably, the range can be [0, 255].

[0036] S104: input the image block after pixel value replacement into a preset neural network, perform self-supervised learning training on the preset neural network, update the peripheral pixel values, and obtain a neural network model after self-supervised learning training.

[0037] Among them, the original image to be learned is a complete image. Before training the neural network model, the peripheral pixel values ​​of the multiple image blocks are masked, and the masked image blocks are assigned initial pixel values. When training the neural network model, the peripheral pixel values ​​are continuously updated to avoid the neural network model from over-relying on the similarity of the peripheral pixels of the image blocks to achieve matching when using image blocks in a disrupted order to reconstruct the entire image, so that the neural network model does not learn the ability to extract features within the image blocks.

[0038] In an optional embodiment, the image block self-supervised learning training method further includes, after obtaining the neural network model after self-supervised learning training,

[0039] Increasing the preset scrambling ratio, scrambling the order of the multiple image blocks according to the current preset scrambling ratio, inputting the image blocks after pixel values ​​are replaced at the current preset scrambling ratio into the neural network model trained by self-supervised learning, performing self-supervised learning training on the preset neural network, updating the peripheral pixel values, and re-obtaining the neural network model trained by self-supervised learning;

[0040] The preset scrambling ratio is increased again, and the above steps are repeated until the preset scrambling ratio is increased to the set number of times to obtain the final neural network model.

[0041] Wherein, before or after the order of the multiple image blocks is scrambled according to the current preset scrambling ratio, the peripheral pixel values ​​of the multiple image blocks are masked, and then the currently preset pixel variables are used to replace the peripheral pixel values ​​of the masked multiple image blocks to obtain the image blocks after the pixel values ​​are replaced at the current preset scrambling ratio. When the image blocks after the pixel values ​​are replaced at the current preset scrambling ratio are input into the neural network model trained by self-supervised learning, the pixels in the image blocks need to be normalized, that is, the pixel values ​​[0,255] of the pixels are normalized to [0,1]. If the preset pixel variable is greater than 255, the corresponding normalized value can also be greater than 1, because the actual pixel value range is [0,255]. After the neural network model training is completed, these values ​​greater than 1 will change to [0,1].

[0042] In a specific embodiment, the initial preset scrambling ratio is set to 1:3, that is, the number of scrambled image blocks is 1 / 3 of the number of unscrambled images. Under this condition, the first preset pixel variable is used to replace the peripheral pixel values ​​of the masked multiple image blocks to obtain image blocks after pixel value replacement; the image blocks after pixel value replacement are input into the preset neural network, the preset neural network is self-supervised learning trained, the peripheral pixel values ​​are updated, and the neural network model after self-supervised learning training is obtained; then the preset scrambling ratio is increased, for example, the preset scrambling ratio is increased to 1:1, that is, the number of scrambled image blocks is equal to the number of unscrambled images, and the multiple images are scrambled according to the preset scrambling ratio at this time. The order of the image blocks is shuffled, and the image blocks after pixel values ​​are replaced at the current preset shuffling ratio are input into the neural network model after self-supervised learning training, the preset neural network is subjected to self-supervised learning training, the peripheral pixel values ​​are updated, and the neural network model after self-supervised learning training is obtained again; thereafter, the preset shuffling ratio can be increased, and the neural network model after self-supervised learning training obtained after the last increase of the preset shuffling ratio is used as the current model to be trained, so as to further train the current model to be trained, and so on, until the preset shuffling ratio is increased to a set number of times, and a final neural network model is obtained, and the final neural network model is the final training result.

[0043] In an optional embodiment, before or after scrambling the order of the multiple image blocks according to the current preset scrambling ratio, it also includes masking the peripheral pixel values ​​of the multiple image blocks and using the currently preset pixel variables to replace the masked peripheral pixel values ​​of the multiple image blocks.

[0044] It should be noted that, under a certain preset scrambling ratio, the preset pixel variables are the initial pixel values ​​of the peripheral pixel values ​​of the multiple masked image blocks. As the neural network model is trained, the peripheral pixel values ​​of the multiple image blocks are updated accordingly. Under different preset scrambling ratios, the preset pixel variables can be the same or different. The current preset pixel variables are the preset pixel variables under the current preset scrambling ratio.

[0045] In an optional embodiment, performing self-supervised learning training on the preset neural network includes:

[0046] The pixel values ​​of the plurality of image blocks of the original image to be learned are used as training targets, and the numerical distance between the image blocks replaced with the pixel values ​​and the corresponding values ​​on the original image to be learned is used as the loss function to perform gradient descent training.

[0047] It should be noted that the output of the neural network model uses the pixel values ​​of multiple image blocks of the original image to be learned as training targets, and the neural network model uses the numerical distance between the image blocks after the pixel values ​​are replaced and the corresponding numerical distance on the original image to be learned as the loss function.

[0048] In an optional embodiment, the image block self-supervised learning training method further includes, after dividing the image to be learned into a plurality of image blocks and before shuffling the order of the plurality of image blocks according to a preset shuffling ratio, determining position codes of the plurality of image blocks; after shuffling the order of the plurality of image blocks according to the preset shuffling ratio, masking the position codes of the shuffled image blocks and replacing the masked position codes with a preset coding variable. The preset coding variable is an initial coding variable.

[0049] In one specific embodiment, the image to be learned is divided into multiple image blocks, the position codes of the multiple image blocks are determined, the order of the multiple image blocks is shuffled according to a preset shuffling ratio, the position codes of the shuffled image blocks are masked, and the masked position codes are replaced with preset coding variables; the image blocks with pixel value replacement are input into a preset neural network, the preset neural network is trained through self-supervised learning, the peripheral pixel values ​​are updated, and a neural network model after self-supervised learning training is obtained. When the neural network model is trained through self-supervised learning, the codes of the multiple image blocks of the original image to be learned are output as training targets, and the final neural network module can restore the original position codes of the image blocks.

[0050] In which, during the self-supervised learning training of the preset neural network, the peripheral pixel values ​​are updated. Specifically, the pixel values ​​of multiple image blocks of the original image to be learned are used as training targets, and the numerical distance between the image blocks replaced by the pixel values ​​and the corresponding numerical distance on the original image to be learned is used as the loss function. The preset neural network is trained with gradient descent, thereby continuously updating the peripheral pixel values ​​during training.

[0051] In an optional embodiment, the encoder of the preset neural network is a visual Transformer, and the decoder of the preset neural network is a Transformer or a fully connected network.

[0052] The visual Transformer encoder can be used to divide a complete image into multiple image blocks for input. The Transformer decoder or fully connected network is used to extract the output features of the visual Transformer encoder and output the network results. The predicted values ​​of each pixel in the image block can be output in sequence. Each pixel in the output of the visual Transformer encoder corresponds to a numerical value. The Transformer decoder or fully connected network processes the numerical values ​​corresponding to the pixels. The visual Transformer can be a model such as ViT or Swin Transformer.

[0053] In an optional embodiment, the image block self-supervised learning training method also includes, after obtaining the neural network model after self-supervised learning training or the final neural network model, using the encoder network and its weights in the neural network model, in conjunction with a preset decoder or a preset output layer, to perform computer vision processing tasks.

[0054] In one specific embodiment, after obtaining a neural network model trained through self-supervised learning or a final neural network model, the encoder network and its weights in the neural network model are combined with other output layers or decoders, and the network weights are fine-tuned through supervised learning using labeled images. This allows the fine-tuned network to be used for other computer vision tasks such as image classification, object detection, semantic segmentation, key point detection, or feature matching. The type of other output layers or decoders depends on the computer vision task. The image block self-supervised learning training method provided in this embodiment of the present invention is primarily intended to train the encoder network and its weights in the neural network model.

[0055] In another specific embodiment, the original image to be learned is divided into four image blocks, and 1, 2, 3, and 4 represent the position codes of the four image blocks respectively. When the initial preset shuffling ratio is 1:1, that is, the number of shuffled image blocks and the number of unshuffled images are both 2, under this condition, the first preset pixel variable is used to replace the peripheral pixel values ​​of the masked multiple image blocks to obtain image blocks after pixel value replacement; the image blocks after pixel value replacement are input into the preset neural network, and the preset neural network is trained by self-supervised learning, and the peripheral pixel values ​​are updated to obtain a neural network model after self-supervised learning training. When the preset shuffling ratio is increased to 3:1, that is, the number of shuffled image blocks is three times the number of unshuffled images, the order of the multiple image blocks is shuffled according to the preset shuffling ratio at this time, and the image blocks after pixel value replacement at the current preset shuffling ratio are input into the neural network model after self-supervised learning training, and the preset neural network is trained by self-supervised learning, and the peripheral pixel values ​​are updated to obtain a neural network model after self-supervised learning training.

[0056] In specific implementation, when the preset scrambling ratio is 3:1, the input order of some image blocks (position codes are 1, 2, 4) is scrambled, the position codes corresponding to the scrambled image blocks are masked, and the masked position codes are replaced by a preset coding variable. The unscrambled image blocks (position codes are 3) are input normally together with the position codes. The output of the neural network uses the pixel values ​​of the normal order image blocks (position codes are 1, 2, 3, 4) of the original image to be learned as the training target. The distance between the model output values ​​and the corresponding values ​​on the original image to be learned for all pixel points of the image blocks (position codes are 1, 2, 4) whose order is scrambled during input is calculated, and this is used as the loss function to perform gradient descent training.

[0057] When shuffled image blocks and unshuffled image blocks are input into the network, the pixel values ​​of several outer circles are masked and replaced with preset pixel variables. During the training of the self-supervised learning model with shuffled image block input order, the proportion of shuffled image blocks is gradually increased. In the first round of training, the order of any image blocks can be kept unchanged (the preset shuffle ratio is 0), and the position codes are also input normally. As the number of training rounds increases, the proportion of shuffled image blocks gradually increases until it reaches the set maximum ratio and no longer changes. Such a training process enables the neural network model to start with a simpler learning task at the beginning, gradually increase the learning difficulty, and improve the training stability.

[0058] Schematic diagram of self-supervised learning training neural network model, such as Figure 2 As shown, Figure 2In the figure, the image blocks in the first row from bottom to top are the image blocks input to the neural network model. The order of these image blocks has been disrupted, and the position codes of the disrupted image blocks have been masked and reassigned. The image blocks in the second row from bottom to top are the features encoded by the encoder, the image blocks in the third row from bottom to top are the output image blocks decoded by the decoder, and the image blocks in the fourth row from bottom to top are the target image blocks, which are image blocks in normal order. The corresponding numerical distance between the image blocks in normal order and the image blocks after pixel value replacement is used as the loss function.

[0059] The image block self-supervised learning training method provided by the embodiment of the present invention obtains an original image to be learned, divides the original image to be learned into multiple image blocks; masks the peripheral pixel values ​​of the multiple image blocks, and shuffles the order of the multiple image blocks according to a preset shuffling ratio; uses preset pixel variables to replace the peripheral pixel values ​​of the multiple masked image blocks to obtain image blocks after pixel value replacement; inputs the image blocks after pixel value replacement into a preset neural network, performs self-supervised learning training on the preset neural network, updates the peripheral pixel values, and obtains a neural network model after self-supervised learning training; through the above method, the model's dependence on the peripheral pixels of the image blocks is reduced, and the training effect of the model is improved.

[0060] The image block self-supervised learning training method provided by the embodiment of the present invention, when training the neural network model, avoids the neural network model from over-relying on the similarity of the peripheral pixels of the image block to achieve matching when using the image blocks in a shuffled order to reconstruct the entire image, so that the neural network model fails to learn the ability to extract features within the image block.

[0061] When training models in existing technologies, a fixed masking ratio is used. Large-ratio masking is too difficult for the model in the early stages of training, leading to unstable training and requiring larger batch sizes and higher GPU memory. Ultimately, the training effect is poor.

[0062] The image block self-supervised learning training method provided by the embodiment of the present invention masks the peripheral pixel values ​​when the image blocks are input into the neural network, and the masked pixel values ​​are replaced by preset pixel variables. During training, the self-supervised learning model with the image block input order disrupted gradually increases the proportion of the image blocks in the disrupted order until it reaches a set maximum proportion and no longer changes. Such a training process enables the neural network model to start with a relatively simple learning task at the beginning, gradually increase the learning difficulty, and improve training stability.

[0063] The image block self-supervised learning training method provided in this embodiment can be constructed based on artificial intelligence. Relevant data can be acquired and processed using artificial intelligence technology to achieve unsupervised artificial intelligence image block self-supervised learning training. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0064] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0065] Figure 3 FIG. 1 is a schematic diagram of the structure of the image block self-supervised learning training device according to the second embodiment of the present invention. Figure 3 As shown, the image block self-supervised learning training device 30 includes an image block division module 31, an image block shuffling module 32, a pixel replacement module 33 and a model training module 34;

[0066] The image block division module 31 is used to obtain the original image to be learned and divide the original image to be learned into multiple image blocks;

[0067] The image block scrambling module 32 is configured to mask peripheral pixel values ​​of the plurality of image blocks and scramble the order of the plurality of image blocks according to a preset scrambling ratio;

[0068] The pixel replacement module 33 is configured to replace peripheral pixel values ​​of the masked image blocks with preset pixel variables to obtain image blocks after pixel value replacement.

[0069] The model training module 34 is used to input the image block after the pixel value is replaced into a preset neural network, perform self-supervised learning training on the preset neural network, update the peripheral pixel values, and obtain a neural network model after self-supervised learning training.

[0070] Among them, the original image to be learned is a complete image. Before training the neural network model, the peripheral pixel values ​​of the multiple image blocks are masked, and the masked image blocks are assigned initial pixel values. When training the neural network model, the peripheral pixel values ​​are continuously updated to avoid the neural network model from over-relying on the similarity of the peripheral pixels of the image blocks to achieve matching when using image blocks in a disrupted order to reconstruct the entire image, so that the neural network model does not learn the ability to extract features within the image blocks.

[0071] Furthermore, the image block self-supervised learning training device also includes a repeated execution module, which is used to increase the preset scrambling ratio after obtaining the neural network model after self-supervised learning training, scramble the order of the multiple image blocks according to the current preset scrambling ratio, input the image blocks after pixel values ​​are replaced at the current preset scrambling ratio into the neural network model after self-supervised learning training, perform self-supervised learning training on the preset neural network, update the peripheral pixel values, and obtain the neural network model after self-supervised learning training again; increase the preset scrambling ratio again, repeat the above steps until the preset scrambling ratio is increased to a set number of times, and obtain the final neural network model.

[0072] Among them, the initial preset scrambling ratio is set to 1:3, that is, the number of scrambled image blocks is 1 / 3 of the number of unscrambled images. Under this condition, the first preset pixel variable is used to replace the peripheral pixel values ​​of the masked multiple image blocks to obtain the image blocks after pixel value replacement; the image blocks after pixel value replacement are input into the preset neural network, the preset neural network is self-supervised learning trained, the peripheral pixel values ​​are updated, and the neural network model after self-supervised learning training is obtained; then the preset scrambling ratio is increased, for example, the preset scrambling ratio is increased to 1:1, that is, the number of scrambled image blocks is equal to the number of unscrambled images, and the multiple image blocks are scrambled according to the preset scrambling ratio at this time. The image blocks after pixel value replacement under the current preset scrambling ratio are input into the neural network model after self-supervised learning training, the preset neural network is trained through self-supervised learning, the peripheral pixel values ​​are updated, and the neural network model after self-supervised learning training is obtained again; thereafter, the preset scrambling ratio can be increased, and the neural network model after self-supervised learning training obtained after the last increase of the preset scrambling ratio is used as the current model to be trained, so as to further train the current model to be trained, and so on, until the preset scrambling ratio is increased to a set number of times, and the final neural network model is obtained. The final neural network model is the final training result.

[0073] Furthermore, the image block self-supervised learning training device also includes an image block acquisition module, which is used to use the current preset pixel variables to replace the peripheral pixel values ​​of the multiple masked image blocks to obtain the image blocks after pixel value replacement under the current preset scrambling ratio.

[0074] The preset pixel variables are the initial pixel values ​​of the peripheral pixels of the masked image blocks at a certain preset scrambling ratio. As the neural network model is trained, the peripheral pixel values ​​of the multiple image blocks are updated accordingly. The preset pixel variables can be the same or different at different preset scrambling ratios. The current preset pixel variables are the preset pixel variables at the current preset scrambling ratio.

[0075] Furthermore, the model training module is also used to perform gradient descent training using the pixel values ​​of multiple image blocks of the original image to be learned as training targets, and the numerical distance between the image blocks after the pixel values ​​are replaced and the corresponding numerical distance on the original image to be learned as the loss function.

[0076] Furthermore, the image block self-supervised learning training device also includes a position coding replacement module, which is used to determine the position coding of the multiple image blocks after dividing the image to be learned into multiple image blocks and before scrambling the order of the multiple image blocks according to a preset scrambling ratio; and is also used to mask the position coding of the image blocks in the scrambled order after scrambling the order of the multiple image blocks according to the preset scrambling ratio, and replace the masked position coding with a preset coding variable.

[0077] Furthermore, the image block self-supervised learning training device also includes a position code updating module, which is used to update the masked position code when performing self-supervised learning training on the preset neural network.

[0078] The encoder of the preset neural network is a visual Transformer, and the decoder of the preset neural network is a Transformer or a fully connected network.

[0079] After obtaining a neural network model trained through self-supervised learning or a final neural network model, the encoder network and its weights in the neural network model are used in conjunction with other output layers or decoders, and the network weights are fine-tuned through supervised learning using labeled images, so that the fine-tuned network can be used for other computer vision tasks such as image classification, object detection, semantic segmentation, key point detection, or feature matching. The type of other output layers or decoders depends on the computer vision task. The image block self-supervised learning training method provided in this embodiment of the present invention is primarily intended to train the encoder network and its weights in the neural network model.

[0080] Among them, when the shuffled image blocks and the unshuffled image blocks are input into the network, the pixel values ​​of several outer circles are masked, and the masked pixel values ​​are replaced by preset pixel variables. During the training of the self-supervised learning model with shuffled image block input order, the proportion of shuffled image blocks is gradually increased. In the first round of training, the order of any image blocks can be kept unchanged, and the position codes are also input normally. As the number of training rounds increases, the proportion of shuffled image blocks gradually increases until it reaches the set maximum proportion and no longer changes. Such a training process enables the neural network model to start from a simpler learning task at the beginning, gradually increase the learning difficulty, and improve the training stability.

[0081] The image block self-supervised learning training device provided by the embodiment of the present invention obtains an original image to be learned and divides the original image to be learned into multiple image blocks; masks the peripheral pixel values ​​of the multiple image blocks and shuffles the order of the multiple image blocks according to a preset shuffling ratio; uses preset pixel variables to replace the peripheral pixel values ​​of the multiple masked image blocks to obtain image blocks after pixel value replacement; inputs the image blocks after pixel value replacement into a preset neural network, performs self-supervised learning training on the preset neural network, updates the peripheral pixel values, and obtains a neural network model after self-supervised learning training; reduces the model's dependence on the peripheral pixels of the image blocks, and improves the training effect of the model.

[0082] The image block self-supervised learning training device provided by an embodiment of the present invention, when training a neural network model, continuously updates the peripheral pixel values, thereby avoiding the neural network model from over-relying on the similarity of the peripheral pixels of the image block to achieve matching when using the image blocks in a shuffled order to reconstruct the entire image, so that the neural network model fails to learn the ability to extract features within the image block.

[0083] Figure 4 FIG. 1 is a schematic diagram of the structure of an electronic device according to a third embodiment of the present invention. Figure 4 As shown, the electronic device 40 includes a processor 41 and a memory 42 communicatively connected to the processor 41 .

[0084] The memory 42 stores program instructions for implementing the image block self-supervised learning training method of any of the above embodiments.

[0085] The processor 41 is configured to execute program instructions stored in the memory 42 to perform code testing.

[0086] The processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having signal processing capabilities. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0087] The memory 42 may be used to store the computer programs and / or modules. The processor 41 implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory 42 and calling data stored in the memory 42. The memory 42 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the like.

[0088] The memory 42 may be integrated into the processor 41 or may be provided separately from the processor 41 .

[0089] A fourth embodiment of the present invention provides a storage medium. The storage medium of the embodiment of the present invention stores program instructions capable of implementing all of the above methods. The storage medium may be non-volatile or volatile. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM). The program instructions may be stored in the above storage medium in the form of a software product, including several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code, or terminal devices such as computers, servers, mobile phones, and tablets.

[0090] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0091] In addition, the functional modules in the various embodiments of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. The above is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

[0092] The above description is only an embodiment of the present invention. It should be pointed out that those skilled in the art can make improvements without departing from the creative concept of the present invention, but these improvements all fall within the scope of protection of the present invention.

Claims

1. A self-supervised learning training method for image blocks, characterized in that: The steps include: Obtaining an original image to be learned, and dividing the original image to be learned into a plurality of image blocks; Masking peripheral pixel values ​​of the plurality of image blocks and shuffling the order of the plurality of image blocks according to a preset shuffling ratio, wherein the peripheral pixel values ​​are pixel values ​​of peripheral pixel points of the image blocks, and within the range of the peripheral pixel points, the distance from the outermost pixel point to the innermost pixel point is a preset multiple of the distance from the outermost pixel point to the center of the image block; Using preset pixel variables to replace peripheral pixel values ​​of the masked image blocks to obtain image blocks after pixel value replacement; Inputting the image block after pixel value replacement into a preset neural network, performing self-supervised learning training on the preset neural network, updating the peripheral pixel values, and obtaining a neural network model after self-supervised learning training; Increasing the preset scrambling ratio, scrambling the order of the multiple image blocks according to the current preset scrambling ratio, inputting the image blocks after pixel values ​​are replaced at the current preset scrambling ratio into the neural network model trained by self-supervised learning, performing self-supervised learning training on the preset neural network, updating the peripheral pixel values, and re-obtaining the neural network model trained by self-supervised learning; The preset scrambling ratio is increased again, and the above steps are repeated until the preset scrambling ratio is increased to the set number of times to obtain the final neural network model.

2. The image block self-supervised learning training method according to claim 1, characterized in that: Before or after scrambling the order of the multiple image blocks according to the current preset scrambling ratio, it also includes masking the peripheral pixel values ​​of the multiple image blocks and using the currently preset pixel variables to replace the masked peripheral pixel values ​​of the multiple image blocks.

3. The image block self-supervised learning training method according to claim 1, characterized in that: Performing self-supervised learning training on the preset neural network includes: The pixel values ​​of the plurality of image blocks of the original image to be learned are used as training targets, and the numerical distance between the image blocks replaced with the pixel values ​​and the corresponding values ​​on the original image to be learned is used as the loss function to perform gradient descent training.

4. The image block self-supervised learning training method according to claim 1, characterized in that: The method further includes, after dividing the image to be learned into a plurality of image blocks and before shuffling the order of the plurality of image blocks according to a preset shuffling ratio, determining position codes of the plurality of image blocks; After the order of the plurality of image blocks is disrupted according to a preset disruption ratio, the position codes of the image blocks in the disrupted order are masked, and the masked position codes are replaced with preset coding variables.

5. The image block self-supervised learning training method according to claim 1, characterized in that: It also includes, after obtaining the neural network model trained by self-supervised learning or the final neural network model, using the encoder network and its weights in the neural network model, in conjunction with a preset decoder or a preset output layer, to perform computer vision processing tasks.

6. The image block self-supervised learning training method according to claim 1, characterized in that: The encoder of the preset neural network is a visual Transformer, and the decoder of the preset neural network is a Transformer or a fully connected network.

7. An image block self-supervised learning training device, characterized in that: It includes image block division module, image block scrambling module, pixel replacement module, model training module and repeated execution module; The image block division module is used to obtain the original image to be learned and divide the original image to be learned into multiple image blocks; The image block scrambling module is configured to mask peripheral pixel values ​​of the plurality of image blocks and scramble the order of the plurality of image blocks according to a preset scrambling ratio, wherein the peripheral pixel values ​​are pixel values ​​of peripheral pixel points of the image block, and within the range of the peripheral pixel points, the distance from the outermost pixel point to the innermost pixel point is a preset multiple of the distance from the outermost pixel point to the center of the image block; The pixel replacement module is configured to replace peripheral pixel values ​​of the masked image blocks with preset pixel variables to obtain image blocks after pixel value replacement. The model training module is used to input the image block after the pixel value replacement into a preset neural network, perform self-supervised learning training on the preset neural network, update the peripheral pixel values, and obtain a neural network model after self-supervised learning training; The repeated execution module is used to increase the preset scrambling ratio, scramble the order of the multiple image blocks according to the current preset scrambling ratio, input the image blocks after pixel values ​​are replaced at the current preset scrambling ratio into the neural network model trained by self-supervised learning, perform self-supervised learning training on the preset neural network, update the peripheral pixel values, and obtain the neural network model trained by self-supervised learning again; increase the preset scrambling ratio again, and repeat the above steps until the preset scrambling ratio is increased a set number of times to obtain the final neural network model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image block self-supervised learning training method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image block self-supervised learning training method according to any one of claims 1 to 6 is implemented.

Images