Image processing method, system and equipment based on reconstruction contrast learning, and medium

Abstract

The embodiment of the invention provides an image processing method and system based on reconstruction contrast learning, equipment and a medium. The method comprises the steps of obtaining a to-be-processed image; performing data augmentation on the to-be-processed image, and transferring the to-be-processed image to multiple views; performing pixel shielding on the to-be-processed image after data augmentation; encoding the to-be-processed image subjected to pixel shielding by adopting a neural network, and converting the to-be-processed image into hidden representation; and decoding the encoded to-be-processed image to obtain a reconstructed image. According to the embodiment of the invention, an information bottleneck principle is utilized, a powerful data augmentation result is obtained through combination of shielding operation and data augmentation, the performance of comparative learning is improved, an effective reconstruction task is provided as a regular term of a discrimination task, the generalization ability of the model is improved, and the method is suitable for popularization and application. The problems that compression is insufficient and discriminant information is over-fitted in existing comparative learning are solved, the performance in various downstream tasks is greatly improved, and meanwhile the model is more interpretable.

Description

Technical Field

[0001] The embodiments disclosed herein belong to the field of image processing technology, specifically relating to an image processing method, system, device, and medium based on reconstruction contrastive learning. Background Technology

[0002] Contrastive learning (CL) is a fundamental and important problem in the field of self-supervised learning. Its purpose is to maximize the similarity of positive pairs and the dissimilarity of negative pairs without explicit data labels. It distinguishes the representations of different samples by bringing positive sample pairs closer and negative sample pairs further apart. It has a powerful ability to expand the training set and extract representative features, and has shown good performance in various tasks such as image classification, semantic segmentation and object detection.

[0003] Existing research on contrastive learning employs data augmentation to obtain multiple views, followed by further encoding using conjoined networks to improve generality. However, due to the low information density of image data, existing augmentation methods still contain a lot of noise and redundant information, which affects the extraction of the required information and thus limits the performance of contrastive learning methods in practice. Summary of the Invention

[0004] The embodiments disclosed herein aim to at least solve one of the technical problems existing in the prior art, and provide an image processing method, system, device and medium based on reconstruction contrastive learning.

[0005] One aspect of this disclosure provides an image processing method based on reconstruction contrastive learning, the method comprising:

[0006] Obtain the image to be processed;

[0007] Data augmentation is performed on the image to be processed, and the image to be processed is transferred to a multi-view;

[0008] Pixel masking is applied to the augmented image to be processed.

[0009] The image to be processed after pixel masking is encoded using a neural network, and the image to be processed is converted into a hidden representation;

[0010] The encoded image to be processed is decoded to obtain the reconstructed image.

[0011] Furthermore, the pixel masking of the augmented image to be processed includes:

[0012] The pixels of the augmented image to be processed are randomly masked at a preset masking rate.

[0013] Furthermore, the step of encoding the pixel-masked image using a neural network includes:

[0014] Repair the image to be processed after pixel masking;

[0015] Mask markers are added to the repaired image to be processed using linear projection and positional embedding.

[0016] Further, decoding the encoded image to be processed includes:

[0017] Add location embeddings to the complete list of tags to encode location information.

[0018] Another embodiment of this disclosure provides an image processing system based on reconstruction contrastive learning, the system comprising:

[0019] The acquisition module is used to acquire the image to be processed;

[0020] An augmentation module is used to perform data augmentation on the image to be processed and transfer the image to be processed into a multi-view;

[0021] A masking module is used to perform pixel masking on the image to be processed after data augmentation.

[0022] The encoding module is used to encode the pixel-masked image to be processed using a neural network, and to convert the image to be processed into a hidden representation;

[0023] The decoding module is used to decode the encoded image to obtain the reconstructed image.

[0024] Furthermore, the masking module is specifically used for:

[0025] The pixels of the augmented image to be processed are randomly masked at a preset masking rate.

[0026] Furthermore, the encoding module is specifically used for:

[0027] Repair the image to be processed after pixel masking;

[0028] Mask markers are added to the repaired image to be processed using linear projection and positional embedding.

[0029] Furthermore, the decoding module is specifically used for:

[0030] Add location embeddings to the complete list of tags to encode location information.

[0031] Another embodiment of this disclosure provides an electronic device, including:

[0032] At least one processor; and,

[0033] A memory communicatively connected to the at least one processor is used to store one or more programs that, when executed by the at least one processor, enable the at least one processor to implement the image processing method based on reconstruction contrastive learning described above.

[0034] Another embodiment of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method based on reconstruction contrastive learning described above.

[0035] This disclosure discloses an image processing method, system, device, and medium based on reconstruction-contrast learning. Utilizing the information bottleneck principle, it defines a minimum sufficient increment and information generalization representation, proposes an additional reconstruction task, and solves the problems of insufficient compression and overfitting of discriminative information in existing contrastive learning methods, significantly improving performance in various downstream tasks. Through a combination of masking operations and data augmentation, powerful data augmentation results are obtained, improving the performance of contrastive learning. Furthermore, an effective reconstruction task is proposed as a regularization term for the discriminative task, thereby enhancing the model's generalization ability. A novel information metric method is presented, providing accurate characterization of the model and techniques, greatly improving the model's interpretability. Attached Figure Description

[0036] Figure 1 This is a schematic flowchart of an image processing method based on reconstruction contrastive learning according to an embodiment of the present disclosure;

[0037] Figure 2 This is a schematic diagram of the structure of an image processing system based on reconstruction contrastive learning according to another embodiment of the present disclosure;

[0038] Figure 3 This is a schematic diagram of the structure of an electronic device according to another embodiment of the present disclosure. Detailed Implementation

[0039] Landmark works in contrastive learning, such as MoCo, BYOL, and SimSiam, have made significant improvements to the model, including projectors, predictors, momentum encoders, and memory banks, to address some key issues in contrastive learning. Meanwhile, many other related studies have also offered new insights and improvements from an information perspective.

[0040] Among them, the InfoMin method offers a new insight: good multi-view sharing with minimal necessary information can better accomplish downstream tasks. It presents a learnable view generator and a method to minimize mutual information between two views.

[0041]

[0042] In the formula, g is the discriminator network, and f1 and f2 are the encoder networks. However, this study only reduces the gap between shared information and optimal information, while ignoring the requirement for minimum information.

[0043] Wang Haoqing et al. argued that non-shared task-related information should not be ignored, and therefore preserved the available non-shared information by maximizing the mutual information between the view and the representation:

[0044]

[0045] This method is helpful for the model to extract more information, but simply including information can lead to an increase in redundant information, affecting the model's performance in downstream tasks. Therefore, while extracting as much information as possible, irrelevant information should be removed to avoid impacting downstream tasks.

[0046] Furthermore, for the setup of the pre-task, current contrastive learning models have chosen a discriminative task, the purpose of which is to measure and separate the distance between positive and negative samples to distinguish different samples. To maintain simplicity and effectiveness, current contrastive learning focuses on sample differentiation at the abstract semantic level of the feature space, rather than complex details. However, a standalone discriminative task can lead the model to a bias of only extracting information that helps in discriminating samples, reducing performance on different downstream tasks and causing overfitting to discriminative information.

[0047] To address the aforementioned issues, more robust information compression methods are needed to produce minimal sufficient augmentation that removes all redundant information. Embodiments of this disclosure improve contrastive learning from the perspective of the Information Bottleneck (IB) criterion. This criterion is applied at each processing stage to extract optimal information, thereby improving the performance of contrastive learning methods. The IB criterion aims to analyze information changes during the learning process of deep neural networks. It primarily proposes the idea that information compression and information representation coexist during network training and has been widely applied to interpretable and engineered deep learning.

[0048] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0049] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0050] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0051] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this disclosure. As used in this disclosure, the term "and / or" includes all combinations of any and more of the associated listed items.

[0052] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing this disclosure, and therefore cannot be used to limit the scope of protection of this disclosure.

[0053] like Figure 1 As shown, one embodiment of this disclosure provides an image processing method based on reconstruction contrastive learning, including:

[0054] Step S1: Obtain the image to be processed.

[0055] Specifically, this disclosure proposes a Masked Reconstruction Contrastive Learning (MRCL) model. Unlike traditional Contrastive Learning (CL), MRCL performs masking operations after data augmentation to compress information at the pixel level; and finally, a decoder is used to reconstruct the image, preventing the model from being biased towards extracting only discriminative information, thereby achieving better performance in downstream tasks. Like traditional CL, MRCL for image processing requires first acquiring the image to be processed.

[0056] Step S2: Perform data augmentation on the image to be processed and transfer the image to a multi-view.

[0057] Specifically, data augmentation transfers the image to be processed x obtained in the previous step S1 to the augmented multiview.

[0058] Step S3: Perform pixel masking on the augmented image to be processed.

[0059] Specifically, the image to be processed after augmentation of the data in the previous step S2 is augmented with a preset occlusion rate r. The pixels are masked to obtain the masked image. Masking is an effective method for removing redundant information; therefore, after regular data augmentation, a masking operation M is further added. r (·) Obtain the obscured augmented view v:

[0060]

[0061] There are various masking strategies, such as "blocking," "grid-like," and "random sampling." Among them, "random sampling" has shown better performance in many applications. Therefore, this embodiment adopts "random sampling" as the default masking strategy for the augmented image to be processed. The pixels are randomly masked. Obviously, the higher the proportion of random sampling, the more pixel blocks are masked, and thus the more redundancy is eliminated.

[0062] Step S4: Encode the image to be processed after pixel masking using a neural network, and convert the image to be processed into a hidden representation.

[0063] Specifically, the encoder will take the masked image obtained in step S4 above. The extracted data is transformed into a hidden representation. The encoder-decoder network structure in this embodiment employs a deep neural network framework, such as CNN, ResNet, or ViT. To better adapt to patch occlusion and improve feature extraction capabilities, this embodiment preferably uses ViT (Vision Transformer) as the encoder backbone. It is only applied to the occluded image. The unmasked patches in the image. Following the original ViT setup, the encoder embeds patches via linear projection, adds positional embedding, and then feeds the resulting patch set into a series of Transformer blocks to obtain the hidden representation z. In practice, MRCL first processes the pixel-masked image... The patching is performed, and then a mask marker is added to each input patch by linear projection and increased position embedding. Next, the patch is randomly shuffled, and the last part of the list is discarded according to the occlusion rate r.

[0064] Step S5: Decode the encoded image to obtain the reconstructed image.

[0065] Specifically, to alleviate the overfitting problem of discriminative information, embodiments of this disclosure use a decoder to reconstruct the view. Although the encoder in the previous step S4 only encodes unmasked blocks, the input to the decoder includes encoded visible blocks and mask markers, which are shared, learned vectors representing the presence of these missing blocks. Furthermore, this embodiment adds position embeddings to the complete list of markers to encode positional information. Like MAE (Masked Autoencoders), embodiments of this disclosure use ViT as the backbone network of the decoder g(·). Due to the independence of the encoder and decoder, the decoder structure can be flexibly designed. The following embodiment implements a lightweight decoder ViT-Base, which, together with the encoder, forms an asymmetric structure, such that the image reconstruction loss can be described as:

[0066]

[0067] because And the entropy term With z k Irrelevant, so maximize The problem is equivalent to minimizing conditional entropy. That is, encourage z k To convey more information about perspective. However, The distribution of is actually difficult to achieve, therefore the assumed prior distribution is... Often used as an approximation For example, Bernoulli distribution, Gaussian distribution, or Laplace distribution. Using the Gaussian distribution as a priori, the derivation of the formula can be further obtained:

[0068]

[0069] Here, g(·) is the decoder network used to calculate the average value of the Gaussian prior distribution, and c is a constant that can be ignored during optimization.

[0070] It is understood that the image processing method based on reconstruction contrastive learning in this disclosure is mainly used for image reconstruction classification tasks, but it is also applicable to converting data into text, speech, language and video sequences.

[0071] This disclosure presents an image processing method based on reconstruction-contrast learning. Utilizing the information bottleneck principle, it defines a minimum sufficient increment and information generalization representation, proposing an additional reconstruction task. This addresses the problems of insufficient compression and overfitting of discriminative information in existing contrastive learning methods, significantly improving performance across various downstream tasks. By combining masking operations and data augmentation, powerful data augmentation results are obtained, enhancing the performance of contrastive learning. Furthermore, an effective reconstruction task is proposed as a regularization term for the discriminative task, thereby improving the model's generalization ability. A novel information metric is presented, providing accurate characterization of the model and techniques, greatly improving the model's interpretability.

[0072] like Figure 2 As shown, another embodiment of this disclosure provides an image processing system based on reconstruction contrastive learning, comprising:

[0073] The acquisition module 210 is used to acquire the image to be processed;

[0074] Augmentation module 220 is used to augment the image to be processed and transfer the image to be processed into a multi-view;

[0075] Masking module 230 is used to perform pixel masking on the image to be processed after data augmentation;

[0076] Encoding module 240 is used to encode the pixel-masked image to be processed using a neural network, and to convert the image to be processed into a hidden representation;

[0077] The decoding module 250 is used to decode the encoded image to obtain a reconstructed image.

[0078] For example, the masking module 230 is specifically used for:

[0079] The pixels of the augmented image to be processed are randomly masked at a preset masking rate.

[0080] For example, the encoding module 240 is specifically used for:

[0081] Repair the image to be processed after pixel masking;

[0082] Mask markers are added to the repaired image to be processed using linear projection and positional embedding.

[0083] For example, the decoding module 250 is specifically used for:

[0084] Add location embeddings to the complete list of tags to encode location information.

[0085] Specifically, an image processing system based on reconstruction contrastive learning according to an embodiment of this disclosure is used to implement the image processing method based on reconstruction contrastive learning described in the above embodiments. The specific implementation process has been described in detail in the above embodiments and will not be repeated here.

[0086] like Figure 3 As shown, another embodiment of this disclosure provides an electronic device, including:

[0087] At least one processor 301; and a memory 302 communicatively connected to the at least one processor 301 for storing one or more programs that, when executed by the at least one processor 301, enable the at least one processor 301 to implement the image processing method based on reconstruction contrast learning described above.

[0088] The memory 302 and processor 301 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 301 and memory 302 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 301 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 301.

[0089] Processor 301 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 302 can be used to store data used by processor 301 during operation.

[0090] Another embodiment of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method based on reconstruction contrastive learning described above.

[0091] The computer-readable storage medium may be included in the systems or electronic devices disclosed herein, or it may exist independently.

[0092] Computer-readable storage media can be any tangible medium that contains or stores a program, and can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, optical fibers, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0093] Computer-readable storage media may also include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code, specific examples of which include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof.

[0094] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. An image processing method based on reconstruction contrastive learning, characterized in that, The method includes: Obtain the image to be processed; Data augmentation is performed on the image to be processed, and the image to be processed is transferred to a multi-view; Pixel masking is applied to the augmented image to be processed. The image to be processed after pixel masking is encoded using a neural network, and the image to be processed is converted into a hidden representation; The encoded image to be processed is decoded to obtain the reconstructed image.

2. The method according to claim 1, characterized in that, The pixel masking of the augmented image to be processed includes: The pixels of the augmented image to be processed are randomly masked at a preset masking rate.

3. The method according to claim 2, characterized in that, The process of encoding the pixel-masked image using a neural network includes: Repair the image to be processed after pixel masking; Mask markers are added to the repaired image to be processed using linear projection and positional embedding.

4. The method according to claim 3, characterized in that, Decoding the encoded image to be processed includes: Add location embeddings to the complete list of tags to encode location information.

5. An image processing system based on reconstruction contrastive learning, characterized in that, The system includes: The acquisition module is used to acquire the image to be processed; An augmentation module is used to perform data augmentation on the image to be processed and transfer the image to be processed into a multi-view; A masking module is used to perform pixel masking on the image to be processed after data augmentation. The encoding module is used to encode the pixel-masked image to be processed using a neural network, and to convert the image to be processed into a hidden representation; The decoding module is used to decode the encoded image to obtain the reconstructed image.

6. The system according to claim 5, characterized in that, The masking module is specifically used for: The pixels of the augmented image to be processed are randomly masked at a preset masking rate.

7. The system according to claim 6, characterized in that, The encoding module is specifically used for: Repair the image to be processed after pixel masking; Mask markers are added to the repaired image to be processed using linear projection and positional embedding.

8. The system according to claim 7, characterized in that, The decoding module is specifically used for: Add location embeddings to the complete list of tags to encode location information.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor is used to store one or more programs that, when executed by the at least one processor, enable the at least one processor to implement the image processing method based on reconstruction contrastive learning as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image processing method based on reconstruction contrastive learning as described in any one of claims 1 to 4.

Images