Multimodal rebalancing self-supervised learning method for rgb-d data

Abstract

The application discloses a multi-modal rebalancing self-supervised learning method for RGB-D data, constructs an RGB-D multi-modal rebalancing self-supervised learning framework to pre-train an encoder with strong modal mask, inputs the pre-trained encoder parameters into a network model of a downstream RGB-D salient object detection task, obtains the trained network model of the downstream RGB-D salient object detection task, and completes inference to output a prediction result; the multi-modal rebalancing self-supervised learning method for RGB-D data comprises an intra-modal contrast learning paradigm and an inter-modal contrast learning paradigm; the application adopts a phased strategy to improve the encoder to sequentially learn single-modal independent features and multi-modal joint features, and simultaneously adopts a mask strategy for strong modes to inhibit the dominance of the strong modes in multi-modal fusion, so as to encourage the weak modes to learn more complementary features, thereby reducing the influence of insufficient complementary feature learning caused by the imbalance between strong and weak modes.

Description

Technical Field

[0001] This invention belongs to the field of multimodal computer vision and self-supervised learning contrastive learning technology, and particularly relates to the RGB-D salient object detection task. It mainly involves a multimodal rebalancing self-supervised learning method for RGB-D data. Background Technology

[0002] Salient Object Detection (SOD) aims to locate the most visually salient objects in a scene and their boundaries. It has wide applications in the field of computer vision, such as semantic segmentation, scene understanding, and pedestrian recognition.

[0003] In recent years, with the advent of depth sensors, combining depth maps rich in spatial information with RGB images containing rich texture and color information can effectively improve performance on salient object detection tasks. Over the past few years, RGB-D salient object detection has developed rapidly, with researchers proposing numerous different datasets and methods. Given the powerful feature learning capabilities and the great success of deep learning tools, various RGB-D salient object detection models based on convolutional neural networks (CNNs) have been proposed. To fully integrate the multi-scale cross-modal aspects of RGB-D pairs, many existing models are typically equipped with multiple cross-modal cross-level fusion paths and modules. Since collecting multimodal data and labeling dense pixel-level tags requires significant manpower and resources, this design introduces considerable complexity to model training. Previous work avoided the scarcity of labeled multimodal data by borrowing pre-trained weights from ImageNet as appropriate initialization for all modalities. However, the domain gap between ImageNet and dense prediction datasets, as well as the modal gap between RGB and depth, often leads to biased initialization and subsequent sub-optimization.

[0004] The rapid development of self-supervised learning (SSL) offers new possibilities for directly overcoming the data shortage problem in multimodal dense prediction. As one of the most promising directions in SSL, contrastive learning (aiming to learn invariant high-level features in image transformations) has been widely applied in multiple fields and has made significant progress in classification tasks. Most existing contrastive learning methods follow the instance recognition paradigm, classifying transformed versions of the input as images from the same source. This concept has been widely adopted and applied to cross-modal recognition of multimodal data, such as speech, video, text, and RGB-D images. Existing multimodal contrastive learning solutions focus on simply learning high-level global representations, but have limited design for multimodal feature fusion and fail to consider the impact of strong and weak modal differences on model pre-training. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies in specifically addressing the RGB-D cross-modal gap and the imbalance of strengths and weaknesses during multimodal data learning. It provides a self-supervised multimodal rebalancing learning method for RGB-D data. This method constructs a self-supervised multimodal rebalancing learning framework for RGB-D data to pre-train the encoder. The pre-trained encoder parameters are then input into the network model for a downstream RGB-D salient object detection task, where the network model undergoes supervised training. This results in a trained network model for the downstream RGB-D salient object detection task, which then performs inference and outputs prediction results. The self-supervised multimodal rebalancing learning framework for RGB-D data includes intra-modal contrastive learning and inter-modal contrastive learning paradigms. The intra-modal contrastive learning paradigm includes RGB modal contrastive learning, deep modal contrastive learning, and intra-modal contrastive loss. The inter-modal contrastive learning paradigm includes a rebalancing fusion encoder and a cross-modal contrastive learning loss. This invention overcomes the problem of insufficient paired RGB-D data labels. Through pre-training, the encoder acquires the ability to efficiently extract RGB-D representations. The phased training strategy of intra-modal and inter-modal pre-training ensures that the encoder acquires joint representations without losing features of individual modalities. The strong modality masking strategy in the rebalanced fusion encoder overcomes the problem of modality training imbalance during pre-training, preventing strong modalities from dominating the pre-training process. The pre-training method of this invention can extract specific cues for individual modalities and joint representations across modalities, thereby promoting the multimodal fusion process in downstream tasks. Unlike previous contrastive learning paradigms, this contrastive learning process fully considers the problem of strong and weak modality training imbalance and balances the independence of individual modal features and the correlation of cross-modal features during training.

[0006] To achieve the above objectives, the technical solution adopted by this invention is as follows: a multimodal rebalancing self-supervised learning method for RGB-D data, which constructs an RGB-D multimodal rebalancing self-supervised framework to pre-train the encoder, encourages the model to learn the representations of different modalities in a balanced manner through strong modality masks, and inputs the pre-trained encoder parameters into the network model of the downstream RGB-D salient object detection task to perform supervised training on the network model, thereby obtaining the trained network model of the downstream RGB-D salient object detection task and completing the inference output prediction results;

[0007] The multimodal rebalancing self-supervised learning framework for RGB-D data includes an intra-modal contrastive learning paradigm and an inter-modal contrastive learning paradigm, wherein...

[0008] The intra-modal contrastive learning paradigm includes RGB modal contrastive learning, deep modal contrastive learning, and intra-modal contrastive learning loss. The RGB modal contrastive learning and deep modal contrastive learning are trained in parallel, and the encoders of both are initially trained using the MOCO framework. The intra-modal contrastive learning loss adopts the InfoNCE loss function.

[0009] The intermodal contrastive learning paradigm includes a rebalancing fusion encoder and a cross-modal contrastive learning loss. The rebalancing fusion encoder suppresses the dominance of RGB modality in the learning process by using strong modality masking, and performs contrastive learning on the multimodal features after fusing RGB and deep features, so that the encoder learns the fused representation of multimodality in a balanced manner. The cross-modal contrastive learning loss adopts the InfoNCE loss function.

[0010] As an improvement to the present invention, this method specifically includes the following steps:

[0011] S1, Obtain the dataset: Obtain the RGB-D cross-modal image dataset, which will be used for pre-training the encoder and training the network model for the downstream RGB-D salient object detection task, respectively, using the RGB-D data multimodal rebalancing self-supervised framework.

[0012] S2, Encoder selection: ResNet50 network structure is adopted as the encoder;

[0013] S3, Construct a self-supervised framework: Construct a multimodal rebalancing self-supervised learning framework for RGB-D data, and perform rebalancing pre-training on the encoder selected in step S2. The pre-training process provides model initialization parameters for the downstream RGB-D salient object detection task.

[0014] S4, Train the network model for the downstream task: Input the self-supervised pre-trained encoder parameters obtained in step S3 into the network model for the downstream RGB-D salient object detection task, and perform supervised training on the network model.

[0015] S5, Predict the output result: Use the downstream RGB-D salient object detection task network model trained in step S4 to complete the inference output of the predicted saliency map.

[0016] As another improvement of the present invention, step S3 specifically includes:

[0017] S31: Each training sample image x will be transformed by two different data augmentation operations t′, t″∈Γ to obtain x′, x″, thereby generating different perspectives of the same sample, where Γ represents the data augmentation operations of random cropping and splicing, rotation and scaling and flipping;

[0018] S32: Each pair of RGB and Depth paired images is then input into the corresponding encoder after undergoing the same data augmentation process as in step S31. θ With momentum encoder e ξ Encode: q = e θ (t′(x′)),k=e ξ (t″(x″)), where q and k are the encoded query representation and key representation, respectively;

[0019] S33: Perform one-stage training on the encoder, with the following intra-modal contrast loss:

[0020]

[0021]

[0022] Where q represents the query sample in a batch, and k + For another key representation that comes from the same anchor point, k i A key representation within a batch;

[0023] S34: Use the pre-trained encoder from step S33 to extract features from each depth modality of the RGB modality. Fuse the feature maps of the RGB modality and the depth map modality to obtain the final feature representation. For an anchor scene, input the two pairs of feature maps obtained after passing the encoder from step S32 into the rebalancing fusion encoder. The rebalancing fusion encoder performs a masking operation on the strong RGB modalities to learn the representations of each modality in a balanced manner. The mask generation process is as follows:

[0024]

[0025] Where c j Let represent the channel attention value of the j-th channel of the RGB feature. `top({.},k)` represents taking the k largest elements from `{.}`. In the experiment, the value of `k` was chosen to be half the number of feature channels, `n`. After the RGB features and depth features are fused using a rebalancing fusion encoder, the fused query representation is obtained. and fusion bond representation

[0026] S35: The second stage of pre-training for cross-modal comparative learning is performed using one-dimensional vectors generated by fusing different modalities. The inter-modal loss function is calculated as follows:

[0027]

[0028] As another improvement of the present invention, in the one-stage training of step S33, the total contrast loss within the modality is:

[0029] L intra =0.5*(Lrgb +L depth )

[0030] Among them, L rgb For the contrastive learning loss of the RGB modality, L depth For contrastive learning loss of deep modalities;

[0031] The total loss of the entire multimodal rebalancing self-supervised learning framework in step S3 is:

[0032] L total =0.5*L intra +0.5*L inter .

[0033] As another improvement of the present invention, in step S32, e ξ The moving average method is used for parameter updates, and the formula is defined as follows:

[0034]

[0035] Where ξ is e ξ The parameter set, For e θ The parameter set.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] (1) This invention adopts the pre-training paradigm of self-supervised learning. Compared with supervised learning, self-supervised learning uses unlabeled RGB-D datasets for pre-training, which overcomes the problem that the labeling of dense prediction tasks in RGB-D is difficult and labeled data is scarce.

[0038] (2) To learn unique features within a modality and complementary features across modalities, this invention designs two contrastive learning modules: intramodal and intermodal. The intramodal contrastive learning module aims to mine independent representations within each modality, emphasizing the differences between modalities. Through parallel single-modal pre-training, intramodal contrastive learning enables the encoder (ResNet50) to learn relatively independent features, fully capturing effective cues within a single modality. The intermodal contrastive learning module aims to mine homogeneous cues and joint representations between modalities, emphasizing the consistency between modalities. This module first fuses the features of two modalities from the same perspective, and then performs contrastive learning to explore homogeneous features between modalities, enabling the encoder (ResNet50) to learn joint representations of multiple modalities.

[0039] (3) Unlike traditional contrastive learning methods, this contrastive learning process takes into account both the independence of single-modal features and the correlation of cross-modal features.

[0040] (4) In order to avoid the influence of insufficient learning of complementary features due to the strong and weak modalities, the present invention adopts a strong modal mask design in the rebalancing fusion encoder in the intermodal contrastive learning. The channel attention map of the RGB modality is masked, and the channel with the greatest influence is removed before contrastive learning is performed. This significantly reduces the training difference between strong and weak modalities, so that the encoder (ResNet50) can learn the joint representation of multimodalities in a balanced manner. Attached Figure Description

[0041] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0042] Figure 2 This is a schematic diagram of the RGB-D cross-modal self-supervised framework of the present invention;

[0043] Figure 3 This is a schematic diagram of the rebalancing fusion encoder of the present invention;

[0044] Figure 4 This is a downstream task effect diagram in Embodiment 1 of the present invention. Detailed Implementation

[0045] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0046] Example 1

[0047] Self-supervised learning methods for multimodal rebalancing of RGB-D data, such as Figure 1 As shown, it includes the following steps:

[0048] Step S1: Obtain RGB-D cross-modal image datasets, which are used for pre-training the encoder and training the network model for the downstream RGB-D salient object detection task, respectively, using the RGB-D data multimodal rebalancing self-supervised learning framework. For the multimodal rebalancing self-supervised learning task for RGB-D data, different unlabeled RGB-D image datasets are collected as the training set for the pre-training framework; for the RGB-D salient object detection task, publicly labeled datasets are used as the training set for supervised learning of the downstream task.

[0049] This embodiment's salient object detection experiment includes 10 publicly available datasets, including NJUD (1985 image pairs), NLPR (1000 image pairs), STERE (1000 image pairs), SIP (929 image pairs), DUT (1200 image pairs), LFSD (100 image pairs), REDWEB-S (3179 image pairs), and COME-15K (15625 image pairs), totaling 25018 RGB-D paired samples as the self-supervised pre-training dataset. In the downstream task stage, we use 1485, 700, and 800 images extracted from NJU2K, NLPR, and DUT, respectively, as the training set for this example, while the test set is the test set from all the remaining datasets.

[0050] Step S2: Use the ResNet50 network structure as the encoder. This invention uses the MoCo contrastive learning self-supervised framework as the baseline framework. Considering the asymmetric structure in the MoCo framework, it is necessary to set up... Figure 2 The two encoders shown have corresponding momentum encoders, with RGB and Depth as anchor samples respectively, and all encoders are set to ResNet50.

[0051] Contrastive learning aims to learn invariant high-level features from transformations of similar samples. Current contrastive learning paradigms are mostly unimodal, failing to consider the imbalance during multimodal feature learning. This paper proposes a multimodal rebalancing self-supervised learning framework for RGB-D data, with the overall structure as follows: Figure 2 As shown, it mainly includes two major designs: intra-modal contrastive learning and inter-modal contrastive learning.

[0052] The intra-modal contrastive learning paradigm includes RGB modal contrastive learning, deep modal contrastive learning, and intra-modal contrastive loss. RGB and deep modal contrastive learning are trained in parallel, with the MOCO framework used for initial training of their encoders. The InfoNCE loss function is employed for intra-modal contrastive learning. This paradigm aims to uncover independent representations within each modality, emphasizing inter-modal differences.

[0053] The intra-modal contrastive learning paradigm is the foundation of this invention's framework. This paradigm uses only intra-modal consistency as the multimodal contrastive loss.

[0054] The intermodal contrastive learning paradigm includes a rebalancing fusion encoder and a cross-modal contrastive learning loss. The rebalancing fusion encoder aims to perform contrastive learning on the multimodal features after fusing RGB and deep features, enabling the encoder to learn the fused multimodal representation in a balanced manner. Its structure is as follows: Figure 3As shown, for RGB and depth modal features, a channel attention mechanism is first applied. Then, the top 50% of channels with higher attention values ​​in the RGB modality are masked to reduce the impact of modal imbalance and fully exploit the features of weak modalities. The mask generation process is as follows:

[0055]

[0056] Where c j Let `top({.},k)` represent the channel attention value of the j-th channel of the RGB feature, where `top({.},k)` represents selecting the k largest elements from `{.}`, and `k` is chosen to be half the number of feature channels `n`. After masking, the modalities of the two modalities are concatenated to obtain a total fused representation for comparative learning. The cross-modal comparative learning loss also adopts the InfoNCE loss function. The inter-modal comparative learning paradigm is used to mine joint representations between modalities to better adapt to downstream multimodal tasks. Following the intra-modal comparative learning framework, features from two modalities under the same perspective are first fused, followed by comparative learning to explore homogeneous features between modalities. Addressing the deficiency that the RGB modality is significantly stronger than the deep modality during pre-training, a strong modality masking comparative learning method is designed to evenly mine relevant cues between the two modalities. This comparative learning process takes into account both the independence of single-modal features and the correlation of cross-modal features, significantly promoting the multimodal feature extraction and fusion process of downstream tasks.

[0057] Step S3: Construct a self-supervised framework: Construct a multimodal rebalancing self-supervised learning framework for RGB-D data. Pre-train the encoder selected in step S2. The pre-training process specifically includes:

[0058] S31: Each training sample image x will be transformed by two different data augmentation operations t′, t″∈Γ to obtain x′, x″, thereby generating different perspectives of the same sample, where Γ represents the data augmentation operations of random cropping and splicing, rotation and scaling and flipping;

[0059] S32: Each pair of RGB and Depth paired images is then input into the corresponding encoder after undergoing the same data augmentation process as in step S31. θ With momentum encoder e ξ Encode: q = e θ (t′(x′)),k=e ξ (t″(x″)), where q and k are the encoded query representation and key representation, respectively;

[0060] S33: Perform one-stage training on the encoder, with the following intra-modal contrast loss:

[0061]

[0062]

[0063] Where q represents the query sample in a batch, and k + For another key representation that comes from the same anchor point, k i A key representation within a batch;

[0064] The total contrast loss within a mode is:

[0065] L intra =0.5*(L RGB +L Depth );

[0066] Save the encoder parameters after the first stage of training for use in the subsequent second stage of pre-training;

[0067] S34: The encoder pre-trained in the first stage is used to extract features of each depth mode of the RGB modality. The feature maps of the RGB modality and the depth map modality are fused to obtain the final feature representation. For an anchor scene, the two pairs of feature maps obtained after the image pair passes through the encoder in step S32 are input into the rebalancing fusion encoder. The rebalancing fusion encoder performs a masking operation on the strong RGB modality to learn the representation of each modality in a balanced manner. The mask generation process is as follows:

[0068]

[0069] Where c j Let represent the channel attention value of the j-th channel of the RGB feature. `top({.},k)` represents taking the k largest elements from `{.}`. In the experiment, the value of `k` was chosen to be half the number of feature channels, `n`. After the RGB features and depth features are fused using a rebalancing fusion encoder, the fused query representation is obtained. and fusion bond representation

[0070] S35: Use the one-dimensional vector generated by fusing different modalities for cross-modal contrastive learning pre-training. The inter-modal loss function is calculated as follows:

[0071]

[0072] The total loss of the entire multimodal rebalancing self-supervised learning framework is:

[0073] L total =0.5*L intra +0.5*L inter .

[0074] In this embodiment, the parameters of the pre-training framework are set as follows: learning rate 0.07, negative sample pool number of samples. The value is 16384, the momentum coefficient m of the momentum encoder is 0.9, and the temperature hyperparameter τ is set to 0.1. The pre-training dataset of this invention is much smaller than ImageNet, with 200 training rounds in the first stage and 50 training rounds in the second stage, and 64 mini-batch samples input on a single GPU.

[0075] Step S4: Supervised pre-training for the RGB-D salient object detection task: Extract the parameters from the RGB and Depth encoders in Step S3 and input them into the encoder of the downstream test network as initialization parameters for the downstream task. Select an existing RGB-D salient object detection model as the downstream network model for supervised training, keeping the training parameters consistent with the original downstream network.

[0076] Step S5: Prediction result output: Based on different RGB-D tasks, use the downstream RGB-D salient object detection network model trained in step S4 to complete the inference and output the prediction result.

[0077] This invention employs a phased strategy to improve the encoder's ability to sequentially learn independent single-modal features and joint multimodal features. Specifically, it first performs intra-modal contrastive learning training, followed by joint training of inter-modal and intra-modal contrastive learning. This invention balances the independence of single-modal features with the correlation of cross-modal features, learning not only intra-modal features but also adaptively learning complementary features between modalities, while mitigating the impact of insufficient complementary feature learning caused by modality imbalance.

[0078] Figure 4 This demonstrates the effectiveness of the present invention in the RGB-D salient object detection task. The first and second columns show the input RGB-depth image pairs, the third column shows the ground truth results for salient object detection and semantic segmentation, and the fourth, fifth, and sixth columns show the results of the present invention's method, the supervised method, and the baseline method, respectively. Figure 4 As shown, the "baseline model" based on traditional contrastive learning focuses more on learning global representations and is not good at capturing local details. This invention, on the other hand, achieves excellent results in both global localization and capturing object details, and achieves performance comparable to ImageNet supervised pre-training schemes.

[0079] Test case

[0080] Performance testing of the self-supervised pre-trained model: To verify the effectiveness of the pre-training parameters of this invention, this test case is tested on the RGB-D salient object detection task and compared with several advanced models.

[0081] The pre-training method used for comparison is as follows:

[0082] Supervised: The downstream backbone network is initialized with parameters pre-trained using ImageNet data.

[0083] Random initialization: Initialize the downstream encoder with random parameters;

[0084] Self-supervised + fine-tuning: The multimodal rebalancing self-supervised learning method for RGB-D data proposed in this invention is used to pre-train the RGB and deep modal encoders, and the parameters are used to initialize the downstream encoders;

[0085] The evaluation metrics for the RGB-D salient object detection task are as follows:

[0086]

[0087] The MAE metric measures the absolute error between the predicted map and the label map; a lower value indicates a better model.

[0088] S α =α*S o +(1-α)*S r

[0089] S α It is a well-known metric for measuring the extraction of foreground and background from an image by a model. It is also applicable to salient object detection tasks and is usually set to 0.5 as an adjustment parameter.

[0090]

[0091] F β β is a common metric for measuring the accuracy and recall of a comprehensive model. In salient object detection tasks, where more attention is paid to the accuracy of the prediction map, the β value is set to 0.3.

[0092]

[0093] E φ It is often used to capture pixel-level data and matching information of local pixels, while φ FM It is an enhanced alignment matrix. Unlike MAE, higher values ​​for the last three metrics indicate a better model.

[0094] The comparison results of the quantitative comparison methods of the present invention with two other initialization strategies and three other advanced models are shown in the table below:

[0095]

[0096] As can be seen from the table above, the pre-training framework of this invention far surpasses the effect of random initialization and exhibits performance comparable to ImageNet pre-trained models. At the same time, the pre-training framework of this invention is also competitive with some current state-of-the-art models.

[0097] First, this invention employs an intra-modal contrastive learning method, using a momentum encoder for single-modal training, which effectively learns intra-modal features. Second, this invention designs an inter-modal contrastive learning module, and for the first time, designs a rebalanced fusion encoder and a fusion momentum encoder for multimodal image contrastive learning. The fusion operation avoids the exclusionary effect of hard-to-handle sample pairs when using alignment operations. Simultaneously, for the first time, it enhances the model's ability to learn complementary features between modalities through RGB modal masking operations, especially the richer features of strong modalities compared to weak modalities. Furthermore, by adopting a phased learning strategy—first learning intra-modal features, then simultaneously learning intra-modal and inter-modal features—the quality of the learned feature representations is significantly improved. Extensive comparative experiments demonstrate and verify the rationality of the proposed multimodal image contrastive learning framework and phased learning strategy.

[0098] The multimodal rebalancing self-supervised learning method for RGB-D data provided in this invention can widely offer high-performance pre-training parameters for downstream models such as pixel-level detection. Specifically, in industry, companies with a large number of unlabeled RGB-D images can maximize the advantages of this contrastive learning framework to obtain models that meet the needs of their specific domains and are close to supervised pre-training.

[0099] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-modal re-balancing self-supervised learning method for RGB-D data, characterized in that The RGB-D data multi-modal rebalancing self-supervision framework is used to pre-train an encoder, the pre-trained encoder parameters are input into a network model of a downstream RGB-D salient object detection task after suppressing strong modal features through mask pre-training, the network model of the downstream RGB-D salient object detection task is supervised trained to obtain a trained network model of the downstream RGB-D salient object detection task, and a prediction result is output through inference; The RGB-D data multi-modal rebalancing self-supervision framework comprises an intra-modal contrast learning paradigm and an inter-modal contrast learning paradigm, wherein, The intra-modal contrast learning paradigm comprises RGB modal contrast learning, depth modal contrast learning and intra-modal contrast learning loss; the RGB modal contrast learning and the depth modal contrast learning are trained in parallel, and the MOCO framework is used to preliminarily train the encoders of the two; the intra-modal contrast learning loss adopts an InfoNCE loss function, and specifically: ； wherein, is the contrastive learning loss for the RGB modality, is the contrastive learning loss for the depth modality; The inter-modal contrast learning paradigm comprises a rebalancing fusion encoder and a cross-modal contrast learning loss; the rebalancing fusion encoder suppresses the dominant position of the RGB modal in the learning process through a strong modal mask method, and performs contrast learning on the multi-modal features after fusing the RGB and depth features, so that the encoder learns the fusion representation of the multi-modal in a balanced manner; the rebalancing fusion encoder adopts a mask operation on the RGB strong modal to learn the modal representation in a balanced manner, and the mask generation process is as follows: ； wherein represents the channel attention value of the jth channel of the RGB feature, top({.}, k) represents taking the maximum k elements in {.}, and k is selected as half of the number of feature channels n, the modalities are spliced after masking, to obtain the total fusion representation for comparison learning, and after the RGB feature and the depth feature complete feature fusion through the rebalanced fusion encoder, the fusion query representation is obtained and the fusion key representation ; The cross-modal contrast learning loss adopts an InfoNCE loss function, and specifically: ； The total loss of the entire multi-modal rebalancing self-supervision learning framework is: 。 2. The RGB-D data oriented multi-modal re-balancing self-supervised learning method of claim 1, wherein, Specifically comprising the following steps: S1, obtaining a data set: obtaining an RGB-D cross-modal image data set, which is used for pre-training of an encoder of an RGB-D data multi-modal rebalancing self-supervision framework and training of a network model of a downstream RGB-D salient object detection task; S2, selecting an encoder: adopting a ResNet50 network structure as the encoder; S3, constructing a self-supervision framework: constructing an RGB-D data multi-modal rebalancing self-supervision learning framework, rebalancing pre-training the encoder selected in step S2, and the pre-training process provides model initialization parameters for a downstream RGB-D salient object detection task; S4, training a network model of a downstream task: inputting the self-supervised pre-trained encoder parameters obtained in step S3 into a network model of a downstream RGB-D salient object detection task, and supervising training of the network model; S5, predicting an output result: using the trained downstream RGB-D salient object detection task network model in step S4 to complete inference and output a prediction saliency map.

3. The RGB-D data oriented multi-modal re-balancing self-supervised learning method of claim 2, wherein: The step S3 specifically comprises: S31: each training sample picture will be transformed by two different data augmentation operations to obtain , thus generating different perspectives of the same sample, wherein denotes digital augmentation operations random crop, rotation, scaling and flipping; S32: Each pair of RGB and Depth paired picture is input into the corresponding encoder after the same data augmentation of step S31 respectively with momentum encoder encoding: wherein and are the generated query representation and key representation after encoding; S33: one-stage training of the encoder, and the intra-modal contrast loss is as follows: ； ； wherein, is a key representation in a batch, is another key representation from the same anchor point as it, is a key representation in a batch; S34: using the pre-trained encoder in step S33 to extract RGB modal and depth modal features, fusing the feature maps of the RGB modal and the depth map modal to obtain a final feature representation, and inputting two pairs of feature maps obtained after encoding the picture pairs through the encoder in step S32 into the rebalancing fusion encoder for an anchor scene; S35: a second stage pre-training of cross-modal contrast learning is performed by using the one-dimensional vector generated by the fusion of different modalities.

4. The RGB-D data oriented multi-modal re-balancing self-supervised learning method of claim 2, wherein: In the step S32, The parameter updating is performed by using the moving average method, and the formula is defined as follows: ； wherein is a parameter set of is a parameter set of is a parameter set of is a parameter set of

Images