Unsupervised salient object detection method and system based on multi-task cooperative training

Abstract

The application discloses a multi-task cooperative training-based unsupervised salient object detection method and system, and belongs to the technical field of image processing. The multi-task cooperative training-based unsupervised salient object detection method comprises the following steps: acquiring multi-modal images without manual annotation and performing data enhancement; generating high-quality pseudo labels automatically through a cross-modal cooperative pseudo label generation network via feature coding, salient information extraction and noise suppression, multi-scale fusion and pseudo label optimization; taking the high-quality pseudo labels as a supervision signal, training a unified modal detector by using a double-enhanced consistency strategy, and completing cross-modal feature interaction and global semantic constraint; and performing pretreatment and forward inference on a to-be-detected image, and outputting a high-precision salient object detection result. According to the application, high-quality pseudo labels of multi-modal data can be automatically generated and cross-task cooperative learning can be realized under the condition of no manual annotation, and the precision and robustness of salient object detection are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an unsupervised salient target detection method and system based on multi-task collaborative training. Background Technology

[0002] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art.

[0003] With the rapid development of deep learning and computer vision technologies, salient object detection (SOD) has become a fundamental core task in fields such as image segmentation, object tracking, behavior recognition, video content understanding, autonomous driving, medical image analysis, and robot visual perception. It aims to automatically locate and segment the foreground object with the highest visual attention from complex scenes.

[0004] Current mainstream salient object detection methods are mostly based on fully supervised learning, which relies on large-scale pixel-level labeled data. However, high-quality labeling is time-consuming, labor-intensive, and costly, and the labeling difficulty is even greater in multimodal scenarios (such as RGB-D and RGB-T), severely limiting the model's generalization ability and practical deployment efficiency. To reduce reliance on manual labeling, unsupervised salient object detection (USOD) has gradually gained attention. Existing unsupervised methods typically generate pseudo-labels through low-level image features, clustering priors, cross-modal consistency, or self-learning strategies, and then use these pseudo-labels as supervision signals to complete network training, which can reduce data dependence to some extent.

[0005] However, the inventors found that the pseudo-label generation process in existing unsupervised methods is prone to introducing noise, lacks effective salient information extraction and noise suppression mechanisms, and is difficult to generate high-confidence, structurally complete supervision signals. Moreover, most existing methods only model independently for a single modality task, and do not fully explore the common features and correlation information across modalities and tasks, resulting in weak cross-modal adaptability and generalization ability. In addition, without manual annotation, the model is difficult to learn stable and complete salient representations and is easily affected by factors such as background interference, illumination changes, occlusion, and scale changes, leading to problems such as discontinuous salient targets, rough boundaries, and missed detection of small targets. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an unsupervised salient target detection method and system based on multi-task collaborative training. This method can achieve high-quality automatic generation of pseudo-labels for multimodal data and cross-task collaborative learning without manual annotation, while improving the accuracy and robustness of salient target detection.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the technical solution of the present invention provides an unsupervised salient target detection method based on multi-task collaborative training, including: We acquire unannotated multimodal images and perform data augmentation on them. Based on the original images and their augmented images, we construct a multi-task collaborative training dataset. Based on a cross-modal collaborative pseudo-label generation network, salient information is extracted from multimodal images to generate salient features. Residual enhancement is performed based on the salient features to generate initial pseudo-labels. Pixel-level optimization is performed on the initial pseudo-labels to generate high-quality pseudo-labels. Using the original image and its enhanced image from the multi-task collaborative training dataset as input images, the input images are processed by a unified modality detector through cross-modal feature complementation, global semantic perception, and multi-level decoding to generate a first saliency prediction map for the original image and a second saliency prediction map for the enhanced image. A loss function is constructed based on the first saliency prediction map, the second saliency prediction map, and high-quality pseudo-labels, and the unified modality detector is trained based on the loss function. The image to be detected is input into the trained unified modality detector, which outputs a high-precision salient target prediction map.

[0008] In at least one embodiment, the multimodal image includes an RGB image, a depth map, and a thermal infrared image; the acquired RGB image, depth map, and thermal infrared image are sized to a preset size and mean-variance normalization is performed. Data augmentation of multimodal images specifically includes applying random flipping and random scaling data augmentation operations to the acquired multimodal images to obtain augmented multimodal images.

[0009] In at least one embodiment, the cross-modal collaborative pseudo-label generation network includes a feature encoding unit, a salient information extraction unit, a pseudo-label generation unit, and a pseudo-label optimization unit connected in sequence.

[0010] In at least one embodiment, generating high-quality pseudo-tags specifically includes: Multi-level feature extraction is performed on the input multimodal image to obtain multiple level feature maps; Extract salient information from mid-to-high-level feature maps and output enhanced salient features; The highest-level feature map is used to generate a global saliency prior. Based on the saliency features at each level and the global saliency prior, residual enhancement is performed and initial pseudo-labels are generated. Adaptive threshold binarization and normalization are performed sequentially on the initial pseudo-labels. The normalized initial pseudo-labels, the RGB image that restores the original representation, and other modal images are input into a conditional random field for pixel-level optimization. Noise is removed by bilateral filtering, and high-quality pseudo-labels are output.

[0011] In at least one embodiment, salient information is extracted from the mid-to-high-level feature maps, and enhanced salient features are output. Specifically, the following steps are taken: the mid-to-high-level features output by the feature encoding unit are used as input features; global statistical features are fused through max pooling and average pooling operations; attention weights are generated through channel remapping and activation functions; the attention weights are multiplied and added to the input features in a weighted manner to supplement the original features and output enhanced salient information.

[0012] In at least one embodiment, the unified modality detector includes a parallel encoder, a cross-modal complementarity module, a global semantic awareness module, and a decoding module; wherein, the global semantic awareness module captures multi-scale global semantic information through multi-branch grouped dilated convolution, and outputs global semantic features by combining cross-modal feature interaction and residual connection, which are used to constrain the semantic consistency of the decoding process.

[0013] In at least one embodiment, the input image is subjected to cross-modal feature complementation, global semantic awareness, and multi-level decoding output based on a unified modality detector, specifically including: Multi-scale features are extracted layer by layer from RGB images and their auxiliary modal images to obtain RGB features and auxiliary modal features at multiple levels; Based on the highest-level RGB modal features and auxiliary modal features, a fused high-level semantic feature is generated; after convolution and PReLU activation, a multi-branch grouped dilated convolution is performed on the fused high-level semantic feature, and after residual connection and CBR operation, the global semantic feature is output. At each level, multi-scale convolution is performed on RGB features and auxiliary modal features to extract features. The two modal features of each scale convolution branch are multiplied to generate cross-modal features. After being refined by depthwise separable convolution, they are concatenated and fused. At the same time, high-level semantic features are introduced and added to the residual of the original fused features. The cross-modal fused features are then output through concatenation and CBR operation. A top-down decoding structure is adopted, starting with the highest-level cross-modal fusion features as the initial input, introducing high-level semantic features for weighted modulation, upsampling, concatenating with the next-level cross-modal fusion features, and then processing with CBR. This process is iterated layer by layer to the lowest level, and finally processed by 1×1 convolution and Sigmoid activation to output a saliency prediction map of the same size as the input.

[0014] In at least one embodiment, the loss function is specifically a weighted sum of the intersection-union loss and the consistency loss.

[0015] In at least one embodiment, the crossover ratio loss includes the crossover ratio loss of the first saliency prediction map and the high-quality pseudo-label, and the crossover ratio loss of the second saliency prediction map and the high-quality pseudo-label. The consistency loss is specifically the consistency loss between the first significance prediction map and the second significance prediction map.

[0016] Secondly, the technical solution of the present invention also provides an unsupervised salient target detection system based on multi-task collaborative training, comprising: The data acquisition and preprocessing module is configured to: acquire unannotated multimodal images, perform data augmentation on the multimodal images, and construct a multi-task collaborative training dataset based on the original images and their augmented images; The cross-modal pseudo-label generation module is configured to: extract salient information from multimodal images based on a cross-modal collaborative pseudo-label generation network to generate salient features, perform residual enhancement based on salient features to generate initial pseudo-labels, and perform pixel-level optimization on the initial pseudo-labels to generate high-quality pseudo-labels. The detector training module is configured to: take the original image and its enhanced image from the multi-task collaborative training dataset as input images, respectively, perform cross-modal feature complementation, global semantic perception and multi-level decoding on the input images based on the unified modality detector, and generate a first saliency prediction map corresponding to the original image and a second saliency prediction map corresponding to the enhanced image; construct a loss function based on the first saliency prediction map, the second saliency prediction map and high-quality pseudo-labels, and train the unified modality detector based on the loss function; The salient target prediction module is configured to input the image to be detected into a trained unified modality detector and output a high-precision salient target prediction map.

[0017] The beneficial effects of the above-described technical solution of the present invention are as follows: 1) The unsupervised salient target detection method based on multi-task collaborative training of the present invention does not require any pixel-level manual annotation. It automatically outputs high-quality pseudo-labels as supervision signals through a cross-modal collaborative pseudo-label generation network, breaking through the dependence of fully supervised learning on expensive labeled data and significantly reducing data acquisition costs and manual workload. At the same time, the method realizes unified modeling of multiple modalities and joint training of multiple tasks. Through parameter sharing and gradient collaborative updates, it fully explores the common features of cross-modal and cross-task, significantly improving the model's cross-modal adaptability and generalization ability in complex scenarios. It has the characteristics of high pseudo-label quality, strong cross-modal adaptability, and excellent detection accuracy, which can effectively reduce data annotation costs and improve model generalization ability.

[0018] 2) This invention does not require any pixel-level manual annotation data. Relying on a cross-modal collaborative pseudo-label generation network, it automatically generates high-quality pseudo-labels as supervision signals through salient information extraction, pseudo-label generation, and optimization of the pseudo-label generation process. This reduces the time and manpower costs of data annotation in multimodal scenarios and solves the core pain points of poor pseudo-label quality and insufficient supervision reliability in existing unsupervised methods. It provides highly reliable supervision signals for subsequent detector training.

[0019] 3) This invention realizes collaborative training of three modal salient target detection tasks: RGB, RGB-D, and RGB-T. Through cross-modal complementarity mechanism and multi-task joint training strategy, it realizes network parameter sharing and gradient collaborative update of the three tasks, fully explores the common features and correlation information between cross-modal and cross-task, and enables the model to flexibly adapt to different modal inputs and maintain excellent detection performance in complex scenarios.

[0020] 4) The detection system of the present invention adopts a modular architecture. The data preprocessing, pseudo-label generation, detector training and inference prediction units are independently encapsulated and work together efficiently, which can meet the actual deployment requirements and can be widely adapted to various computer vision application scenarios such as intelligent monitoring, autonomous driving and robot vision. It has strong practicality and engineering application value. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a schematic diagram of the unsupervised salient target detection method based on multi-task collaborative training disclosed in Embodiment 1 of the present invention; Figure 2 This is a flowchart of the unsupervised salient target detection method based on multi-task collaborative training disclosed in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the cross-modal collaborative pseudo-label generation network structure disclosed in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the salient information extraction unit structure disclosed in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the pseudo-tag generation unit structure disclosed in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of the unified modality detector structure disclosed in Embodiment 1 of the present invention; Figure 7 This is a schematic diagram of the global semantic perception module structure disclosed in Embodiment 1 of the present invention; Figure 8This is a schematic diagram of the cross-modal complementary module structure disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0023] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] As described in the background section, the purpose of this invention is to overcome the shortcomings of the prior art and provide an unsupervised salient target detection method and system based on multi-task collaborative training. This method can achieve high-quality automatic generation of pseudo-labels for multimodal data and cross-task collaborative learning without manual annotation, while improving the accuracy and robustness of salient target detection.

[0025] Example 1 In a typical embodiment of the present invention, such as Figures 1 to 8 As shown in the figure, this embodiment discloses an unsupervised salient target detection method based on multi-task collaborative training, which specifically includes the following steps: S1. Obtain unannotated multimodal images and perform data augmentation on the multimodal images. Construct a multi-task collaborative training dataset based on the original images and their augmented images. S2. Based on the cross-modal collaborative pseudo-label generation network, salient information is extracted from multimodal images to generate salient features. Residual enhancement is performed based on the salient features to generate initial pseudo-labels. Pixel-level optimization is performed on the initial pseudo-labels to generate high-quality pseudo-labels. S3. Using the original image and its enhanced image from the multi-task collaborative training dataset as input images, the input images are processed by a unified modality detector through cross-modal feature complementation, global semantic perception, and multi-level decoding to generate a first saliency prediction map for the original image and a second saliency prediction map for the enhanced image. A loss function is constructed based on the first saliency prediction map, the second saliency prediction map, and high-quality pseudo-labels. The unified modality detector is then trained based on the loss function. S4. Input the image to be detected into the trained unified modality detector, and output a high-precision salient target prediction map.

[0026] The following detailed description of the above-mentioned unsupervised salient target detection method based on multi-task collaborative training, with reference to specific implementation methods, is provided below.

[0027] S1. Obtain unannotated multimodal images and perform data augmentation on the multimodal images. Construct a multi-task collaborative training dataset based on the original images and their augmented images.

[0028] In this step, multimodal images without manual pixel-level annotation are acquired, including RGB images, depth maps, and thermal infrared images, and the acquired multimodal images are preprocessed.

[0029] The preprocessing of multimodal images includes: aligning the RGB image, depth map, and thermal infrared image to a preset size, and performing mean-variance normalization.

[0030] Simultaneously, data augmentation operations involving random flipping and random scaling are applied to the acquired multimodal images to obtain enhanced multimodal images.

[0031] The preprocessed original images and their enhanced images are divided into training and validation sets to construct a multi-task collaborative training dataset.

[0032] As an example, RGB images, depth maps, and thermal infrared images are obtained from three types of publicly available datasets: RGB, RGB-D, and RGB-T. All images are sized to 320×320, mean-variance normalization is performed, and random horizontal flipping and random scaling are applied for data augmentation. The processed original images and their augmented images are divided into training and validation sets in an 8:2 ratio to construct a multi-task collaborative training set.

[0033] S2. Based on the cross-modal collaborative pseudo-label generation network, salient information is extracted from multimodal images to generate salient features. Residual enhancement is performed based on the salient features to generate initial pseudo-labels. Pixel-level optimization is performed on the initial pseudo-labels to generate high-quality pseudo-labels.

[0034] In this step, the preprocessed multimodal image is input into a cross-modal collaborative pseudo-label generation network. This network includes a feature encoding unit, a salient information extraction unit, a pseudo-label generation unit, and a pseudo-label optimization unit. Through multi-level feature extraction, salient information extraction, multi-scale fusion, and refinement enhancement, it outputs high-quality pseudo-labels, such as... Figure 3 As shown.

[0035] S21. Multi-level feature extraction based on feature coding units.

[0036] In this step, the feature encoding unit uses ResNet-50 as the encoder to perform multi-level feature extraction on the input multimodal image, resulting in five levels of feature maps, denoted as: .in, and These are shallow features with high resolution, capable of capturing local details such as edges and textures. and These are deep features, which have lower resolution but contain rich global semantic information.

[0037] S22. Based on the salient information extraction unit, extract salient information from the mid-to-high-level features and output the enhanced salient features.

[0038] In this step, such as Figure 4 As shown, the mid-to-high-level feature maps output by the feature encoding unit. and Input features as salient information extraction units The three input features are processed by global average pooling and global max pooling respectively, and then fused to generate global statistical features. After channel remapping through 1×1 convolution, the feature distribution is adjusted by group normalization, and then attention weights are generated by sigmoid activation. Specifically, it is expressed as follows:

[0039] In the formula, This represents the generated attention weights; Indicates input features, ; Represents the Sigmoid function; It is a 1×1 convolution; For group normalization operations; and These are global average pooling and global max pooling operations, respectively.

[0040] Attention weight Input features Multiply element by element and then combine with the input features The summation outputs the enhanced salient features, which can be specifically represented as:

[0041] In the formula, This indicates the enhanced salient features.

[0042] S23. Perform residual enhancement on salient features based on the pseudo-label generation unit and generate initial pseudo-labels.

[0043] In this step, such as Figure 5 As shown, firstly, the highest-level feature map output by the feature encoding unit is processed. After performing global max pooling and sigmoid function processing, it is then compared with the highest-level feature map. Multiplying and performing bilinear upsampling generates a globally significant prior, which can be specifically expressed as:

[0044] In the formula, Indicates global significance prior; This indicates a bilinear upsampling operation; Represents the Sigmoid function; This is a global max pooling operation.

[0045] Then, the multi-level salient features output by the salient information extraction unit are... Upsampled to a uniform size, the difference between the salient features at each level after upsampling and the global salient prior is calculated. The difference results at each level are then processed by a 3×3 convolution and a sigmoid function to generate spatial gating weights, specifically as follows:

[0046] In the formula, For spatial gating weights, ; Conv 3 represents a 3×3 convolution; This represents the Sigmoid function.

[0047] Subsequently, the upsampled salient features Spatial gating weights Multiply and then combine with the upsampled salient features The addition is used to complete residual enhancement and generate enhancement features at each level, which can be specifically represented as follows:

[0048] In the formula, To enhance features, .

[0049] Next, the enhanced features at each level are concatenated with the global saliency prior, specifically represented as follows:

[0050] In the formula, This is for splicing operations.

[0051] Finally, the channel splicing results A 3×3 convolution dimensionality reduction and integration process is performed, the feature distribution is adjusted through normalization, and the foreground-background inversion problem is corrected through a foreground correction function to output the initial pseudo-labels, specifically as follows:

[0052] In the formula, Indicates the initial pseudo-tag; This is a normalization operation; This is the foreground correction function.

[0053] S24. Pixel-level optimization of the initial pseudo-labels is performed based on the pseudo-label optimization unit to generate high-quality pseudo-labels.

[0054] In this step, the pseudo-label optimization unit adopts a combination strategy of adaptive threshold segmentation, conditional random field (CRF) and bilateral filtering, and sequentially performs binarization, normalization, inverse normalization, pixel-level thinning and noise filtering operations to output high-quality thinned pseudo-labels.

[0055] Specifically, the optimal threshold is first automatically calculated using the Otsu method for the initial pseudo-labels. Perform binarization to generate a binary image of the initial pseudo-label, which can be represented as follows:

[0056] In the formula, The binary image of the initial pseudo-label; This is the Otsu algorithm.

[0057] Then, the binary image of the initial pseudo-label. Normalization is performed, specifically as follows:

[0058] In the formula, This is the result of normalization; and These are functions for calculating the maximum and minimum values, respectively.

[0059] Next, denormalization is performed on the input RGB image and other modal images to restore the original image representation, specifically as follows:

[0060] In the formula, This represents the raw, unnormalized RGB input image; This represents a standardized RGB image. This represents the original, unnormalized input image from other modalities. Represents other modal images after standardization; Indicates standard deviation, ; This represents the mean. .

[0061] Subsequently, the normalized initial pseudo-labels, the original RGB image recovering the original representation, and other modal images are input into a Conditional Random Field (CRF) model to perform pixel-level optimization on the initial pseudo-labels, resulting in optimized pseudo-labels, specifically represented as follows:

[0062] In the formula, This represents the optimized pseudo-tag; This represents a conditional random field model; This represents the maximum a posteriori criterion.

[0063] Finally, bilateral filtering is used to further remove noise and output high-quality pseudo-labels, specifically as follows:

[0064] In the formula, Indicates a high-quality pseudo-label; This is a bilateral filtering operation.

[0065] S3. Using the original image and its enhanced image from the multi-task collaborative training dataset as input images, the input images are processed by a unified modality detector to perform cross-modal feature complementation, global semantic perception, and multi-level decoding output, generating a first saliency prediction map for the original image and a second saliency prediction map for the enhanced image. A loss function is constructed based on the first saliency prediction map, the second saliency prediction map, and high-quality pseudo-labels, and the unified modality detector is trained based on the loss function.

[0066] like Figure 6 As shown, in this step, a globally semantically guided unified modality detector (UMD) is constructed. High-quality pseudo-labels are used as supervision signals, and the original and enhanced images from the multi-task collaborative training dataset are used as input images. A dual-enhancement consistency strategy is employed to train the UMD, achieving cross-modal feature complementation, global semantic awareness, and multi-level decoding output. The UMD includes a parallel encoder, a cross-modal complementation module, a global semantic awareness module, and a decoding module. In the UMD, the processing of the original and enhanced images is completely consistent; therefore, the original image is used as an input example to illustrate the training process of the UMD.

[0067] S31. Extracting multimodal features based on a parallel encoder.

[0068] In this step, a dual-stream PVTv2 encoder is used as the encoder for the RGB image and its auxiliary modal images (RGB, depth map, and thermal infrared image). Multi-scale features are extracted layer by layer to obtain RGB features at four levels. and auxiliary modal features .

[0069] S32. Based on the global semantic awareness module, output global semantic features to constrain the semantic consistency of the decoding process.

[0070] In this step, such as Figure 7 As shown, the highest-level RGB modal features and auxiliary modal features output by the parallel encoder are multiplied, added, and concatenated, respectively. The results of the multiplication, addition, and concatenation operations are then concatenated to fuse them, generating the fused high-level semantic features, specifically represented as follows:

[0071] In the formula, These are the high-level semantic features after fusion; For splicing operations; and For modal identification, Indicates RGB mode, Indicates auxiliary mode.

[0072] Then, the fused high-level semantic features are compressed by a 1×1 convolution to reduce the number of channels, and non-linearity is introduced through the PReLU activation function, specifically as follows:

[0073] In the formula, It is a 1×1 convolution; This represents the PReLU function.

[0074] Next, multi-branch grouped dilated convolution is performed to capture multi-scale global semantic information, and the output of each branch is refined by 1×1 convolution. The dilation rates of the multi-branch grouped dilated convolution are 1, 3, and 6, respectively. This process is specifically represented as follows:

[0075] In the formula, It is a 1×1 convolution; The void ratio is Grouped dilated convolution.

[0076] Finally, the high-level semantic features are fused with those obtained through 1×1 convolution. Perform residual connections, and after CBR operation, output global semantic features, specifically represented as follows:

[0077] In the formula, For global semantic features; This represents a combination of 3×3 convolution operation, batch normalization operation, and ReLU function.

[0078] S33. Based on the cross-modal complementarity module, perform cross-modal feature complementarity on multimodal features to generate cross-modal fusion features.

[0079] In this step, such as Figure 8 As shown, at each level, pooling attention is performed on RGB features and auxiliary modal features to enhance modal representation. Then, batch normalization is used to stabilize the feature distribution, and features are extracted through multi-scale convolution branches. The kernel sizes of the multi-scale convolutions are 1, 3, 5, and 9, respectively. This process is specifically represented as follows:

[0080] In the formula, For the first At the hierarchical level, the convolution scale is The branch output Modal features, The first output of the parallel encoder hierarchical Modal features, , Represents a mode, which can be taken as a criterion. or ,Pick Indicates that it is an RGB feature, take This indicates that it is an auxiliary modal feature; The kernel size is k Convolution operation, ; For batch normalization operations, This is for pooling attention operations.

[0081] Then, at each level, the two RGB features and auxiliary modal features from each scale convolutional branch are multiplied to perform cross-modal fusion, generating cross-modal features. These features are then further refined through depthwise separable convolutions, followed by batch normalization. This process is specifically represented as follows:

[0082] In the formula, These are the normalized cross-modal features; For depthwise separable convolution; This is for batch normalization operations.

[0083] Next, the cross-modal features normalized from the convolutional branches at each scale are fused by concatenation, specifically as follows:

[0084] In the formula, These are the initial features after cross-modal fusion.

[0085] The concatenated and fused data undergoes a CBR operation to adjust the number of channels, while high-level semantic features are introduced for weighted compensation. The highest-level CCM module introduces global semantic features. Finally, these are added to the residual of the original fused features, as shown below:

[0086] In the formula, These are the high-level semantic features after weighted compensation. For the Sigmoid function; Finally, the cross-modal fused features are output through concatenation and CBR operations, specifically as follows:

[0087] In the formula, For the first Hierarchical cross-modal fusion features.

[0088] S34. Based on the decoding module, perform multi-level decoding output to generate a saliency prediction map corresponding to the input image.

[0089] In this step, a top-down decoding structure is adopted, using the highest-level cross-modal feature fusion. Using this as the initial input, high-level semantic features are introduced, with the highest level being global semantic features. These features are then processed by CBR and weighted and modulated using sigmoid-generated weights, specifically as follows:

[0090] In the formula, This is the output of the decoding module; For the Sigmoid function; CBR This represents a combination of 3×3 convolution operation, batch normalization operation, and ReLU function.

[0091] Then, to Bilinear upsampling is performed, and the resulting concatenation with the cross-modal fusion features of the next level is then processed by CBR. This process is iterated layer by layer until the lowest level is reached. Specifically, this is represented as follows:

[0092] In the formula, These are the initial prediction features; For splicing operations, This is a bilinear upsampling operation.

[0093] Finally, after a 1×1 convolution and sigmoid activation, the output is a saliency prediction map of the same size as the input, specifically represented as:

[0094] In this step, A plot showing the significance of predictions.

[0095] Based on the different input images of the unified modality detector, when the original image in the multi-task collaborative training dataset is used as the input image, the unified modality detector generates the first saliency prediction map corresponding to the original image. When the enhanced image of the original image is used as the input image, a second saliency prediction map of the corresponding enhanced image is generated based on the unified modality detector. .

[0096] S35. Using high-quality pseudo-labels as supervision signals, a dual-enhancement consistency strategy is employed to train the unified modality detector.

[0097] In this step, a dual-enhancement consistency strategy is used to train the unified modality detector, and the prediction is based on the first saliency map. Second significance prediction plot Loss calculations were performed using high-quality pseudo-labels.

[0098] Specifically, based on the first significance prediction map Second significance prediction plot Calculate the cross-union loss (CUI) of the two with the high-quality pseudo-labels, using the following formula:

[0099] In the formula, To compare the losses between the two; H and W These are the height and width of the image, respectively; and These represent the predicted saliency map and pseudo-labels, respectively. First significance prediction plot hour, For high-quality pseudo-tags; when For the second significance prediction plot hour, The pseudo-labels are high-quality pseudo-labels that have undergone the same enhancement transformation operation as in step S1. To prevent numerically unstable minimal constants.

[0100] Simultaneously, calculate the first significance prediction map. With the second significance prediction plot The consistency loss between them is specifically expressed as:

[0101] In the formula, This results in a loss of consistency.

[0102] The intersection-union loss and consistency loss are weighted and summed to construct the total loss function, which is specifically expressed as:

[0103] In the formula, First significance prediction plot Cross-Union Loss and Second Significance Prediction Plot with High-Quality Pseudo-Labels The sum of the cross-union loss with high-quality pseudo-labels; The weight for consistency loss is typically set to 0.2.

[0104] Based on total losses Perform backpropagation to update the network parameters of the unified modality detector, and iterate the training until convergence, thus completing the training of the unified modality detector.

[0105] S4. Input the image to be detected into the trained unified modality detector, and output a high-precision salient target prediction map.

[0106] The image to be detected is input into the trained unified modality detector, and a high-precision salient target prediction map is output through forward inference to obtain the final salient target prediction result.

[0107] Example 2 In a typical embodiment of the present invention, this embodiment discloses an unsupervised salient target detection system based on multi-task collaborative training, comprising: The data acquisition and preprocessing module is configured to: acquire unannotated multimodal images, perform data augmentation on the multimodal images, and construct a multi-task collaborative training dataset based on the original images and their augmented images; The cross-modal pseudo-label generation module is configured to: extract salient information from multimodal images based on a cross-modal collaborative pseudo-label generation network to generate salient features, perform residual enhancement based on salient features to generate initial pseudo-labels, and perform pixel-level optimization on the initial pseudo-labels to generate high-quality pseudo-labels. The detector training module is configured to: take the original image and its enhanced image from the multi-task collaborative training dataset as input images, respectively, perform cross-modal feature complementation, global semantic perception and multi-level decoding on the input images based on the unified modality detector, and generate a first saliency prediction map corresponding to the original image and a second saliency prediction map corresponding to the enhanced image; construct a loss function based on the first saliency prediction map, the second saliency prediction map and high-quality pseudo-labels, and train the unified modality detector based on the loss function; The salient target prediction module is configured to input the image to be detected into a trained unified modality detector and output a high-precision salient target prediction map.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An unsupervised salient target detection method based on multi-task collaborative training, characterized in that, include: We acquire unannotated multimodal images and perform data augmentation on them. Based on the original images and their augmented images, we construct a multi-task collaborative training dataset. Based on a cross-modal collaborative pseudo-label generation network, salient information is extracted from multimodal images to generate salient features. Residual enhancement is performed based on the salient features to generate initial pseudo-labels. Pixel-level optimization is performed on the initial pseudo-labels to generate high-quality pseudo-labels. Using the original image and its enhanced image from the multi-task collaborative training dataset as input images, the input images are processed by a unified modality detector through cross-modal feature complementation, global semantic perception, and multi-level decoding to generate a first saliency prediction map for the original image and a second saliency prediction map for the enhanced image. A loss function is constructed based on the first saliency prediction map, the second saliency prediction map, and high-quality pseudo-labels, and the unified modality detector is trained based on the loss function. The image to be detected is input into the trained unified modality detector, which outputs a high-precision salient target prediction map. Based on a unified modality detector, the input image is subjected to cross-modal feature complementation, global semantic awareness, and multi-level decoding output, specifically including: Multi-scale features are extracted layer by layer from RGB images and their auxiliary modal images to obtain RGB features and auxiliary modal features at multiple levels; Based on the highest-level RGB modal features and auxiliary modal features, a fused high-level semantic feature is generated; after convolution and PReLU activation, a multi-branch grouped dilated convolution is performed on the fused high-level semantic feature, and after residual connection and CBR operation, the global semantic feature is output. At each level, multi-scale convolution is performed on RGB features and auxiliary modal features to extract features. The two modal features of each scale convolution branch are multiplied to generate cross-modal features. After being refined by depthwise separable convolution, they are concatenated and fused. At the same time, high-level semantic features are introduced and added to the residual of the original fused features. The cross-modal fused features are then output through concatenation and CBR operation. A top-down decoding structure is adopted, starting with the highest-level cross-modal fusion features as the initial input, introducing high-level semantic features for weighted modulation, upsampling, concatenating with the next-level cross-modal fusion features, and then processing with CBR. This process is iterated layer by layer to the lowest level, and finally processed by 1×1 convolution and Sigmoid activation to output a saliency prediction map of the same size as the input.

2. The unsupervised salient target detection method based on multi-task collaborative training as described in claim 1, characterized in that, The multimodal image includes an RGB image, a depth map, and a thermal infrared image; the acquired RGB image, depth map, and thermal infrared image are aligned to a preset size and mean-variance normalization is performed. Data augmentation of multimodal images specifically includes applying random flipping and random scaling data augmentation operations to the acquired multimodal images to obtain augmented multimodal images.

3. The unsupervised salient target detection method based on multi-task collaborative training as described in claim 1, characterized in that, The cross-modal collaborative pseudo-label generation network consists of a feature encoding unit, a salient information extraction unit, a pseudo-label generation unit, and a pseudo-label optimization unit connected in sequence.

4. The unsupervised salient target detection method based on multi-task collaborative training as described in claim 3, characterized in that, Generating high-quality pseudo-tags specifically includes: Multi-level feature extraction is performed on the input multimodal image to obtain multiple level feature maps; Extract salient information from mid-to-high-level feature maps and output enhanced salient features; The highest-level feature map is used to generate a global saliency prior. Based on the saliency features at each level and the global saliency prior, residual enhancement is performed and initial pseudo-labels are generated. Adaptive threshold binarization and normalization are performed sequentially on the initial pseudo-labels. The normalized initial pseudo-labels, the RGB image that restores the original representation, and other modal images are input into a conditional random field for pixel-level optimization. Noise is removed by bilateral filtering, and high-quality pseudo-labels are output.

5. The unsupervised salient target detection method based on multi-task collaborative training as described in claim 4, characterized in that, The process involves extracting salient information from mid-to-high-level feature maps and outputting enhanced salient features. Specifically, this includes: using mid-to-high-level features output by the feature encoding unit as input features; fusing global statistical features through max pooling and average pooling operations; generating attention weights through channel remapping and activation functions; multiplying the attention weights by the input features and then adding them together to supplement the original features; and outputting enhanced salient information.

6. The unsupervised salient target detection method based on multi-task collaborative training as described in claim 1, characterized in that, The unified modality detector includes a parallel encoder, a cross-modal complementarity module, a global semantic awareness module, and a decoding module. The global semantic awareness module captures multi-scale global semantic information through multi-branch grouped dilated convolutions, and combines cross-modal feature interaction and residual connections to output global semantic features, which are used to constrain the semantic consistency of the decoding process.

7. The unsupervised salient target detection method based on multi-task collaborative training as described in claim 1, characterized in that, The loss function is specifically a weighted sum of the intersection-union loss and the consistency loss.

8. The unsupervised salient target detection method based on multi-task collaborative training as described in claim 7, characterized in that, The crossover and union ratio (CUNR) loss includes the CUNR loss of the first significance prediction map with high-quality pseudo-labels and the CUNR loss of the second significance prediction map with high-quality pseudo-labels. The consistency loss is specifically the consistency loss between the first significance prediction map and the second significance prediction map.

9. An unsupervised salient target detection system based on multi-task collaborative training, characterized in that, include: The data acquisition and preprocessing module is configured to: acquire unannotated multimodal images, perform data augmentation on the multimodal images, and construct a multi-task collaborative training dataset based on the original images and their augmented images; The cross-modal pseudo-label generation module is configured to: extract salient information from multimodal images based on a cross-modal collaborative pseudo-label generation network to generate salient features, perform residual enhancement based on salient features to generate initial pseudo-labels, and perform pixel-level optimization on the initial pseudo-labels to generate high-quality pseudo-labels. The detector training module is configured to: take the original image and its enhanced image from the multi-task collaborative training dataset as input images, respectively, perform cross-modal feature complementation, global semantic perception and multi-level decoding on the input images based on the unified modality detector, and generate a first saliency prediction map corresponding to the original image and a second saliency prediction map corresponding to the enhanced image; construct a loss function based on the first saliency prediction map, the second saliency prediction map and high-quality pseudo-labels, and train the unified modality detector based on the loss function; The salient target prediction module is configured to input the image to be detected into a trained unified modality detector and output a high-precision salient target prediction map. Based on a unified modality detector, the input image is subjected to cross-modal feature complementation, global semantic awareness, and multi-level decoding output, specifically including: Multi-scale features are extracted layer by layer from RGB images and their auxiliary modal images to obtain RGB features and auxiliary modal features at multiple levels; Based on the highest-level RGB modal features and auxiliary modal features, a fused high-level semantic feature is generated; after convolution and PReLU activation, a multi-branch grouped dilated convolution is performed on the fused high-level semantic feature, and after residual connection and CBR operation, the global semantic feature is output. At each level, multi-scale convolution is performed on RGB features and auxiliary modal features to extract features. The two modal features of each scale convolution branch are multiplied to generate cross-modal features. After being refined by depthwise separable convolution, they are concatenated and fused. At the same time, high-level semantic features are introduced and added to the residual of the original fused features. The cross-modal fused features are then output through concatenation and CBR operation. A top-down decoding structure is adopted, starting with the highest-level cross-modal fusion features as the initial input, introducing high-level semantic features for weighted modulation, upsampling, concatenating with the next-level cross-modal fusion features, and then processing with CBR. This process is iterated layer by layer to the lowest level, and finally processed by 1×1 convolution and Sigmoid activation to output a saliency prediction map of the same size as the input.

Images