A method, device and electronic equipment for detecting salient objects based on multi-modal prediction reconstruction error

Abstract

The application provides a salient object detection method and device based on multi-modal prediction reconstruction error and an electronic device. The method comprises: acquiring images of at least one mode and establishing a mode existence mask vector; performing feature coding on each existing mode image to obtain multi-scale features, and performing feature fusion to obtain a multi-scale global scene representation; performing self-prediction reconstruction and cross-modal prediction reconstruction based on the multi-scale global scene representation to obtain self-reconstruction results and cross-modal reconstruction results; calculating multi-scale self-reconstruction errors and multi-scale cross-modal reconstruction errors corresponding to each mode to obtain multi-scale error maps and initial saliency maps; constructing a gaze area and cropping image blocks from each mode image to calculate initial local saliency maps, and obtaining a refined saliency map through iterative refinement; calculating a gating saliency map corresponding to a global error feature vector, and fusing the gating saliency map and the refined saliency map to obtain a final salient object detection result.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and specifically to a salient target detection method, apparatus, and electronic device based on multimodal prediction and reconstruction error. Background Technology

[0002] Saliency detection aims to automatically identify the regions in an image or video that are most visually appealing to humans, and is widely used in object tracking, image segmentation, scene understanding, and other fields. With the development of multimodal sensing technology, saliency detection integrating multimodal data sources such as RGB-Depth and RGB-Thermal has become a research hotspot. However, related multimodal saliency detection methods mainly suffer from the following problems:

[0003] First, the modal combinations are not fixed, and the relevant detection networks lack flexibility.

[0004] In real-world applications, available modalities may be single RGB images, or combinations of RGB-D, RGB-T, or even RGB-DT. Traditional methods design dedicated networks for a fixed number of modalities, requiring model retraining whenever the modal combination changes; or they introduce masking mechanisms to handle missing modalities, but these have limited feature compensation capabilities for missing information, resulting in significant performance degradation as the number of modalities decreases. This strong modality dependency limits the model's deployment flexibility in real-world environments.

[0005] Second, it relies on the strength of single-mode features and is susceptible to noise interference.

[0006] Most related multimodal saliency detection methods perform modal stitching or attention weighting in the feature space, essentially still relying on the intensity response of single-modal features. This makes it difficult to distinguish between truly salient targets and pseudo-salient regions in single-modal images, such as noise points in depth maps or overexposed areas in infrared images. These regions respond strongly in single-modal images, but are inconsistent with other modal information, and traditional methods easily misclassify them as salient targets.

[0007] Third, the ability to characterize target boundaries is limited, and details are easily lost.

[0008] Although related methods generally employ multi-scale feature fusion, they typically involve only a single forward propagation process, and the perception of target boundaries is limited by the size of the network's receptive field. In scenarios where the target scale varies greatly and the boundaries are blurred, the generated saliency map has a coarse outline and insufficient detail retention.

[0009] Fourth, fusion measurement lacks scene adaptation capabilities.

[0010] Correlation fusion strategies often employ fixed weights or simple attention mechanisms, failing to dynamically adjust the contribution weights of each modality based on the scene. For example, RGB information degrades significantly under low-light conditions, so its weight should be reduced; while in well-lit scenes with high depth noise, more RGB information should be relied upon. Correlation fusion measurements are ill-suited to adapting to these differences in information quality across varying scenes. Summary of the Invention

[0011] In view of this, this application provides a salient target detection method, apparatus and electronic device based on multimodal prediction and reconstruction error, so as to achieve robust salient target detection under arbitrary modal combinations.

[0012] Specifically, this application is implemented through the following technical solution:

[0013] According to a first aspect of the embodiments of this specification, a salient target detection method based on multimodal prediction and reconstruction error is provided, comprising:

[0014] Step S1: Acquire images of at least one modality and establish a modality presence mask vector, wherein the modality presence mask vector is used to indicate whether an image of each set modality exists;

[0015] Step S2: Encode the features of each existing modality image to obtain multi-scale features, and fuse the features of all existing modality images at each scale to obtain a multi-scale global scene representation.

[0016] Step S3: Based on the multi-scale global scene representation, perform self-predictive reconstruction on the multi-scale features of each existing modality, and when there are two or more modal images, perform multi-scale feature cross-modal prediction reconstruction from the source modality to the target modality on each pair of existing modalities, and obtain self-reconstruction results and cross-modal reconstruction results respectively.

[0017] Step S4: Based on the multi-scale features, self-reconstruction results and cross-modal reconstruction results, calculate the multi-scale self-reconstruction error and multi-scale cross-modal reconstruction error corresponding to each modality, and fuse them to construct a multi-scale error map, so as to obtain an initial saliency map by fusing the error maps of all scales.

[0018] Step S5: Based on the multi-scale error map and the initial saliency map, the gaze region is obtained. Within each candidate gaze region, the corresponding image patch is cropped from each modal image and enlarged to a fixed high resolution to perform steps S2 to S4 to obtain an initial local saliency map. Based on the pixel difference between the initial local saliency map and the corresponding initial saliency map, an updated saliency map is obtained. Step S5 is then re-executed on the updated saliency map until the preset iteration exit condition is met to obtain a refined saliency map.

[0019] Step S6: Statistically analyze the self-reconstruction errors and cross-modal reconstruction errors for all modalities and scales to obtain a global error feature vector, which is then input into a gating network to generate adaptive weights. Based on the adaptive weights, recalculate the multi-scale error map for the self-reconstruction errors and cross-modal reconstruction errors. Fuse all the recalculated multi-scale error maps to obtain a gating saliency map. Fuse the gating saliency map with the refined saliency map to obtain the final salient target detection result.

[0020] According to a second aspect of the embodiments of this specification, a salient target detection device based on multimodal prediction and reconstruction error is provided, comprising:

[0021] An image acquisition unit is configured to acquire an image of at least one modality and establish a modality presence mask vector, wherein the modality presence mask vector is used to indicate whether an image of each set modality exists;

[0022] The feature extraction unit is used to encode the features of each existing modality image to obtain multi-scale features, and at each scale, the features of all existing modality images are fused to obtain a multi-scale global scene representation.

[0023] The feature reconstruction unit is used to perform self-predictive reconstruction on the multi-scale features of each existing modality based on the multi-scale global scene representation, and when there are two or more modal images, to perform multi-scale feature cross-modal prediction reconstruction from the source modality to the target modality on each pair of existing modalities, and obtain self-reconstruction results and cross-modal reconstruction results respectively.

[0024] The error calculation unit is used to calculate the multi-scale self-reconstruction error and multi-scale cross-modal reconstruction error corresponding to each modality based on the multi-scale features, self-reconstruction results and cross-modal reconstruction results, and to fuse and construct a multi-scale error map so as to obtain an initial saliency map by fusing the error maps of all scales.

[0025] The iterative calculation unit is used to obtain the gaze region based on the multi-scale error map and the initial saliency map, and within each candidate gaze region, to crop the corresponding image patch from each modal image and enlarge it to a fixed high resolution to calculate the initial local saliency map. Based on the pixel difference between the initial local saliency map and the corresponding initial saliency map, an updated saliency map is obtained. The iterative calculation is then re-executed on the updated saliency map until a preset iteration exit condition is met to obtain a refined saliency map.

[0026] The saliency calculation unit is used to statistically analyze the self-reconstruction error and cross-modal reconstruction error for all modalities and scales, obtain a global error feature vector, and input it into the gating network to generate adaptive weights. Based on the adaptive weights, the multi-scale error map is recalculated for the self-reconstruction error and cross-modal reconstruction error. All recalculated multi-scale error maps are fused to obtain a gated saliency map. The gated saliency map is then fused with the refined saliency map to obtain the final salient target detection result.

[0027] According to a third aspect of the embodiments of this specification, an electronic device is provided, including a processor; and a computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method described in the first aspect.

[0028] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor of the method described in the first aspect.

[0029] The embodiments of this application have at least the following beneficial effects:

[0030] (1) The embodiments of this application improve the anti-interference capability based on the predictability saliency measurement mechanism: through the dual mechanism of self-predictive reconstruction and cross-modal prediction reconstruction, the global scene representation is used to reconstruct images of each modality, and the source modality is used to predict the target modality. When the self-reconstruction error of a local region is small while the cross-modal reconstruction error is large, it indicates that the local region contains unique information that is difficult to predict by other modalities, that is, there is cross-modal inconsistency, suggesting that the local region may contain a significant target. Thus, using whether it is easy to be predicted by other modalities as the judgment criterion, it can better distinguish between real targets and single-modal noise, and the detection is more accurate in complex scenes such as weak contrast and modal information conflict.

[0031] (2) The embodiments of this application are based on the human-like visual gaze mechanism to achieve refined iterative optimization of the boundary: based on the initial saliency map and error distribution, several high error areas are adaptively selected as gaze areas. Encoding, reconstruction and error calculation are re-executed in the gaze areas at a higher resolution to obtain a more detailed local saliency map. Then, the global result is updated by weighted fusion. In this way, the target boundary can be gradually focused and refined through multiple iterations, effectively improving the problem of rough target outline and loss of details.

[0032] (3) The embodiments of this application utilize error statistics to drive adaptive fusion and enhance scene adaptability: the global error statistics of multimodal and multi-scale are used as implicit representations reflecting the quality of scene information and input into the gating network to generate parameters such as the weight of each modality and the self / cross reconstruction error weight. Thus, the degree of trust in each modal information can be automatically adjusted according to the actual scene (e.g., low illumination, large depth noise, etc.) to achieve scene adaptation of the fusion strategy and improve the robustness of the system.

[0033] (4) The embodiments of this application provide interpretable error visualization output to support high-reliability applications: the measurement mechanism based on reconstruction error is interpretable and can output self-reconstruction and cross-modal reconstruction error maps of each modality, intuitively showing which regions in the image have modal uniqueness or cross-modal inconsistency. These intermediate results not only help developers understand the model decision-making process and analyze failure cases, but also provide additional semantic clues and auxiliary judgment basis for downstream tasks with high reliability requirements such as medical image analysis and autonomous driving. Attached Figure Description

[0034] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0035] Figure 1 This is a schematic flowchart illustrating an exemplary embodiment of the present application of a salient target detection method based on multimodal prediction and reconstruction error;

[0036] Figure 2 This is a schematic diagram illustrating the salient target detection result in an exemplary embodiment of this application;

[0037] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment of this application;

[0038] Figure 4 This is a block diagram of a salient target detection method apparatus based on multimodal prediction and reconstruction error, as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0040] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0041] The embodiments described in this specification will now be described in detail.

[0042] This application provides a salient target detection method based on multimodal prediction and reconstruction error. Figure 1 This is a schematic flowchart illustrating an exemplary embodiment of the present application of a salient target detection method based on multimodal prediction and reconstruction error, as shown below. Figure 1 The salient target detection method shown includes the following steps:

[0043] Step S1: Acquire images of at least one modality and establish a modality presence mask vector, wherein the modality presence mask vector is used to indicate whether an image of each set modality exists.

[0044] The salient target detection method provided in this application can be executed by hardware or software entities with data processing capabilities, including but not limited to servers, processors, computer equipment, chips, system-on-a-chip (SoC), application-specific integrated circuits (ASIC), programmable logic devices (FPGA), distributed computing nodes, cloud computing instances, edge computing devices, or functional modules / systems integrating corresponding processing logic. For clarity, the following embodiments will use a server as the execution entity for illustrative purposes.

[0045] The modal types in this embodiment include, but are not limited to, RGB images, depth images, thermal infrared images, and polarization images; the modality has a mask vector of... Where M represents the total number of modal types, Indicates the first A modal image exists. Indicates the first One modality image is missing.

[0046] Step S2: Encode the features of each existing modal image to obtain multi-scale features, and fuse the features of all existing modal images at each scale to obtain a multi-scale global scene representation.

[0047] This embodiment applies to each image containing a modality. via modality-specific adapter Preliminary encoding is performed to obtain adaptation features with uniform dimensions. , The adaptation features of all modalities are fed into a parameter-shared backbone coding network to extract multi-scale features for each modality. , , For the overall scale. In At the scale, all features of existing modalities are embedded and concatenated with their corresponding modality IDs, and a global scene representation is obtained through a cross-modal aggregator. , ,in This is a cross-modal aggregation operator used to fuse multimodal features with modality presence information.

[0048] For example, a modality-specific adapter Use 1-2 layers Convolutional structures map the original number of channels from different modalities to a unified dimension. The parameters share the backbone encoding network, which adopts the ResNet architecture; the cross-modal aggregator adopts a 1-2 layer Transformer Encoder structure and fuses different modal features through a self-attention mechanism.

[0049] Step S3: Based on the multi-scale global scene representation, perform self-predictive reconstruction on the multi-scale features of each existing modality, and when there are two or more modal images, perform multi-scale feature cross-modal prediction reconstruction from the source modality to the target modality on each pair of existing modalities, and obtain self-reconstruction results and cross-modal reconstruction results respectively.

[0050] Step S4: Based on the multi-scale features, self-reconstruction results and cross-modal reconstruction results, calculate the multi-scale self-reconstruction error and multi-scale cross-modal reconstruction error corresponding to each modality, and fuse them to construct a multi-scale error map, so as to obtain an initial saliency map by fusing the error maps of all scales.

[0051] Step S5: Based on the multi-scale error map and the initial saliency map, the gaze region is obtained. Within each candidate gaze region, the corresponding image block is cropped from each modal image and enlarged to a fixed high resolution to perform steps S2 to S4 to obtain an initial local saliency map. Based on the pixel difference between the initial local saliency map and the corresponding initial saliency map, an updated saliency map is obtained. Step S5 is then re-executed on the updated saliency map until the preset iteration exit condition is met to obtain a refined saliency map.

[0052] Step S6: Statistically analyze the self-reconstruction errors and cross-modal reconstruction errors for all modalities and scales to obtain a global error feature vector, which is then input into a gating network to generate adaptive weights. Based on the adaptive weights, recalculate the multi-scale error map for the self-reconstruction errors and cross-modal reconstruction errors. Fuse all the recalculated multi-scale error maps to obtain a gating saliency map. Fuse the gating saliency map with the refined saliency map to obtain the final salient target detection result.

[0053] This step involves generating error maps for all modes and all scales. Perform global statistics, including but not limited to mean, variance, and maximum value, and concatenate the statistical results into a global error feature vector. ;Will Input to a gated network MLP, output adaptive weights The segmented mapping will then be used to recalculate the multi-scale error map.

[0054] For example, the gated network MLP adopts a 2-3 layer fully connected structure, with intermediate layer dimensions of 64-128, and the output layer uses a Sigmoid activation function to restrict the weights to a certain value. Interval.

[0055] like Figure 1 As shown in the salient target detection method, this embodiment employs a dual mechanism of self-predictive reconstruction and cross-modal predictive reconstruction. It utilizes global scene representation to reconstruct images of each modality and predicts the target modality using the source modality. When the self-reconstruction error of a local region is small while the cross-modal reconstruction error is large, it indicates that the local region contains unique information that is difficult for other modalities to predict, i.e., there is cross-modal inconsistency. This suggests that the local region may contain a salient target. Thus, using the ease with which it can be predicted by other modalities as the criterion can better distinguish between real targets and single-modal noise, resulting in more accurate detection in complex scenes such as weak contrast and modal information conflict.

[0056] Based on the initial saliency map and error distribution, this embodiment adaptively selects several high-error regions as staring regions. Encoding, reconstruction, and error calculation are re-executed at a higher resolution within the staring regions to obtain a more detailed local saliency map. Then, the global result is updated through weighted fusion. This process can be iterated multiple times to gradually focus on and refine the target boundary, effectively improving the problems of rough target contours and loss of details.

[0057] Furthermore, this embodiment uses multimodal and multi-scale global error statistical features as an implicit representation of scene information quality, inputting them into a gating network to generate parameters such as modal weights and self / cross reconstruction error weights. This allows the system to automatically adjust the level of trust in each modal information according to the actual scene (e.g., low illumination, high depth noise), thereby achieving scene adaptation of the fusion strategy and improving system robustness.

[0058] Furthermore, the reconstruction error-based measurement mechanism is interpretable and can output self-reconstruction and cross-modal reconstruction error maps for each modality, intuitively showing which regions in the image have modal uniqueness or cross-modal inconsistency. These intermediate results not only help developers understand the model decision-making process and analyze failure cases, but also provide additional semantic clues and auxiliary judgment basis for downstream tasks with high reliability requirements, such as medical image analysis and autonomous driving.

[0059] In some embodiments, step S3 includes:

[0060] For each existing mode, at each scale, the existing mode image is downsampled to the corresponding scale resolution and then processed by a self-reconstruction decoder. Decoding the global scene representation at the same scale yields self-reconstruction results. ;

[0061] For each pair of existing modes At each scale, the decoder is reconstructed via cross-modal reconstruction. Global scene representation and source modal features at the same scale Decoding is performed to obtain the cross-modal reconstruction result of the target mode. Here Represents the source mode exist Features at scale.

[0062] The self-reconstruction decoder shares the main decoding structure with the cross-modal reconstruction decoder, differing only in the structure of the last convolutional layer. For example, the self-reconstruction decoder... Employing a symmetric decoding structure combining upsampling and convolution; a cross-modal reconstruction decoder. Introducing source modal features during decoding Conditional generation is achieved through feature splicing or FiLM modulation; for missing modes, the corresponding reconstruction branch is skipped.

[0063] In some embodiments, step S4 includes:

[0064] The multi-scale self-reconstruction error and the multi-scale cross-modal reconstruction error are calculated based on the error metric function, wherein the error metric function is used to measure the pixel differences, local structural consistency differences, and high-frequency differences between the multi-scale features and the self-reconstruction results, as well as between the multi-scale features and the cross-modal reconstruction results.

[0065] For example, the error measurement function The form is:

[0066] (1)

[0067] in It is a high-pass filter operator; Preset weights; It serves as a local structural consistency index.

[0068] In some embodiments, step S5 obtains the gaze area through the following steps:

[0069] Based on the multi-scale error map and the initial saliency map, a heat map of the staring region is generated;

[0070] A penalty term is applied to the gaze region heatmap determined during the iteration process;

[0071] The heatmap of the gaze region after penalty processing is divided into multiple grids, the average heat value of each grid is calculated, and a preset number of gaze regions are selected based on a preset sorting algorithm.

[0072] In one example, generating a gaze region heatmap based on the multi-scale error map and the initial saliency map includes:

[0073] The error maps at each scale are upsampled to the same resolution and then weighted and summed. The weighted and summed error maps are then normalized to obtain a comprehensive significance map.

[0074] The heatmap of the gaze region is obtained by weighted summation of the comprehensive saliency map and the initial saliency map.

[0075] In another example, the application of a penalty term to the gaze region heatmap determined during the iteration process includes:

[0076] Set access mask image The access mask map Size and heat map of the gaze area Similarly, the initial value of the access mask map is zero;

[0077] The values ​​corresponding to the gaze regions identified during the iteration process will be marked as 1. The heatmap of the gaze regions after penalty processing is as follows: .

[0078] In some embodiments, step S5 obtains the updated saliency map through the following steps:

[0079] Obtain the soft mask and confidence weight of the gaze region corresponding to the initial local saliency map. The soft mask has a pixel value of 1 at the center position and a Gaussian decay of pixel values ​​from the center position to the edge.

[0080] The pixel differences between the mapped local saliency map and the initial local saliency map are calculated, and the pixel differences are weighted and summed based on the soft mask and the confidence weight. The weighted summation result is then weighted and fused with the initial saliency map to obtain the updated saliency map.

[0081] To illustrate the salient target detection process in this embodiment in detail, the following embodiments will be used as a reference.

[0082] Step 1: Construct a multimodal saliency detection dataset.

[0083] We have collected a saliency detection dataset containing multiple modal combinations such as RGB-D, RGB-T, and RGB-DT. Each sample contains images of at least two modalities, along with manually annotated pixel-level ground truth masks. The dataset can cover a variety of complex scenes, including indoor and outdoor environments, varying lighting conditions, occlusion, and low contrast.

[0084] The training, validation, and test sets are divided in a ratio of 7:1.5:1.5, ensuring a balanced distribution of different modality combinations across the subsets. In the training set, one or more modalities are randomly discarded from each sample group to construct modality-missing samples, allowing the model to learn to adapt to any modality subset. For example, the modality-missing probability is set as follows: 30% for single-modality samples, 40% for bimodal samples, and 30% for trimodal samples.

[0085] Step 2: Perform image preprocessing and modality configuration.

[0086] Adjust all input images uniformly to Resolution, for example, is maintained using bilinear interpolation to preserve smoothness. The following processing is exemplarily applied to various modal images:

[0087] The RGB image is normalized by mapping pixel values ​​from the range of 0–255 to the range of 0–1, and then normalized using ImageNet statistics.

[0088] For depth images, nearest neighbor interpolation is used to fill in invalid depth values, and then the images are normalized to the 0–1 range.

[0089] For thermal infrared images, normalize directly to the 0–1 range.

[0090] After preprocessing each mode, a mode existence mask vector is established. If the input image contains three modalities: RGB image, depth image, and thermal infrared image, then If only RGB and depth images are included, then Assign a unique ID to each modality type; for example, ID 1 for RGB images, ID 2 for depth images, ID 3 for thermal infrared images, and ID 4 for polarized images, to distinguish modality types through ID embedding.

[0091] Data augmentation is employed during training, including random horizontal flipping, random rotation, and random brightness and contrast adjustments, while simultaneously transforming the ground truth mask for saliency.

[0092] Step 3: Perform multimodal adaptive coding.

[0093] For each existing mode Through the corresponding modality-specific adapter Perform preliminary coding; for example:

[0094] RGB Modal Specific Adapter Two layers Convolutional layer: first layer has 3 input channels and 32 output channels; second layer has 64 output channels. ;

[0095] Deep Modality Specific Adapter Two layers Convolution: the first layer has 1 input channel and 32 output channels, and the second layer has 64 output channels.

[0096] Thermal infrared mode-specific adapter The structure is the same as that of the depth modality-specific adapter;

[0097] Each modality-specific adapter is followed by a BatchNorm and ReLU activation function to apply the adaptation features of each modality. The parameters are shared with the ResNet-50 backbone network, which is a network pre-trained on ImageNet.

[0098] For example, ResNet-50 outputs feature maps at four scales, corresponding to the outputs of the conv2_x, conv3_x, conv4_x, and conv5_x layers, respectively;

[0099] First scale ( ): ;

[0100] Second scale ( ): ;

[0101] Third scale ( ): ;

[0102] Fourth Scale ( ): ;

[0103] exist At scale, for all features that exist in a modality Perform cross-modal aggregation. For example, first add a modal ID embedding to each modal feature. The ID embedding is a learnable vector with dimensions and number of channels. Similarly, the features are broadcast and added to each spatial location of the feature map; then all modal features are concatenated along the channel dimension. The number of channels is ; through a Convolution reduces the dimensionality of spliced ​​features to Then, the input is fed into a cross-modal aggregator; for example, the cross-modal aggregator adopts a single-layer Transformer Encoder structure, where the input feature map is first reshaped into a sequence. By fusing information from different locations and modalities through multi-head self-attention, and then reshaping back... The global scene representation output in this way It contains complementary information from all modalities.

[0104] Step 4: Perform modal prediction and reconstruction.

[0105] Based on global scene representation We construct two types of decoders: self-reconstruction decoders and cross-modal reconstruction decoders.

[0106] For each existing mode Design a self-reconstruction decoder separately The self-reconstruction decoder employs a symmetrical upsampling structure, from a scale perspective... Gradually restore to the original resolution; at the fourth scale ( For example, the decoder contains four upsampling operations:

[0107] First step: Upsampling of transposed convolution (stride=2, kernel=4) to The number of channels decreased to 1024;

[0108] Second time: transposed convolution upsampling to The number of channels decreased to 512;

[0109] Third time: transposed convolution upsampling to The number of channels decreased to 256;

[0110] Fourth time: transposed convolution upsampling to The number of channels has been reduced to 128;

[0111] Finally passed The convolution is mapped to the number of channels corresponding to the modality (3 for RGB images, 1 for depth images, and 1 for thermal infrared images).

[0112] Each transposed convolution is followed by BatchNorm and ReLU activation, and the last layer uses the Tanh activation function, with an output range of (-1, 1). Matching this output range with the normalized input range, the self-reconstruction result can be obtained. The self-reconstructed decoders for all modalities share the transposed convolutional backbone parameters, with only the final 1×1 convolutional layer being independent based on the target modal channel number.

[0113] For each pair of existing modes Design a cross-modal reconstruction decoder The cross-modal reconstruction decoders for all target modalities share a backbone, while the last layer is independent based on the target modality, incorporating source modality features during the decoding process. The fusion method employs feature concatenation, where the source modality features are combined after each upsampling. The data is upsampled to the corresponding size using bilinear interpolation, and then concatenated to the decoded features along the channel dimension. For example, the features after the first upsampling are... ,Will from Upsampling After splicing, it becomes Then through Convolution dimensionality reduction to This fusion approach enables the cross-modal reconstruction decoder to utilize the structural information of the source modality to guide the reconstruction of the target modality. Finally, it also... Convolution is mapped to the target modal channel number, and Tanh activation is used to obtain cross-modal reconstruction results. .

[0114] for There are 1 modality, and each modality has 1 self-reconstruction result and 1... A cross-modal reconstruction result, for the missing mode ( It automatically skips all its reconstruction branches and does not perform forward computation, thus saving computing resources.

[0115] Step 5: Perform multi-level error calculation and significance response construction.

[0116] Refer to the error measurement function in formula (1) above. Exemplary Preset weights; measure pixel-wise differences using the L1 norm and local structural consistency differences using SSIM; window size set to... For high-pass filter operators In this embodiment, the Sobel gradient operator is used to calculate the L2 norm of the horizontal and vertical gradients and extract high-frequency edge components.

[0117] For each existing mode , in scale Above, the original modal image Downsampling to this scale resolution The multi-scale self-reconstruction error is calculated using the following formula (2). :

[0118] (2)

[0119] In the self-reconstruction error map In this context, a larger value indicates that the location is more difficult to reconstruct using the global scene representation.

[0120] For each pair of existing modes, for each target mode Collect all source modes Cross-modal reconstruction results The multi-scale cross-modal reconstruction error is calculated using the following formula (3). :

[0121] (3)

[0122] Optionally, for multi-scale cross-modal reconstruction errors Using a minimum pooling strategy, we obtain In this embodiment, minimum pooling is used to represent the error of the most predictable source mode. If the minimum error is still large, it indicates that all modes in this region are difficult to predict, and the cross-modal inconsistency is high.

[0123] Regarding scale The multi-scale error map is calculated using the following formula (4). :

[0124] (4)

[0125] in For modal scale weights, For error type weights, It acts as a mask to prevent missing modes from participating in the calculation. For example, it is initially set to... ; This is to emphasize cross-modal inconsistency.

[0126] In this way, error maps at four scales can be obtained. The error maps at these four scales are fused through an error inference network, so that the error maps at the four scales are uniformly upsampled to the first scale resolution through bilinear interpolation. ), and concatenate them along the channel dimension to obtain a 4-channel feature map, through three layers Convolutional layers (32, 16, and 8 channels respectively) are progressively fused, with each layer followed by BatchNorm and ReLU; finally, the convolutions are passed through... The convolution output is a single channel, which is then passed through the Sigmoid activation function to obtain the initial saliency map. ,Will Upsampled to original resolution .

[0127] Step 6: Perform error-driven coarse-to-fine gaze refinement.

[0128] Based on the initial significance map and multi-scale error plot Constructing a heat map of the gaze region Based on the error map, coarse-to-fine gaze refinement is performed, and the specific process is as follows:

[0129] First, the error maps at each scale are upsampled to the original resolution, and then weighted and summed using the following formula (5):

[0130] (5)

[0131] in For scale weights, deeper layers have larger weights, for example... , This is the upsampling operator.

[0132] Then, a heat map of the staring region is generated based on formula (6). :

[0133] (6)

[0134] In formula (6) It is used to normalize the error map to .

[0135] To avoid repeated staring, maintain the access mask. The initial matrix is ​​all zeros, and visited regions are marked as 1; then the heatmap of the gaze region is... Imposing punishment:

[0136] (7)

[0137] Next, the heat map Divided into The grid, each grid For each pixel, calculate the average thermal value, and sort by Top- A grid, as an example .

[0138] To make the selection process differentiable, Gumbel-Softmax soft selection is employed. This involves adding Gumbel noise to the grid thermal values, normalizing them using softmax, and setting the temperature parameter to 0.5. Specifically, during the training phase, Gumbel-Softmax is used to perform soft selection on the grid thermal values ​​with temperature=0.5, and the output soft mask makes subsequent operations differentiable. During the inference phase, the top-6 grids are directly selected, and hard thresholding and non-maximum suppression are applied to merge adjacent grids.

[0139] Next, nonmaximum suppression is applied to the selected grid, and adjacent grids are merged to ultimately obtain 3–6 gaze regions. This embodiment expands each region to Pixels, ensuring sufficient context is included.

[0140] Next, for each gaze area Cropping corresponding modal original images The image patch is enlarged using bicubic interpolation. To increase the resolution, repeat steps 3 through 5 on the high-resolution patch to generate a local saliency map. .

[0141] Then, for the local saliency map , take it from Zoom to To match the original gaze area size; in On the full map, the location area central position Construct a soft mask with a radius of 32 pixels. Soft mask The center pixel value is 1, and the pixel values ​​from the center to the edge decrease Gaussian, with a standard deviation of, for example, 16 pixels. The confidence weight is calculated based on the following formula (8). :

[0142] (8)

[0143] The updated saliency plot is then calculated using the following formula (9):

[0144] (9)

[0145] According to formula (9), for the region Outside pixels, If the value is close to 0, keep the original value; for the region Within a given pixel, local and global information are fused according to confidence level; access mask. The area that was gazed at is marked as 1.

[0146] Finally, perform iterative refinement.

[0147] Repeat the above process Next, this embodiment sets .

[0148] Each iteration is based on the updated Regenerate the heatmap of the gazed area, selecting new high-error regions. Due to the penalty of the access mask, subsequent iterations will focus on different regions, achieving progressive refinement of the entire map.

[0149] Optionally, an early stopping condition is set, such as: if the change in the saliency plot between two consecutive iterations is less than a threshold ( The iteration update was terminated early, resulting in a refined saliency map. .

[0150] Step 7: Perform adaptive gating fusion.

[0151] For all modes All scales Error plot Calculate global statistics. For example, calculate the mean for each error plot. Standard deviation maximum value 75th percentile Concatenate them into a global error feature vector :

[0152] (10)

[0153] Based on four-scale features in three modalities The dimension is dimension.

[0154] Optionally, by analyzing the global error eigenvector of L... LayerNorm normalization is performed to stabilize the numerical range.

[0155] The gated network in this embodiment is used to output adaptive weights. For example, a three-layer fully connected network is designed as a gated MLP, wherein:

[0156] First floor, ReLU activation;

[0157] The second floor, ReLU activation, add Dropout ( Prevent overfitting;

[0158] The third layer, Output dimension The total number of required weight parameters, including... One modal scale weight, two error type weights, The weights are updated locally. This embodiment... The output vector is... The segmented mapping is as follows:

[0159] The first 12 elements correspond ;

[0160] The 13th–14th elements correspond to ;

[0161] Elements 15–20 correspond to the confidence weights of the six gaze regions. .

[0162] Alternatively, the output of the gated network can be restricted to a certain value using a sigmoid activation function. Interval.

[0163] After obtaining the above adaptive weights, the multi-scale error map is recalculated:

[0164] (11)

[0165] Fusion of all scales through output head network Generate gated saliency maps .

[0166] Finally, the gated saliency map and the refined saliency map are weighted and fused to obtain the final salient object detection result. :

[0167] (12)

[0168] The weight ratio in formula (12) can be trained or fixed. In this embodiment, it is fixed at 0.4:0.6 to emphasize the refinement of the results and the final salient target detection results. .

[0169] Optionally, the self-reconstruction error map for each mode can also be output. Cross-modal reconstruction error diagram Or output the gating weight vector. Visualize the importance of each modality in the current scene; or output the location of the gaze area. and its confidence level This is to show the key areas that the model focuses on.

[0170] Step 8: Construct a multi-task loss function to achieve end-to-end joint optimization.

[0171] The multi-task loss function in this embodiment includes a saliency graph construction supervision loss. Reconstruction quality loss and gaze sparsity loss .

[0172] The saliency graph supervised loss for:

[0173] (13)

[0174] in,

[0175] , ,

[0176] As a truth significance mask, This is a significance plot for prediction.

[0177] For the initial saliency map Saliency plot after refinement and final significance plot Calculate separately Weighted summation:

[0178] (14)

[0179] The reconstruction quality loss for:

[0180] (15)

[0181] in,

[0182] , .

[0183] The gaze sparsity loss The gaze sparsity loss is used to encourage sparsity in gaze region selection and avoid computational redundancy caused by excessive gaze. for:

[0184] (16)

[0185] In formula (16), the first term penalizes excessively high confidence, and the second term penalizes excessively large mask areas.

[0186] The first penalty is for excessively high confidence levels, and the second penalty is for excessively large mask areas.

[0187] Therefore, the multi-task loss function in this embodiment is:

[0188] (17)

[0189] For example, .

[0190] Minimize the total loss using the Adam optimizer, with an initial learning rate. , The learning rate is scheduled using cosine annealing, with a period of 50 epochs, and a minimum learning rate of 1% of the initial value; the weight decay coefficient is set to... The batch size is set to 8, and the training lasts for 200 epochs.

[0191] Step 9, Model Training Strategy.

[0192] During model training, a portion of modalities are randomly discarded for each batch to simulate modality loss in real-world applications.

[0193] Single-mode retention probability: ;

[0194] Bimodal retention probability: ;

[0195] Three-mode retention probability: ;

[0196] For the retained modal combinations, set the corresponding... =1, missing mode It is 0.

[0197] During forward propagation, the encoding and reconstruction branches of missing modalities are automatically skipped.

[0198] Parameter initialization: Convolutional layer weights are initialized using Kaiming and adapted to the ReLU activation function; fully connected layer weights are initialized using Xavier; shared backbone ResNet-50 uses ImageNet pre-trained weights; modal adapters, decoders, and gating networks are randomly initialized.

[0199] During training, random horizontal flipping, random scaling, random cropping, random brightness adjustment of RGB images, random addition of Gaussian noise to depth images, and all enhancements are applied synchronously to the ground truth mask.

[0200] Performance was evaluated on the validation set after each epoch, with metrics including: maximum F-measure ( , Mean absolute error S-measure. Select the best model based on the validation set S-measure.

[0201] An early stopping strategy is adopted: if no performance improvement is verified for 20 consecutive epochs, training is stopped.

[0202] Step 10, Model Inference and Performance Evaluation.

[0203] Input any combination of modal images, and set the modal mask vector according to the actual input. Perform steps 3 through 7 to generate the final salient target detection results. Optionally, it can output the error map of each mode and the location of the staring region.

[0204] Inference time: Approximately 50ms for a single forward propagation (excluding gaze refinement), and approximately 180ms for 3 gaze iterations.

[0205] Evaluation was performed on multiple public datasets: RGBD significance: NJU2K, NLPR, STERE, SIP, DUT-RGBD; RGBT significance: VT821, VT1000, VT5000; evaluation metrics included the most... MAE, S-measure, and E-measure are used. Table 1 shows the performance comparison of different modal combinations, Table 2 shows the effect of the number of gaze refinement iterations, and Table 3 shows the ablation experiment of the core module.

[0206]

[0207] Table 1

[0208]

[0209] Table 2

[0210]

[0211] Table 3

[0212] Combining Tables 1 to 3 and Figure 2As shown, cross-modal reconstruction is the most critical module, and its performance decreases by 4.2% after removal; the modal adapter is crucial for handling heterogeneous modalities, and its performance decreases by 5.5% after removal; gaze refinement and adaptive gating bring improvements of 2.9% and 2.1% respectively; the four modules work together, and the overall performance is better than simple stacking.

[0213] Figure 3 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 3 At the hardware level, the device includes a processor 302, an internal bus 304, a network interface 306, memory 308, a hardware acceleration device 310, and non-volatile memory 312, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 302 reads the corresponding computer program from the non-volatile memory 312 into memory 308 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0214] Figure 4 This is a block diagram illustrating an exemplary embodiment of a salient target detection device based on multimodal prediction and reconstruction error, which can be applied to, for example... Figure 3 The electronic device shown implements the technical solution of this application. The salient target detection device includes: an image acquisition unit 410, a feature extraction unit 420, a feature reconstruction unit 430, an error calculation unit 440, an iterative calculation unit 450, and a salientity calculation unit 460, wherein:

[0215] Image acquisition unit 410 is used to acquire images of at least one modality and establish a modality presence mask vector, wherein the modality presence mask vector is used to indicate whether an image of each set modality exists;

[0216] The feature extraction unit 420 is used to encode the features of each existing modality image to obtain multi-scale features, and at each scale, to fuse the features of all existing modality images to obtain a multi-scale global scene representation.

[0217] The feature reconstruction unit 430 is used to perform self-predictive reconstruction on the multi-scale features of each existing modality based on the multi-scale global scene representation, and when there are two or more modal images, to perform multi-scale feature cross-modal prediction reconstruction from the source modality to the target modality on each pair of existing modalities, so as to obtain self-reconstruction results and cross-modal reconstruction results respectively.

[0218] The error calculation unit 440 is used to calculate the multi-scale self-reconstruction error and multi-scale cross-modal reconstruction error corresponding to each modality based on the multi-scale features, self-reconstruction results and cross-modal reconstruction results, and to fuse and construct a multi-scale error map so as to obtain an initial saliency map by fusing the error maps of all scales.

[0219] The iterative calculation unit 450 is used to obtain the gaze region based on the multi-scale error map and the initial saliency map, and within each candidate gaze region, to crop the corresponding image block from each modal original image and enlarge it to a fixed high resolution to calculate the initial local saliency map, so as to obtain the updated saliency map based on the pixel difference between the initial local saliency map and the corresponding initial saliency map, and to re-execute the iterative calculation on the updated saliency map until the preset iteration exit condition is met to obtain the refined saliency map;

[0220] The saliency calculation unit 460 is used to statistically analyze the self-reconstruction error and cross-modal reconstruction error of all modalities and all scales, obtain a global error feature vector, input it into the gating network, generate adaptive weights, recalculate the multi-scale error map based on the adaptive weights for the self-reconstruction error and cross-modal reconstruction error, fuse all the recalculated multi-scale error maps to obtain a gating saliency map, and fuse the gating saliency map with the refined saliency map to obtain the final salient target detection result.

[0221] In some embodiments, the feature reconstruction unit 430 is configured to, for each existing modality, downsample the existing modality image to the corresponding scale resolution at each scale, and decode the global scene representation at the same scale using a self-reconstruction decoder to obtain a self-reconstruction result; and for each pair of existing modalities, decode the global scene representation at the same scale and the source modality features at each scale using a cross-modal reconstruction decoder to obtain a cross-modal reconstruction result of the target modality.

[0222] In some embodiments, the error calculation unit 440 is used to calculate the multi-scale self-reconstruction error and the multi-scale cross-modal reconstruction error based on the error metric function, wherein the error metric function is used to measure the pixel difference, local structural consistency difference and high frequency difference between the multi-scale features and the self-reconstruction result and between the multi-scale features and the cross-modal reconstruction result.

[0223] In some embodiments, the iterative calculation unit 450 is configured to generate a gaze region heatmap based on the multi-scale error map and the initial saliency map; apply a penalty term to the gaze region heatmap determined during the iteration process; divide the penalized gaze region heatmap into multiple grids, calculate the average heat value of each grid, and select a preset number of gaze regions based on a preset sorting algorithm.

[0224] In some embodiments, the iterative calculation unit 450 is used to upsample the error maps at each scale to the same resolution and perform weighted summation, normalize the weighted summation error maps to obtain a comprehensive saliency map, and perform weighted summation of the comprehensive saliency map and the initial saliency map to obtain the gaze region heatmap.

[0225] In some embodiments, the iterative calculation unit 450 is used to set the access mask map. The access mask map Size and heat map of the gaze area Similarly, the initial value of the access mask map is zero; the value corresponding to the gaze region determined during the iteration process will be marked as 1, and the gaze region heatmap after penalty processing is... .

[0226] In some embodiments, the iterative calculation unit 450 is used to obtain the soft mask and confidence weight of the gaze region corresponding to the initial local saliency map, wherein the soft mask has a pixel value of 1 at the center position and the pixel value decreases Gaussian from the center position to the edge; calculate the pixel difference between the mapped local saliency map and the initial local saliency map, and perform a weighted summation of the pixel difference based on the soft mask and the confidence weight, and perform a weighted fusion of the weighted summation result with the initial saliency map to obtain the updated saliency map.

[0227] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0228] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0229] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0230] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0231] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0232] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0233] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0234] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0235] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0236] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0237] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0238] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A salient target detection method based on multimodal prediction and reconstruction error, characterized in that, Includes the following steps: Step S1: Acquire images of at least one modality and establish a modality presence mask vector, wherein the modality presence mask vector is used to indicate whether an image of each set modality exists; Step S2: Encode the features of each existing modality image to obtain multi-scale features, and fuse the features of all existing modality images at each scale to obtain a multi-scale global scene representation. Step S3: Based on the multi-scale global scene representation, perform self-predictive reconstruction on the multi-scale features of each existing modality, and when there are two or more modal images, perform multi-scale feature cross-modal prediction reconstruction from the source modality to the target modality on each pair of existing modalities, and obtain self-reconstruction results and cross-modal reconstruction results respectively. Step S4: Based on the multi-scale features, self-reconstruction results and cross-modal reconstruction results, calculate the multi-scale self-reconstruction error and multi-scale cross-modal reconstruction error corresponding to each modality, and fuse them to construct a multi-scale error map, so as to obtain an initial saliency map by fusing the error maps of all scales. Step S5: Based on the multi-scale error map and the initial saliency map, the gaze region is obtained. Within each candidate gaze region, the corresponding image patch is cropped from each modal image and enlarged to a fixed high resolution to perform steps S2 to S4 to obtain an initial local saliency map. Based on the pixel difference between the initial local saliency map and the corresponding initial saliency map, an updated saliency map is obtained. Step S5 is then re-executed on the updated saliency map until the preset iteration exit condition is met to obtain a refined saliency map. Step S6: Statistically analyze the self-reconstruction errors and cross-modal reconstruction errors for all modalities and scales to obtain a global error feature vector, which is then input into a gating network to generate adaptive weights. Based on the adaptive weights, recalculate the multi-scale error map for the self-reconstruction errors and cross-modal reconstruction errors. Fuse all the recalculated multi-scale error maps to obtain a gating saliency map. Fuse the gating saliency map with the refined saliency map to obtain the final salient target detection result.

2. The method according to claim 1, characterized in that, Step S3 includes: For each existing modality, at each scale, the existing modality image is downsampled to the corresponding scale resolution, and the global scene representation at the same scale is decoded by a self-reconstruction decoder to obtain the self-reconstruction result; For each pair of existing modes, at each scale, the global scene representation and source mode features at the same scale are decoded by a cross-modal reconstruction decoder to obtain the cross-modal reconstruction result of the target mode.

3. The method according to claim 1, characterized in that, Step S4 includes: The multi-scale self-reconstruction error and the multi-scale cross-modal reconstruction error are calculated based on the error metric function, wherein the error metric function is used to measure the pixel differences, local structural consistency differences, and high-frequency differences between the multi-scale features and the self-reconstruction results, as well as between the multi-scale features and the cross-modal reconstruction results.

4. The method according to claim 1, characterized in that, Step S5 obtains the gaze area through the following steps: Based on the multi-scale error map and the initial saliency map, a heat map of the staring region is generated; A penalty term is applied to the gaze region heatmap determined during the iteration process; The heatmap of the gaze region after penalty processing is divided into multiple grids, the average heat value of each grid is calculated, and a preset number of gaze regions are selected based on a preset sorting algorithm.

5. The method according to claim 4, characterized in that, The step of generating a gaze region heatmap based on the multi-scale error map and the initial saliency map includes: The error maps at each scale are upsampled to the same resolution and then weighted and summed. The weighted and summed error maps are then normalized to obtain a comprehensive significance map. The heatmap of the gaze region is obtained by weighted summation of the comprehensive saliency map and the initial saliency map.

6. The method according to claim 4, characterized in that, The penalty term applied to the gaze region heatmap determined during the iteration process includes: Set access mask image The access mask map Size and heatmap of the gaze area Similarly, the initial value of the access mask map is zero; The values ​​corresponding to the gaze regions identified during the iteration process will be marked as 1. The heatmap of the gaze regions after penalty processing is as follows: .

7. The method according to claim 1, characterized in that, Step S5 obtains the updated saliency map through the following steps: Obtain the soft mask and confidence weight of the gaze region corresponding to the initial local saliency map. The soft mask has a pixel value of 1 at the center position and a Gaussian decay of pixel values ​​from the center position to the edge. Calculate the pixel difference between the initial local saliency map and the corresponding initial saliency map, and perform a weighted summation of the pixel differences based on the soft mask and the confidence weight. Then, perform a weighted fusion with the initial saliency map to obtain the updated saliency map.

8. A salient target detection device based on multimodal prediction and reconstruction error, characterized in that, The device includes: An image acquisition unit is configured to acquire an image of at least one modality and establish a modality presence mask vector, wherein the modality presence mask vector is used to indicate whether an image of each set modality exists; The feature extraction unit is used to encode the features of each existing modality image to obtain multi-scale features, and at each scale, the features of all existing modality images are fused to obtain a multi-scale global scene representation. The feature reconstruction unit is used to perform self-predictive reconstruction on the multi-scale features of each existing modality based on the multi-scale global scene representation, and when there are two or more modal images, to perform multi-scale feature cross-modal prediction reconstruction from the source modality to the target modality on each pair of existing modalities, and obtain self-reconstruction results and cross-modal reconstruction results respectively. The error calculation unit is used to calculate the multi-scale self-reconstruction error and multi-scale cross-modal reconstruction error corresponding to each modality based on the multi-scale features, self-reconstruction results and cross-modal reconstruction results, and to fuse and construct a multi-scale error map so as to obtain an initial saliency map by fusing the error maps of all scales. The iterative calculation unit is used to obtain the gaze region based on the multi-scale error map and the initial saliency map, and within each candidate gaze region, to crop the corresponding image patch from each modal image and enlarge it to a fixed high resolution to calculate the initial local saliency map. Based on the pixel difference between the initial local saliency map and the corresponding initial saliency map, an updated saliency map is obtained. The iterative calculation is then re-executed on the updated saliency map until a preset iteration exit condition is met to obtain a refined saliency map. The saliency calculation unit is used to statistically analyze the self-reconstruction error and cross-modal reconstruction error for all modalities and scales, obtain a global error feature vector, and input it into the gating network to generate adaptive weights. Based on the adaptive weights, the multi-scale error map is recalculated for the self-reconstruction error and cross-modal reconstruction error. All recalculated multi-scale error maps are fused to obtain a gated saliency map. The gated saliency map is then fused with the refined saliency map to obtain the final salient target detection result.

9. An electronic device, characterized in that, include: processor; as well as A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is executed by a processor according to any one of claims 1 to 7.

Images