Arbitrary modal saliency target detection method, system, device and medium

Abstract

A method, system, device and medium for detecting an arbitrary modal saliency target relate to the field of image processing, and the method comprises the following steps: collecting uncertain modal saliency target detection image data; constructing an arbitrary modal feature extractor, extracting multi-scale features from the collected image data, predicting the features belonging to modal categories, and calculating modal category loss; constructing a feature fusion model of any mode, and fusing the features of any input mode to obtain fused features; constructing a saliency target detection decoder, obtaining a saliency target detection result by using the multi-modal fusion features, calculating the cross entropy loss of the saliency target detection result and a real saliency target detection result, adding the cross entropy loss with the loss of a corresponding modal category, then calculating the gradient of the total loss, carrying out back propagation, and updating model parameters; the system, the equipment and the medium are used for implementing the method. The method reduces the training and deployment cost of various models, improves the calculation efficiency, and can adapt to various scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and pattern recognition, and in particular to a method, system, device and medium for detecting significant targets in any modality. Background Art

[0002] Salient object detection aims to simulate the human visual perception system to locate the most attractive object in the scene. It detects the most visually attractive object from the input image and simulates the human visual perception system to locate the most attractive object in the scene. With the rapid development of deep learning technology, the performance of salient detection has been greatly improved and has been widely used in various fields. Multimodal salient detection can enhance the model in diversified features by learning different forms of images. Multimodality can realize salient object detection in a variety of complex scenes. Multimodality can compensate for the missing information of other modalities and achieve better detection results. Existing salient object detection mainly focuses on salient object detection in specific scenes, that is, the modal information that can be obtained is known, such as known visible light modal images and depth modal images. The scene is relatively single and has its own limitations. Especially in practical applications, the modal information given by the current scene is not necessarily known. Therefore, how to achieve good salient object detection under any modal input is an urgent problem to be solved. In this case, arbitrary modal salient object detection technology is proposed, aiming to achieve salient object detection under arbitrary modal input.

[0003] Arbitrary modality salient object detection faces two major challenges: how to use a feature extractor to achieve better feature extraction operations for arbitrary modality data and how to achieve better modality information fusion under dynamic input conditions. The feature extractor is due to the fact that the number and type of input modes cannot be determined under arbitrary modality input conditions. Therefore, if a specific feature extractor is set for each modality information, this will lead to an explosive increase in the number of model parameters, which violates the requirement of arbitrary modality salient object detection to use only one model for feature extraction of all features. Arbitrary modality feature fusion is also due to the uncertainty of the input modality, resulting in the existing structures for specific modality input being unsuitable for arbitrary modality feature fusion.

[0004] Most existing models only focus on salient object detection under specific modal inputs, and there is a lack of research on indefinite modal salient object detection. However, arbitrary modalities are very common in real-world scenarios. For example, robots, mobile phones, and drones often do not carry all modal acquisition devices, and the modalities that may be used in each real-world scenario do not overlap. In daytime and other situations, only visible light modality may be used; depth or infrared modal features may be used in dark light and other scenes; event cameras may be used to obtain dynamic point features in high-speed moving scenes. This makes it impossible to design an algorithm for each scenario, because it will result in a huge and redundant number of algorithm parameters.

[0005] The Chinese patent application with publication number CN 118262093 A discloses an RGB-D salient object detection method based on hierarchical cross-modal attention and cascaded aggregate decoding. However, since it can only be used in RGB-D scenes, it has the disadvantage of a single scene and cannot adapt to scenes with multiple input data. In addition, its fusion method is single and can only fuse RGB-D data, and cannot achieve feature fusion under any type of modality data input.

[0006] Han Junwei and others published the Visual Saliency Transformer (Liu N, Zhang N, Wan K, etal. Visual saliency transformer[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 4722-4732.), which uses a twin network structure to extract features from visible light and depth images. However, it cannot extract features from multiple modal data inputs. If the modality is increased, the network model structure increases, resulting in a huge model, which has the disadvantage of being unable to adapt to more modal data inputs. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, device and medium for detecting significant targets in any modality. Based on the modal feature extractor (MSFE) of the modal prompter, the corresponding modal features can be extracted from indefinite modal inputs, and the model parameters will not be increased under multi-modal inputs; through the arbitrary modal feature fusion model (DFM), the fusion of features can be completed under any number of modal inputs, thereby realizing the self-adaptation of multi-type modal fusion.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is: A method for detecting salient objects in any modality comprises the following steps: Step 1: Collect and integrate multimodal salient object detection datasets to obtain indefinite modality salient object detection datasets. For k indefinite modality requirements, collect image data containing these k indefinite modalities. Step 2: Construct an arbitrary modality feature extractor (MSFE) based on the modality prompter. The arbitrary modality feature extractor (MSFE) uses the improved Resnet50 model structure to realize the corresponding modality feature extraction of any modality under the corresponding modality prompt input. The feature is a multi-scale feature. For the image data of k kinds of indefinite modalities collected in step 1, input them into the arbitrary modality feature extractor (MSFE) to obtain the modality features of the modality image data , and the modal features extracted for each input modal data Make predictions for each modal category and calculate the corresponding modal category loss ; Step 3: Construct a dynamic fusion method for arbitrary modal features. Through the improved Transformer network model, a dynamic fusion model (DFM) for arbitrary modal features is constructed to fuse the features extracted from a random number of modalities. The modal features of the indefinite number of image data obtained in step 2 are Perform data fusion to obtain multimodal fusion features ; Step 4: Construct a salient object detection decoder SPD, and realize multi-scale feature fusion through upsampling + fusion method, and transform the multimodal fusion features obtained in step 3 Input into the salient object detection decoder SPD to obtain the final output salient object detection result , calculate the salient target detection results The cross entropy loss with the true salient target detection result and the corresponding modality category loss calculated in step 2 The gradient of the total loss is then calculated to backpropagate and update the model parameters.

[0009] The step 2 of constructing a modality feature extractor (MSFE) based on a modality indicator specifically includes: extracting features from the image data of k types of indefinite modalities collected in step 1 to obtain features of the corresponding modalities, such as the formula ,in: represents the arbitrary modal feature extractor (MSFE), and the parameters of the module are ; Indicates belonging to The input data of the modality, , represents k types of modal data; express Features extracted by the random modal feature extractor (MSFE); The steps of extracting features through the random modal feature extractor (MSFE) are as follows: First go through Will The initial features are extracted through convolution operation , Indicates scales, among which , Indicates modal, , Represents the convolutional layer network, and the network parameters are ; Next, go through The module obtains m-mode and i-scale features The weight of the reminder parameter , where the modal prompter Indicates the current input mode of the module. Represents the mapping network, and the parameters are , Represents the feature concatenation function, which concatenates features into a new feature according to a certain dimension. Represents the global pooling function; Last passed The function multiplies the weights by the preliminary features to obtain the final features after specific mode selection. ; The above represents a modal feature extractor (MSFE) based on a modal indicator. , realize the feature extraction of single modality feature extractor (MSFE) for any modality input; and in order to enable the learned features to represent the corresponding modality, that is, the extracted features can reflect which modality the current input is, use the modality-aware loss device, the specific steps are as follows: First, the features extracted by the arbitrary modal feature extractor (MSFE) Select the scale feature i=5 ,go through Function gets the predicted modal category ,in, Represents a linear mapping layer with parameters , Represents the Softmax function, which maps features to a nonlinear function from 0 to 1; Next use , the predicted modal category Classification with True Mode Do cross entropy loss, represents the cross entropy loss function.

[0010] The method of constructing a dynamic fusion model (DFM) of arbitrary modal features in step 3 is specifically as follows: using a feature fusion module based on a Transformer structure to transform the modal features obtained in step 2 To perform fusion, the dynamic fusion model (DFM) of any modal feature is expressed by a single function as follows: in, Indicates that the parameter is Arbitrary modal feature dynamic fusion model (DFM), Represents the fusion operation for single-modal input, Represents the fusion operation under two modal inputs, represents the fusion operation under k modal input conditions, That is, the features after DFM fusion; The steps of DFM feature fusion are as follows: 1. After Function, specifically for the i-th scale feature The parameters are of Downsampling convolution function , get the initial features , ; 2. Features of N input modes Operation, to obtain new fusion features that combine multiple modalities, It represents the feature concatenation operation under N modal input conditions, and the scale after concatenation is , N represents the number of modes, C represents the feature channel dimension, H and W represent the feature scale; 3. Pass The function calculates the query value Q, key value K, and feature value V required by Transformer, which are expressed as ,in, represents the extraction network of the key value K, represents the extraction network of the query value Q, Represents the extraction network of eigenvalue V, and its parameters are , Represents the splicing features obtained after feature splicing; Fourth, calculate the similarity relationship between the query value Q and the key value K. First, the feature Extracted according to the channel layer , and then by Convert the dimensions to get , Indicates that the feature is changed from Transform to ; 5. Pass Calculate the correlation weight between the query value Q and the key value K , Next, multiply the weight by the eigenvalue V and transform the dimension to get The features after channel fusion , , c represents the cth channel, , i represents the i-th scale, , N represents the Nth mode, , Represents the Softmax function, which maps the feature to a nonlinear function from 0 to 1. express The reverse operation is Transformed to ; VI. Pass Take the mean of the modal channels to obtain the fused features , ,in represents the features after channel merging, represents the average pooling function in the modality dimension; 7. Based on the above steps 1 to 6, the arbitrary modal features of each scale are fused to obtain the fused features at that scale. By adding feature dimensions, the fusion of arbitrary modalities is realized to meet the requirement of feature fusion under arbitrary modal input, and finally the fused features at 5 scales are obtained: .

[0011] The construction of the salient target detection decoder SPD in step 4 is based on the multimodal fusion features obtained by fusion in step 3 Finally, the salient targets in the image data are detected. Through the upsampling + fusion method, the salient target detection results at each scale are obtained. The specific steps are as follows: First, through the function Small-scale features Upsampling and large-scale features The new scale features are obtained by splicing and feature dimensionality reduction. , where i represents the i-th scale, Represents the multimodal fusion features obtained in step 3 , represents the upsampling operation, that is, sampling small-scale features into large-scale features. Indicates that the parameter is The convolution operation is used to fuse the up-sampled features with the i-1 scale features; For the features after splicing use Predict the salient target detection results at the i-th scale , It represents the detection result of the salient target at the i-th scale. The result is displayed as a mask, that is, the target pixel position is represented as 1 and the background position is represented as 0. Indicates that the parameter is The convolution operation is used to finally output the target detection mask; In order to enable the feature extraction, feature fusion, and target detection model to update parameters, in addition to the modal category loss in step 2 , for each scale detected target , adding pixel-level cross entropy loss, expressed as a function ,in represents the true result of the saliency target at the i-th scale; The final total loss is .

[0012] An arbitrary modality salient object detection system, comprising: The data collection module collects and integrates the multimodal salient object detection dataset to obtain the indefinite modality salient object detection dataset. For k indefinite modal requirements, the image data containing these k indefinite modalities are collected. The arbitrary modality feature extraction module builds an arbitrary modality feature extractor (MSFE) based on the modality prompter. The arbitrary modality feature extractor (MSFE) uses the improved Resnet50 model structure to realize the corresponding modality feature extraction of arbitrary modality under the corresponding modality prompt input. The feature is a multi-scale feature. For image data of multiple indefinite modalities, it is input into the arbitrary modality feature extractor (MSFE) to obtain the modality features of the modality image data , and the modal features extracted for each input modal data Make predictions for each modal category and calculate the corresponding modal category loss ; The arbitrary modal feature fusion module constructs a dynamic fusion method for arbitrary modal features. Through the improved Transformer network model, the arbitrary modal feature fusion model (DFM) is used to fuse the features extracted from random number of modalities to perform modal features on the image data. Perform data fusion to obtain multimodal fusion features ; The salient object detection module builds a salient object detection decoder SPD, realizes multi-scale feature fusion through upsampling + fusion method, and integrates multi-modal fusion features Input into the salient object detection decoder SPD to obtain the final output salient object detection result , calculate the salient target detection results The cross entropy loss with the true salient target detection result and the corresponding modality category loss The gradient of the total loss is then calculated to backpropagate and update the model parameters.

[0013] A device for detecting a salient object in any modality, comprising: Memory: used for storing a computer program for implementing the arbitrary modality salient object detection method; Processor: used to implement the arbitrary modality salient target detection method when executing the computer program.

[0014] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the arbitrary modality salient object detection method are implemented.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. The present invention constructs a modal feature extractor (MSFE) based on a modal prompter, which instructs the feature extraction network to learn the features of any modality through the modal prompt. It has the ability to extract features of any input modality, avoids the use of multiple models, reduces the number of models required, and reduces the cost of training and deploying multiple models.

[0016] 2. The present invention constructs a dynamic fusion method for arbitrary modal features. Through the improved Transformer structure, it can fuse any number and arbitrary modal features. It can adaptively fuse multiple modal input features, improve computing efficiency, reduce the computing cost of the model, and adapt to various scenarios.

[0017] 3. The present invention realizes multi-scale feature fusion through the upsampling + fusion method, obtains significant target detection results at multiple scales, and has the characteristics of feature diversity.

[0018] In summary, the present invention has the characteristics of being able to extract features of any input modality, adaptively fuse multiple modal input features, and feature diversity, avoiding the use of multiple models, reducing the number of required models, reducing the cost of training and deploying multiple models, improving computing efficiency, reducing the computing cost of the model, and being able to adapt to a variety of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention discloses a flowchart of a method for detecting salient targets in any modality.

[0020] Figure 2 This is the salient object detection network proposed in this invention.

[0021] Figure 3 This is a schematic diagram of the dynamic feature fusion and salient target detection structure proposed in the present invention.

[0022] Figure 4 Schematic diagram of the data set selection and integration method of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0024] like Figure 1 As shown, a method for detecting salient objects in any modality includes the following steps: Step 1: Collect and integrate multimodal salient object detection datasets to obtain indefinite modality salient object detection datasets. For k indefinite modality requirements, collect image data containing these k indefinite modalities. Step 2: Construct a modality-based MSFE, such as Figure 2 As shown in the figure, the arbitrary modality feature extractor (MSFE) uses the improved Resnet50 model structure to realize the corresponding modality feature extraction of arbitrary modality under the corresponding modality prompt input. The feature is a multi-scale feature. The image data of k kinds of indefinite modalities collected in step 1 are input into the arbitrary modality feature extractor (MSFE) to obtain the modality features of the modality image data. That is, for any number of modal inputs, the modal feature extractor (MSFE) based on the modal prompt is used to complete the feature extraction of any modality, and the modal features extracted for each input modal data are Make predictions for each modal category and calculate the corresponding modal category loss ; Step 3: Construct a dynamic fusion method for arbitrary modal features, such as Figure 3 As shown in the figure, an arbitrary modal feature dynamic fusion model (DFM) is constructed through the improved Transformer network model to fuse the features extracted from the random number of modalities and the modal features of the indefinite number of image data obtained in step 2. Perform data fusion to obtain multimodal fusion features ; Step 4: Construct a salient object detection decoder SPD, and realize multi-scale feature fusion through upsampling + fusion method, and transform the multimodal fusion features obtained in step 3 Input into the salient object detection decoder SPD to obtain the final output salient object detection result , calculate the salient target detection results The cross entropy loss with the true salient target detection result and the corresponding modality category loss calculated in step 2 The gradient of the total loss is then calculated for backpropagation and updating of model parameters for salient object detection.

[0025] like Figure 2 As shown, the step 2 of constructing a modality feature extractor (MSFE) based on a modality prompter specifically includes: extracting features from the k kinds of indefinite modal image data collected in step 1 to obtain features of the corresponding modality, as shown in formula ,in: represents the arbitrary modal feature extractor (MSFE), and the parameters of the module are ; Indicates belonging to The input data of the modality, , represents k types of modal data; express Features extracted by the random modal feature extractor (MSFE); The steps of extracting features through the random modal feature extractor (MSFE) are as follows: First go through Will The initial features are extracted through convolution operation , Indicates scales, among which , Indicates modal, , Represents the convolutional layer network, and the network parameters are ; Next, go through The module obtains m-mode and i-scale features The weight of the reminder parameter , where the modal prompter Indicates the current input mode of the module. Represents the mapping network, and the parameters are , Represents the feature concatenation function, which concatenates features into a new feature according to a certain dimension. Represents the global pooling function; Last passed The function multiplies the weights by the preliminary features to obtain the final features after specific mode selection. ; The above represents a modal feature extractor (MSFE) based on a modal indicator. , realize the feature extraction of single modality feature extractor (MSFE) for any modality input; and in order to enable the learned features to represent the corresponding modality, that is, the extracted features can reflect which modality the current input is, use the modality-aware loss device, the specific steps are as follows: First, the features extracted by the arbitrary modal feature extractor (MSFE) Select the scale feature i=5 ,go through Function gets the predicted modal category ,in, Represents a linear mapping layer with parameters , Represents the Softmax function, which maps features to a nonlinear function from 0 to 1; Next use , the predicted modal category Classification with True Mode Do cross entropy loss, represents the cross entropy loss function.

[0026] like Figure 3 As shown, the method of constructing a dynamic fusion model (DFM) of arbitrary modal features in step 3 is specifically as follows: using a feature fusion module based on a Transformer structure to transform the modal features obtained in step 2 To perform fusion, the dynamic fusion model (DFM) of any modal feature is expressed by a single function as follows: in, Indicates that the parameter is The arbitrary modal feature dynamic fusion model (DFM) Represents the fusion operation for single-modal input, Represents the fusion operation under two modal inputs, represents the fusion operation under k modal input conditions, That is, the features after DFM fusion; The steps of DFM feature fusion are as follows: 1. After Function, specifically for the i-th scale feature The parameters are of Downsampling convolution function , get the initial features , ; 2. Features of N input modes Operation, to obtain new fusion features that combine multiple modalities, It represents the feature concatenation operation under N modal input conditions. Unlike the previous concatenation in the feature channel dimension, the concatenation here is in a new dimension, which we call the modal dimension. That is, the scale after concatenation is , N represents the number of modes, C represents the feature channel dimension, H and W represent the feature scale; 3. Pass The function calculates the query value Q, key value K, and feature value V required by Transformer, which are expressed as ,in, represents the extraction network of the key value K, represents the extraction network of the query value Q, Represents the extraction network of eigenvalue V, and its parameters are , Represents the splicing features obtained after feature splicing; Fourth, calculate the similarity relationship between the query value Q and the key value K. First, the feature Extracted according to the channel layer , and then by Convert the dimensions to get , Indicates that the feature is changed from Transform to ; 5. Pass Calculate the correlation weight between the query value Q and the key value K , Next, multiply the weight by the eigenvalue V and transform the dimension to get The features after channel fusion , , c represents the cth channel, , i represents the i-th scale, , N represents the Nth mode, , Represents the Softmax function, which maps the feature to a nonlinear function from 0 to 1. express The reverse operation is Transformed to ; VI. Pass Take the mean of the modal channels to obtain the fused features , ,in represents the features after channel merging, represents the average pooling function in the modality dimension; 7. Based on the above steps 1 to 6, the arbitrary modal features of each scale are fused to obtain the fused features at that scale. By adding feature dimensions, the fusion of arbitrary modalities is realized to meet the requirement of feature fusion under arbitrary modal input, and finally the fused features at 5 scales are obtained: .

[0027] The construction of the salient target detection decoder SPD in step 4 is based on the multimodal fusion features obtained by fusion in step 3 Finally, the salient targets in the image data are detected. Through the upsampling + fusion method, the salient target detection results at each scale are obtained. The specific steps are as follows: First, through the function Small-scale features Upsampling and large-scale features The new scale features are obtained by splicing and feature dimensionality reduction. , where i represents the i-th scale, Represents the multimodal fusion features obtained in step 3 , represents the upsampling operation, that is, sampling small-scale features into large-scale features. Indicates that the parameter is The convolution operation is used to fuse the up-sampled features with the i-1 scale features; For the features after splicing use Predict the salient target detection results at the i-th scale , It represents the detection result of the salient target at the i-th scale. The result is displayed as a mask, that is, the target pixel position is represented as 1 and the background position is represented as 0. Indicates that the parameter is The convolution operation is used to finally output the target detection mask; In order to enable the feature extraction, feature fusion, and target detection model to update parameters, in addition to the modal category loss in step 2 , for each scale detected target , adding pixel-level cross entropy loss, expressed as a function ,in represents the true result of the saliency target at the i-th scale; The final total loss is .

[0028] An arbitrary modality salient object detection system, comprising: The data collection module collects and integrates the multimodal salient target detection dataset to obtain an indefinite modality salient target detection dataset. For k indefinite modal requirements, the image data containing the k indefinite modalities are collected to implement step 1 of an arbitrary modality salient target detection method. The arbitrary modality feature extraction module builds an arbitrary modality feature extractor (MSFE) based on the modality prompter. The arbitrary modality feature extractor (MSFE) uses the improved Resnet50 model structure to realize the corresponding modality feature extraction of arbitrary modality under the corresponding modality prompt input. The feature is a multi-scale feature. For image data of multiple indefinite modalities, it is input into the arbitrary modality feature extractor (MSFE) to obtain the modality features of the modality image data , and the modal features extracted for each input modal data Make predictions for each modal category and calculate the corresponding modal category loss , used to implement step 2 of an arbitrary modality salient object detection method; The arbitrary modal feature fusion module constructs a dynamic fusion method for arbitrary modal features. Through the improved Transformer network model, the arbitrary modal feature fusion model (DFM) is used to fuse the features extracted from random number of modalities to perform modal features on the image data. Perform data fusion to obtain multimodal fusion features , used to implement step 3 of an arbitrary modality salient object detection method; The salient object detection module builds a salient object detection decoder SPD, realizes multi-scale feature fusion through upsampling + fusion method, and integrates multi-modal fusion features Input into the salient object detection decoder SPD to obtain the final output salient object detection result , calculate the salient target detection results The cross entropy loss with the true salient target detection result and the corresponding modality category loss Add, and then calculate the gradient of the total loss for back propagation and update the model parameters to implement step 4 of an arbitrary modality salient object detection method.

[0029] A device for detecting a salient object in any modality, comprising: Memory: used for storing a computer program for implementing the arbitrary modality salient object detection method; Processor: used to implement the arbitrary modality salient target detection method when executing the computer program.

[0030] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the arbitrary modality salient target detection method are implemented.

[0031] Simulation experiment 1. Simulation conditions All simulation experiments were carried out on an Ubuntu 20.04.5 operating system and a GPU Nvidia GeForceGTX 2080Ti hardware environment. The PyTorch deep learning framework was used to implement the salient object detection datasets in 10 different scenarios. The training datasets finally formed included: 5000 visible light images, 2985 visible light-depth image pairs, 2500 visible light-infrared image pairs, and 1048 visible light-depth-infrared image pairs; the test datasets included: 5000 visible light images, 3121 visible light-depth image pairs, 4321 visible light-infrared image pairs, and 1000 visible light-depth-infrared image pairs. Figure 4 shown.

[0032] In step 1, after training a part of the data set of various modal combinations, the other part of the data is tested using the model. The salient target detection results obtained by the model are compared with the real results. The similarity between the detection results and the real results is compared, and the following indicators are used: (1) Mean absolute error M in, and Represent the predicted value and the true value respectively, which are used to measure the average difference between the predicted significant target detection results and the true significant target results; (2) F index in, By convention, it is set to 0.3. represents the accuracy of the prediction, Represents the predicted recall rate. The F index is used to comprehensively evaluate the precision and recall rate of significant target detection. (3) S index and It represents the target structure similarity and the regional structure similarity, and it represents the structural similarity between the detection result and the true result; (4) E index in, The bias matrix representing the true target value, represents the bias matrix for predicting target values, represents the Hadamard product, and the E indicator is used to capture the global statistical information and local pixel matching information in the saliency map.

[0033] 2. Simulation content and result analysis The results of experiments on the data sets collected and collated in the present invention using the salient target detection methods combining the present invention with various existing modalities are objectively evaluated using the above evaluation indicators. The evaluation simulation results are shown in Tables 1, 2, 3, 4, 5, 6, and 7: Table 1 Experimental results on the collected datasets

[0034] Among them, OUR represents the method proposed by the present invention, RGB represents visible light image data, T represents infrared image data, D represents depth image data, and SOD represents significant object detection.

[0035] Table 2 The present invention collects data sets

[0036] Among them, OUR represents the method proposed by the present invention, RGB represents visible light image data, T represents infrared image data, D represents depth image data, and SOD represents significant object detection.

[0037] Table 3 Ablation experiment of the method of the present invention

[0038] Among them, RGB represents visible light image data, T represents infrared image data, D represents depth image data, and M under the parameter represents the model size.

[0039] Among them, Baseline represents the baseline model, which removes the modality-cue-based arbitrary modality feature extractor (MSFE), arbitrary modality feature fusion model (DFM) and modality-specific feature extractor (MS).

[0040] Table 4 M index and F index under different inputs

[0041] Among them, RGB represents visible light image data, T represents infrared image data, and D represents depth image data.

[0042] Table 5 Detection efficiency experiment of the method of the present invention

[0043] Among them, RGB represents visible light image data, T represents infrared image data, D represents depth image data, and SOD represents salient object detection.

[0044] FLOPS indicates the amount of computing required by the model, Speed ​​indicates the inference speed, and Params indicates the number of model parameters.

[0045] Table 6 Ablation experiment of modal feature extractor based on modal cue

[0046] Among them, RGB represents visible light image data, T represents infrared image data, D represents depth image data, and SOD represents salient object detection.

[0047] w / o means removing this module, w means having this module, MID means modal indicator, DFM means modal fusion module, MIWG means modal indicator weight generator, and L_modality means modal perception loss.

[0048] Table 7 Index changes after changing the fusion strategy

[0049] Among them, w / o means removing this module, w means including this module, Baseline means the basic network, DFM means the modal fusion module, Channel means the channel dimension fusion between modalities, and Global means the global feature fusion.

[0050] The lower the M index, the better, the higher the F index, the better, the lower the S index, the better, and the higher the E index, the better.

[0051] The results shown in Tables 1 and 2 confirm the effectiveness of the model designed by the present invention. Due to the use of arbitrary modalities for training, compared with the model specifically used for visible light, the M index is 0.055 lower and the F index is 0.803 lower, but it is higher than other methods in depth modality and infrared modality; on the multimodal fusion data set, under the setting of all modalities, the M index is 0.049 higher and the F index is 0.816 higher. From these indicators, it can be seen that the present invention is superior to other methods under most modal combination settings and is suitable for arbitrary modal settings. From Tables 3 to 7, it can be seen that the effectiveness of the module designed by the present invention can realize reasoning and significant target detection in any modality. By removing the arbitrary modal feature extractor (MSFE) based on the modal prompter and the arbitrary modal feature fusion model (DFM), it can be seen that the effectiveness of the method proposed by the present invention is good, and through the detection efficiency experiment, it can be seen that the real-time performance of the present invention is good, which can reach more than 35 frames, meeting the real-time requirements. Through this series of experiments, the effectiveness and superiority of the method of the present invention are fully demonstrated.

[0052] The embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge of ordinary technicians in the relevant technical field without departing from the purpose of the present invention.

[0053] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting salient objects in any modality, characterized in that: The following steps are involved: Step 1: Collect and integrate multimodal salient object detection datasets to obtain indefinite modality salient object detection datasets. For k indefinite modality requirements, collect image data containing these k indefinite modalities. Step 2: Construct an arbitrary modality feature extractor (MSFE) based on the modality prompter. The arbitrary modality feature extractor (MSFE) uses the improved Resnet50 model structure to realize the corresponding modality feature extraction of any modality under the corresponding modality prompt input. The feature is a multi-scale feature. For the image data of k kinds of indefinite modalities collected in step 1, input them into the arbitrary modality feature extractor (MSFE) to obtain the modality features of the modality image data , and the modal features extracted for each input modal data Make predictions for each modal category and calculate the corresponding modal category loss ; Step 3: Construct a dynamic fusion method for arbitrary modal features. Through the improved Transformer network model, a dynamic fusion model (DFM) for arbitrary modal features is constructed to fuse the features extracted from a random number of modalities. The modal features of the indefinite number of image data obtained in step 2 are Perform data fusion to obtain multimodal fusion features ; Step 4: Construct a salient object detection decoder SPD, and realize multi-scale feature fusion through upsampling + fusion method, and transform the multimodal fusion features obtained in step 3 Input into the salient object detection decoder SPD to obtain the final output salient object detection result , calculate the significant target detection results The cross entropy loss with the true salient target detection result and the corresponding modality category loss calculated in step 2 The gradient of the total loss is then calculated to backpropagate and update the model parameters.

2. The arbitrary modality salient object detection method according to claim 1, characterized in that: The step 2 of constructing a modality feature extractor (MSFE) based on a modality indicator specifically includes: extracting features from the image data of k types of indefinite modalities collected in step 1 to obtain features of the corresponding modalities, such as the formula ,in: represents the arbitrary modal feature extractor (MSFE), and the parameters of the module are ; Indicates belonging to The input data of the modality, , represents k types of modal data; express Features extracted by the random modal feature extractor (MSFE); The steps of extracting features through the random modal feature extractor (MSFE) are as follows: First go through Will The initial features are extracted through convolution operation , Indicates scales, among which , Indicates modal, , Represents the convolutional layer network, and the network parameters are ; Next, go through The module obtains m-mode and i-scale features The weight of the reminder parameter , where the modal prompter Indicates the current input mode of the module. Represents the mapping network, and the parameters are , Represents the feature concatenation function, which concatenates features into a new feature according to a certain dimension. Represents the global pooling function; Last passed The function multiplies the weights by the preliminary features to obtain the final features after specific mode selection. ; The above represents a modal feature extractor (MSFE) based on a modal indicator. , realize the feature extraction of single modality feature extractor (MSFE) for any modality input; and in order to enable the learned features to represent the corresponding modality, that is, the extracted features can reflect which modality the current input is, use the modality-aware loss device, the specific steps are as follows: First, the features extracted by the arbitrary modal feature extractor (MSFE) Select the scale feature i=5 ,go through Function gets the predicted modal category ,in, Represents a linear mapping layer with parameters , Represents the Softmax function, which maps features to a nonlinear function from 0 to 1; Next use , the predicted modal category Classification with True Mode Do cross entropy loss, represents the cross entropy loss function.

3. The arbitrary modality salient object detection method according to claim 1, characterized in that: The method of constructing a dynamic fusion model (DFM) of arbitrary modal features in step 3 is specifically as follows: using a feature fusion module based on a Transformer structure to transform the modal features obtained in step 2 To perform fusion, the dynamic fusion model (DFM) of any modal feature is expressed by a single function as follows: in, Indicates that the parameter is The arbitrary modal feature dynamic fusion model (DFM) Represents the fusion operation for single-modal input, Represents the fusion operation under two modal inputs, represents the fusion operation under k modal input conditions, That is, the features after DFM fusion; The steps of DFM feature fusion are as follows:

1. After Function, specifically for the i-th scale feature The parameters are of Downsampling convolution function , get the initial features , ; 2. Features of N input modes Operation, to obtain new fusion features that combine multiple modalities, It represents the feature concatenation operation under N modal input conditions, and the scale after concatenation is , N represents the number of modes, C represents the feature channel dimension, H and W represent the feature scale; 3. Pass The function calculates the query value Q, key value K, and feature value V required by Transformer, which are expressed as ,in, represents the extraction network of the key value K, represents the extraction network of the query value Q, Represents the extraction network of eigenvalue V, and its parameters are , Represents the splicing features obtained after feature splicing; Fourth, calculate the similarity relationship between the query value Q and the key value K. First, the feature Extracted according to the channel layer , and then by Convert the dimensions to get , Indicates that the feature is changed from Transform to ; 5. Pass Calculate the correlation weight between the query value Q and the key value K , Next, multiply the weight by the eigenvalue V and transform the dimension to get The features after channel fusion , , c represents the cth channel, , i represents the i-th scale, , N represents the Nth mode, , Represents the Softmax function, which maps the feature to a nonlinear function from 0 to 1. express The reverse operation is Transformed to ; VI. Pass Take the mean of the modal channels to obtain the fused features , ,in represents the features after channel merging, represents the average pooling function in the modality dimension; 7. Based on the above steps 1 to 6, the arbitrary modal features of each scale are fused to obtain the fused features at that scale. By adding feature dimensions, the fusion of arbitrary modalities is realized to meet the requirement of feature fusion under arbitrary modal input, and finally the fused features at 5 scales are obtained: .

4. The arbitrary modality salient object detection method according to claim 1, characterized in that: The construction of the salient target detection decoder SPD in step 4 is based on the multimodal fusion features obtained by fusion in step 3 Finally, the salient targets in the image data are detected. Through the upsampling + fusion method, the salient target detection results at each scale are obtained. The specific steps are as follows: First, through the function Small-scale features Upsampling and large-scale features The new scale features are obtained by splicing and feature dimensionality reduction. , where i represents the i-th scale, Represents the multimodal fusion features obtained in step 3 , represents the upsampling operation, that is, sampling small-scale features into large-scale features. Indicates that the parameter is The convolution operation is used to fuse the up-sampled features with the i-1 scale features; For the features after splicing use Predict the salient target detection results at the i-th scale , It represents the detection result of the salient target at the i-th scale. The result is displayed as a mask, that is, the target pixel position is represented as 1 and the background position is represented as 0. Indicates that the parameter is The convolution operation is used to finally output the target detection mask; In order to enable the feature extraction, feature fusion, and target detection model to update parameters, in addition to the modal category loss in step 2 , for each scale detected target , adding pixel-level cross entropy loss, expressed as a function ,in represents the true result of the saliency target at the i-th scale; The final total loss is .

5. A detection system based on the arbitrary modality salient object detection method according to any one of claims 1 to 4, characterized in that: include: The data collection module collects and integrates the multimodal salient object detection dataset to obtain the indefinite modality salient object detection dataset. For k indefinite modal requirements, the image data containing these k indefinite modalities are collected. The arbitrary modality feature extraction module builds an arbitrary modality feature extractor (MSFE) based on the modality prompter. The arbitrary modality feature extractor (MSFE) uses the improved Resnet50 model structure to realize the corresponding modality feature extraction of arbitrary modality under the corresponding modality prompt input. The feature is a multi-scale feature. For image data of multiple indefinite modalities, it is input into the arbitrary modality feature extractor (MSFE) to obtain the modality features of the modality image data , and the modal features extracted for each input modal data Make predictions for each modal category and calculate the corresponding modal category loss ; The arbitrary modal feature fusion module constructs a dynamic fusion method for arbitrary modal features. Through the improved Transformer network model, the arbitrary modal feature fusion model (DFM) is used to fuse the features extracted from random number of modalities to perform modal features on the image data. Perform data fusion to obtain multimodal fusion features ; The salient object detection module builds a salient object detection decoder SPD, realizes multi-scale feature fusion through upsampling + fusion method, and integrates multi-modal fusion features Input into the salient object detection decoder SPD to obtain the final output salient object detection result , calculate the significant target detection results The cross entropy loss with the true salient target detection result and the corresponding modality category loss The gradient of the total loss is then calculated to backpropagate and update the model parameters.

6. An arbitrary modality salient object detection device, characterized in that: include: Memory: used to store a computer program for implementing a method for detecting a salient object in any modality as claimed in any one of claims 1 to 4; Processor: used to implement a method for detecting salient targets in any modality as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a method for detecting a salient target in any modality as described in any one of claims 1 to 4 are implemented.

Images