Adaptive multi-modal scene segmentation method based on attention mechanism in dark and weak environment

Abstract

The application provides an adaptive multi-modal scene segmentation method based on an attention mechanism in a dark and weak environment, comprising: inputting an RGB image and a thermal image collected in a dark and weak environment into corresponding encoders respectively to obtain an RGB image feature and a deep thermal image feature; inputting the RGB image feature and the deep thermal image feature into an adaptive module, combining an artificially labeled environment label, obtaining a thermal image credibility, and thus obtaining an RGB image credibility; sequentially transmitting each layer output of the encoder into a multi-modal feature fusion module based on an attention mechanism to obtain two feature stream results of the RGB image and the thermal image; and transmitting the two feature stream results of the RGB image and the thermal image into a decoder module for feature fusion to obtain a scene segmentation result. The method has the advantages of being able to effectively cope with complex image scenes, and being able to produce good results for different environmental conditions.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image processing, and particularly relates to an adaptive multi-modal scene segmentation method based on an attention mechanism in a dark and weak environment. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] In recent years, scene segmentation, as a basic but challenging task in computer vision, has received increasing research attention. Scene segmentation divides an image into different objects or regions, in other words, it is a process of classifying each pixel in the image. With the continuous development of computer vision technology, scene segmentation has been widely used in many fields, such as path planning, autonomous driving and video surveillance.

[0004] With the emergence of the first scene segmentation network FCN, many networks for semantic segmentation through CNN have emerged, such as Fast SCNN, DABNet, BiSeNet, etc., which have significant performance in speed and accuracy. However, most of the existing scene segmentation networks based on deep learning mainly process RGB images. However, RGB images can provide less information for model training, and in the case of similar texture and complex environment, inaccurate prediction results will be produced.

[0005] In a dark and weak environment (such as night, heavy fog, sandstorm), single RGB image information is not enough to effectively handle the changes in the scene. With the popularity of thermal imagers, researchers have found that thermal information is often robust and effective in dark and weak environments. Therefore, the thermogram created by the thermal imager can be used as an important information supplement to improve the performance of RGB-T semantic segmentation.

[0006] In order to adapt to the dark and weak environment, how to combine the thermal map from the thermal sensor and the traditional RGB image to realize scene segmentation in the dark and weak environment is the technical problem to be solved by the present application. SUMMARY

[0007] In order to overcome the shortcomings of the prior art, the present application provides an adaptive multi-modal scene segmentation method based on an attention mechanism in a dark and weak environment, which combines the thermal map from the thermal sensor and the traditional RGB image, and realizes scene segmentation in a dark and weak environment through a multi-modal feature fusion module based on an attention mechanism and an adaptive module.

[0008] To achieve the above purpose, one or more embodiments of the present application provide the following technical solutions:

[0009] In a first aspect, a self-adaptive multi-modal scene segmentation method based on an attention mechanism in a dark and weak environment is disclosed, comprising:

[0010] Input the RGB image and the thermal image collected in the dark and weak environment into the corresponding encoder respectively to obtain the deep features of the RGB image and the thermal image;

[0011] Input the deep features of the RGB image and the thermal image into the self-adaptive module, combine the environment label input by the human, and obtain the reliability of the thermal image, and then obtain the reliability of the RGB image according to the reliability of the thermal image;

[0012] Transmit the output of each layer of the encoder into the multi-modal feature fusion module based on the attention mechanism in sequence to obtain two feature flow results of the RGB image and the thermal image;

[0013] Transmit the two feature flow results of the RGB image and the thermal image into the decoder module for feature fusion to obtain the scene segmentation result.

[0014] As a further technical solution, input the RGB image and the thermal image collected in the dark and weak environment into the corresponding encoder respectively, and the processing steps are as follows:

[0015] Image preprocessing: use Wiener filtering to process the input image, and process the image into a uniform size at the same time;

[0016] Feature extraction: input the RGB image or the thermal image into the network architecture to extract features and obtain multi-layer feature output;

[0017] Convolve each layer of feature output result respectively to adjust the channel number of each layer of feature, and obtain the encoding output result.

[0018] As a further technical solution, the reliability of the thermal image can be obtained according to the self-adaptive module, so as to obtain the reliability of the RGB image, and specifically:

[0019] The reliability of the thermal image is α, and the reliability of the RGB image is 1-α.

[0020] As a further technical solution, it also includes constructing a Ghost module:

[0021] Use convolution operation on the input RGB feature map and thermal feature map to obtain intrinsic feature map;

[0022] Perform linear transformation operation on each feature map in the intrinsic feature map to generate Ghost feature map;

[0023] Splice the intrinsic feature map and the Ghost feature map and output.

[0024] As a further technical solution, it also includes constructing a cross-modal attention complementary sub-module:

[0025] The RGB feature map input is projected using convolution to obtain a first projection result;

[0026] The thermal feature map input is projected using convolution to obtain a second projection result; the second projection result is transposed to obtain a third projection result;

[0027] The second projection result and the third projection result are multiplied using matrix multiplication, and then processed using a function to obtain a fourth projection result;

[0028] The fourth projection result and the first projection result are multiplied using matrix multiplication to obtain the spatial weight of the thermal map;

[0029] Based on the thermal map reliability, the spatial weight of the thermal map and the RGB feature map input are added using matrix addition to obtain the improved RGB feature map.

[0030] In a second aspect, an adaptive multi-modal scene segmentation system based on an attention mechanism in a dark and weak environment is disclosed, comprising:

[0031] An encoder module configured to process and output RGB image features and deep thermal map features respectively for an RGB image and a thermal map collected in a dark and weak environment;

[0032] An adaptive module configured to map the RGB image features and the deep thermal map features to a low-dimensional space and perform corresponding operations to obtain an estimated value reflecting the reliability of the thermal map, and combine the artificially input environment label to obtain the reliability of the thermal map, and thus obtain the reliability of the RGB image;

[0033] A multi-modal feature fusion module based on an attention mechanism configured to sequentially process each layer output of the encoder to obtain two feature stream results of the RGB image and the thermal map;

[0034] A decoder module configured to perform feature fusion on the two feature stream results of the RGB image and the thermal map to obtain a scene segmentation result.

[0035] As a further technical solution, the encoder module has the following processing steps:

[0036] Image preprocessing: using Wiener filtering to process the input image, and processing the image to a uniform size;

[0037] Feature extraction: inputting the RGB image or the thermal map into a network architecture to extract features and obtain multi-layer feature outputs;

[0038] Each layer of feature output result is convolved respectively to adjust the channel number of each layer of feature, and an encoding output result is obtained.

[0039] As a further technical solution, the confidence of the RGB image is obtained according to the estimated value of the confidence of the heat map obtained by the adaptive module, specifically:

[0040] The confidence of the heat map is alpha, and the confidence of the RGB image is 1-alpha.

[0041] The above one or more technical solutions have the following beneficial effects:

[0042] The technical solution of the present application is based on the concept of multi-modal feature fusion based on attention mechanism, and the confidence of the heat map is adaptively adjusted to realize scene segmentation in dark and weak environments. The advantage of this method is that it can effectively deal with complex image scenes, and good results can be obtained for different environmental conditions. In addition, this method can also be used for various image scenes, such as road traffic scenes, indoor and outdoor scenes, urban street scenes, etc.

[0043] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0044] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation of the present application.

[0045] Figure 1 The structure schematic diagram of the RGB encoder of the embodiment of the present application is shown in the figure;

[0046] Figure 2 The structure schematic diagram of the adaptive module of the embodiment of the present application is shown in the figure;

[0047] Figure 3 The structure schematic diagram of the Ghost module of the embodiment of the present application is shown in the figure;

[0048] Figure 4 The structure schematic diagram of the cross-modal attention complementary sub-module of the embodiment of the present application is shown in the figure;

[0049] Figure 5 The structure schematic diagram of the multi-modal feature fusion module based on attention mechanism of the embodiment of the present application is shown in the figure;

[0050] Figure 6 The structure schematic diagram of the multi-modal scene segmentation model based on the encoder-decoder architecture of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0052] It is to be noted that the terms used herein are merely for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application.

[0053] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0054] In a dark and weak environment, the quality of the RGB image is often low, so it is difficult for the traditional scene segmentation method to achieve good results. Therefore, by using the feature fusion of the RGB image and the thermal image information, a scene segmentation method suitable for dark and weak environments is proposed, so as to realize effective scene segmentation.

[0055] Embodiment one

[0056] The embodiment discloses an adaptive multi-modal scene segmentation method based on an attention mechanism in a dark and weak environment, comprising:

[0057] Step one: construct an encoder module, and input the RGB image and the thermal image into the encoder as input respectively. This step takes the RGB encoder as an example, as shown in Figure 1 .

[0058] (1) pre-process the image, filter the RGB image using the Wiener filter to solve the image blur problem, and unify the RGB image size to 640*480;

[0059] (2) input the RGB image into the ResNet-152 architecture to extract features, and obtain five layer outputs;

[0060] (3) respectively convolve each layer output result to adjust the channel number of each layer feature, and obtain five output results of RGB 1 , RGB 2 , RGB 3 , RGB 4 , and RGB 5 .

[0061] The Thermal encoder taking the thermal image as input is constructed as above, specifically:

[0062] (1) pre-process the image, filter the thermal image using the Wiener filter to solve the image blur problem, and unify the thermal image size to 640*480;

[0063] (2) input the thermal image into the ResNet-152 architecture to extract features, and obtain five layer outputs;

[0064] (3) respectively convolve each layer output result to adjust the channel number of each layer feature, and obtain five output results;

[0065] Regarding the encoder in this embodiment, first, filter is used to reduce noise pollution in dark and weak environment for RGB image. The encoder uses ResNet-152 as the main framework. The full connection layer and the average pool layer of the original framework are removed in the process of extracting features, because they may cause the loss of spatial information and details. At the same time, the RGB encoder and the thermal encoder share the parameters of each branch, which not only makes the extracted features remain in the same feature space, but also reduces the parameter amount. In addition, the channel size of each layer of features is modified using the convolution layer, which reduces the calculation cost with fewer channel numbers.

[0066] Step two: construct an adaptive module (AM), as shown in Figure 2 .

[0067] (1) The fifth layer output of the RGB encoder and the thermal encoder is taken as the input of the module;

[0068] (2) First, the deep features of the RGB image and the thermal image are mapped to a low-dimensional space through two average pooling layers and a full connection layer;

[0069] (3) Through batch normalization and ReLU operation, the estimated value can be obtained, which reflects the reliability of the thermal image;

[0070] (4) The module can selectively intervene in feature fusion artificially, and the artificial input environment label w can be given as the correction reference;

[0071] (5) Through the idea of F-meature, the weighted harmonic mean of and w is calculated, and finally the reliability a of the thermal image is obtained. Relatively, the reliability of the RGB image is 1-a.

[0072] Regarding the adaptive module, the average pooling layer and the full connection layer in the module map the deep features to a low-dimensional space, which significantly improves the generalization ability of the result. Through the intervention of the optional artificial input environment label, the reliability of the thermal image can be obtained. In different dark and weak environments, artificial intervention can be adjusted according to the severity of the environment, and the reliability of the thermal image is affected through weighted harmonic mean. It is introduced in the subsequent multi-modal feature fusion module based on attention mechanism to prevent pollution from unreliable thermal images in the fusion process.

[0073] Step three: construct a ghost module, as shown in Figure 3 .

[0074] (1) First, the input feature map is used to obtain the intrinsic feature map through the primary convolution (Primary Conv);

[0075] (2) Secondly, Cheap Operation operation is performed on each feature map in the result of (1) to generate a Ghost feature map;

[0076] (3) Finally, the intrinsic feature map generated in (1) and the Ghost feature map generated in (2) are spliced to obtain the final result Output.

[0077] Regarding the Ghost module, this module comes from the Huawei Ghost network series. In the traditional feature extraction process, it is found that there are some highly similar feature maps. This module does not deliberately avoid generating these redundant feature maps, but generates similar redundant feature maps-Ghost pairs with the help of linear transformation, splices the feature maps generated by the regular convolution and linear transformation operation to obtain the final output, so as to reduce the feature redundancy.

[0078] Step four: construct a cross-modal attention complementary sub-module, such as Figure 4 . Take the example of using thermal feature map to perfect the RGB feature map (Thermal-to-RGB Attention Module, T-RAM), the input is the output feature map F r and F t generated by the RGB feature and the thermal feature respectively after the Ghost module. As shown in Figure 4 (a).

[0079] (1) use 1x1 convolution to project the RGB feature map F r with channel number C, height H and width W into F r ', and the size of F r ' is CxHW;

[0080] (2) use 1x1 convolution to project the thermal feature map F t with channel number C, height H and width W into , and use transpose operation on to obtain , and the size of is the size of

[0081] (3) use matrix multiplication to multiply and , and use Softmax function to obtain F' t , and the size of F' t is HWxHW;

[0082] (4) use matrix multiplication to multiply F' t and F' rF = F * F tr , and the final result is F r + a · F tr ;

[0083] Similarly, the RGB-to-Thermal Attention Module (R-TAM) is symmetrical to the T-R AM structure, as shown in (b) of FIG. 2, and the specific steps are as follows: Figure 4

[0084] (1) The thermal feature map F t with a channel number of C, a height of H, and a width of W is projected to F′ t using a 1x1 convolution, and the size of F′ t is CxHW

[0085] (2) The thermal feature map F r with a channel number of C, a height of H, and a width of W is projected to using a 1x1 convolution, and the size of is CxHW The size of is CxHW

[0086] (3) After multiplying and using matrix multiplication, F r ′ is obtained using the Softmax function, and the size of F r ′ is HWxHW;

[0087] (4) F r ′ and F t ′ are multiplied using matrix multiplication to obtain the spatial weight F rt of the thermal map, and the final result is (1-a)·F rt +F t ;

[0088] ​​As for the cross-modal attention complementary sub-module, take T-R AM as an example, it uses the thermal feature to generate a spatial weight for the RGB feature, because the thermal feature often provides useful information for the RGB branch in the dark and weak environment. Among them, the multiplication of the thermal feature map and its transpose can be used to calculate the correlation between the features, which helps to identify different regions or objects in the image. The thermal map generated by the adaptive module is used to control the interaction between the RGB feature and the thermal feature. In very poor dark and weak environment, a = 1, at this time the thermal map is highly reliable, more thermal features will be introduced into the RGB branch; when a = 0, the RGB branch will become the dominant branch, and only a few thermal features will be used.

[0089] Step five: construct a multi-modal feature fusion module based on attention mechanism (Muti-Modelity Attention Module, MAM), use R-T attention to realize attention mechanism, as shown in Figure 5

[0090] (1) the RGB feature map and the thermal feature map generated by the encoder module are respectively transmitted into the Ghost module in step three to obtain F r and F t ;

[0091] (2) F r and F t are transmitted into T-R AM (step four) to obtain the RGB feature map after using the thermal map to perfect it;

[0092] (3) F r and F t are transmitted into R-T AM to obtain the thermal feature map after using the RGB map to perfect it;

[0093] (4) the results of (3) and (4) are output respectively.

[0094] The multi-modal feature fusion module based on attention mechanism is used to capture different features in the RGB map and the thermal map to enhance the image features that are difficult to capture in the dark and weak environment. This module uses self-attention features to learn the complementarity between cross-modal features from the perspective of cross-modal, which can prevent unreliable information pollution in the dark and weak environment.

[0095] In the dark and weak environment, the thermal map is often reliable, but the RGB image is easy to encounter interference from similar background, and usually cannot focus well on the area of prominent objects. At this time, the features of the depth map can provide complementary information to suppress background noise to enhance the feature map. Similarly, when the thermal map tends to be unreliable, with the help of the adaptive module, the thermal information at this time has little effect on the feature map. In summary, this module can learn the complementarity of the RGB map and the thermal map from the perspective of cross-modal, and can also produce very good results in the dark and weak environment.​

[0096] Step six: construct a multi-modal scene segmentation model based on an encoder-decoder architecture, as shown in Figure 6 :

[0097] (1) input the RGB image and the thermal image into the encoder module of step one, respectively;

[0098] (2) input the output of the last layer of the encoder into the adaptive module of step two to obtain the confidence of the RGB image and the thermal image, respectively;

[0099] (3) sequentially input the output of each layer of the encoder module into the module of step five to obtain two feature stream results of the RGB image and the thermal image;

[0100] (4) input the results of (3) into the decoder module for feature fusion to obtain the final scene segmentation result, wherein the decoder module:

[0101] 1) uses two branch decoders to serially fuse the multi-scale features of the RGB feature stream and the thermal image feature stream, respectively.

[0102] 2) finally generates the final result by performing multi-modal feature fusion on the outputs of the two decoders.

[0103] In view of the defects of the traditional RGB image-based method, the present embodiment combines the thermal image from the thermal sensor and the traditional RGB image, and uses the multi-modal feature fusion module based on the attention mechanism and the adaptive module to realize scene segmentation in a dark and weak environment.

[0104] The present application proposes multiple modules in the feature fusion stage, effectively integrating the complementarity of multi-modal information. In the decoder stage, the double-stream decoding can effectively avoid the information pollution between the RGB information and the thermal information.

[0105] For example, training on the open-source MFNet dataset:

[0106] (1) training process:

[0107] 1) pre-process the data in the MFNet dataset, randomly adjust the brightness of the image or perform blur processing to simulate the RGB image in a dark and weak environment. At the same time, according to the degree of pre-processing, an initial artificial environment label is given to each image;

[0108] 2) select the RGB image to input the image RGB image and the thermal image into the encoder, respectively, to obtain five scale feature maps;

[0109] 3) input the fifth layer feature map into the adaptive module to obtain the confidence of the thermal image.

[0110] 4) The results of (1) are respectively transmitted into the multi-modal feature fusion module based on attention mechanism. Through feature fusion, the feature flow results of the RGB image and the heat map can be obtained respectively;

[0111] 5) The results of (3) are transmitted into the decoder to obtain the final output. The network is trained on the MFNet training set.

[0112] (2) Test process:

[0113] Select some pictures in the test set, randomly adjust the brightness or use the blur operation to simulate the RGB image in the dark and weak environment, and manually label the environment label according to the severity of the environment. The manually labeled environment label, the adjusted RGB image and the heat map are transmitted into the trained model to obtain the final prediction result.

[0114] Taking an actual application scenario as an example:

[0115] (1) Urban street monitoring

[0116] This method can be applied to urban street monitoring to segment different targets such as roads, buildings, trees, etc. in the city in dark and weak environments. Through real-time monitoring of urban street scenes, the urban environment quality can be better evaluated, and effective suggestions can be made for urban environmental management.

[0117] (2) Security monitoring

[0118] This method can be applied to security monitoring to segment different targets such as people, vehicles, animals, etc. in dark and weak environments. Through real-time monitoring of safe environments, dangerous situations can be better identified to protect personal safety and property safety.

[0119] (3) Autonomous driving

[0120] This method can also be applied to autonomous driving to segment different targets such as roads, vehicles, pedestrians, etc. in dark and weak environments. Through real-time monitoring of road traffic scenes, autonomous driving can be better achieved to ensure driving safety.

[0121] Embodiment Two

[0122] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the steps of the above method.

[0123] Embodiment Three

[0124] The purpose of this embodiment is to provide a computer-readable storage medium.

[0125] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above method.

[0126] Embodiment four

[0127] The purpose of the embodiment is to provide an attention mechanism-based adaptive multi-modal scene segmentation system in a dark and weak environment, comprising:

[0128] An encoder module configured to process and output RGB image features and deep features of a thermal image respectively for an RGB image and a thermal image collected in a dark and weak environment;

[0129] An adaptive module configured to map the RGB image features and the deep features of the thermal image into a low-dimensional space and perform corresponding operations to obtain an estimated value reflecting the reliability of the thermal image, and combine the artificially input environment label to obtain the reliability of the thermal image, thereby obtaining the reliability of the RGB image;

[0130] An attention mechanism-based multi-modal feature fusion module configured to sequentially process the output of each layer of the encoder to obtain two feature stream results of the RGB image and the thermal image;

[0131] A decoder module configured to perform feature fusion on the two feature stream results of the RGB image and the thermal image to obtain a scene segmentation result.

[0132] The steps and methods involved in the above embodiments two, three and four correspond to embodiment one, and the specific implementation can refer to the relevant description part of embodiment one. The term "computer readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any method in the present application.

[0133] Those skilled in the art should understand that each module or step of the above application can be realized by a general computer device, alternatively, they can be realized by program code executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0134] The above describes the specific embodiments of the present application in combination with the drawings, but is not a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. An adaptive multi-modal scene segmentation method based on attention mechanism in dark and weak environments, characterized in that, The method comprises the steps of: inputting the RGB image and the thermal image collected in the dark and weak environment into corresponding encoders respectively to obtain an RGB feature map and a thermal feature map, wherein the RGB feature map comprises deep features of the RGB image, and the thermal feature map comprises deep features of the thermal image; inputting the deep features of the RGB image and the thermal image into an adaptive module to obtain an estimated value of the reliability of the thermal image, and obtaining the reliability of the RGB image according to the estimated value of the reliability of the thermal image; sequentially inputting each layer output of the encoder into a multi-modal feature fusion module based on an attention mechanism to obtain two feature flow results of the RGB image and the thermal image; the multi-modal feature fusion module comprises a Ghost module and a cross-modal attention complementary sub-module; inputting the two feature flow results of the RGB image and the thermal image into a decoder module to perform feature fusion and obtain a scene segmentation result; the cross-modal attention complementary sub-module is constructed as follows: inputting the RGB feature map using convolution and performing projection to obtain a first projection result; inputting the thermal feature map using convolution and performing projection to obtain a second projection result, and performing transposition on the second projection result to obtain a third projection result; multiplying the second projection result and the third projection result using matrix multiplication and then processing using a function to obtain a fourth projection result; multiplying the fourth projection result and the first projection result using matrix multiplication to obtain a spatial weight of the thermal image, and obtaining a perfected RGB feature map based on the reliability of the thermal image.

2. The adaptive multi-modal scene segmentation method based on attention mechanism in dark and weak environment according to claim 1, characterized in that, The method comprises the steps of: image preprocessing: performing processing on the input image using Wiener filtering, and processing the image into a uniform size; feature extraction: inputting the RGB image or the thermal image into a network architecture to extract features, and obtaining multi-layer feature output; performing convolution on each layer feature output result to adjust the channel number of each layer feature, and obtaining an encoding output result.

3. The method of claim 1, wherein the method further comprises: determining a first attention map of the first modality based on the first feature map and the first attention map; determining a second attention map of the second modality based on the second feature map and the first attention map; and determining a third attention map based on the first attention map and the second attention map. The reliability of the RGB image is obtained according to the reliability of the thermal image obtained by the adaptive module, and specifically: The reliability of the heat map is , and the reliability of the RGB map is .

4. The method of claim 1, wherein the method further comprises: the Ghost module is constructed as follows: performing convolution on the input feature map to obtain an intrinsic feature map; performing operation on each feature map in the intrinsic feature map to generate a Ghost feature map; splicing the intrinsic feature map and the Ghost feature map to obtain a final feature map.

5. An adaptive multi-modal scene segmentation system based on attention mechanism in dark and weak environments, characterized in that, The method comprises the steps of: an encoder module configured to process an RGB image and a thermal image collected in a dark and weak environment respectively and output an RGB feature map and a thermal feature map, wherein the RGB feature map comprises deep features of the RGB image, and the thermal feature map comprises deep features of the thermal image; an adaptive module configured to map the deep features of the RGB image and the thermal image into a low-dimensional space and perform corresponding operations to obtain an estimated value reflecting the reliability of the thermal image, and obtain the reliability of the thermal image in combination with an artificially input environment label, thereby obtaining the reliability of the RGB image; a multi-modal feature fusion module based on an attention mechanism configured to sequentially process each layer output of the encoder to obtain two feature flow results of the RGB image and the thermal image; the multi-modal feature fusion module comprises a Ghost module and a cross-modal attention complementary sub-module; a decoder module configured to perform feature fusion on the two feature flow results of the RGB image and the thermal image to obtain a scene segmentation result. Also comprising constructing a cross-modal attention complementary sub-module: Using convolution to input and project the RGB feature map, obtain the first projection result; Using convolution to input and project the thermal feature map, obtain the second projection result, and obtain the third projection result by transposing the second projection result; Using matrix multiplication to multiply the second projection result and the third projection result, and using a function to obtain the fourth projection result; Using matrix multiplication to multiply the fourth projection result and the first projection result to obtain the spatial weight of the thermal map, and based on the thermal map reliability, obtain the improved RGB feature map. 6.The dark and weak environment based attention mechanism adaptive multi-modal scene segmentation system of claim 5, wherein, The processing steps of the encoder module are: Image preprocessing: using Wiener filtering to process the input image, and processing the image to a uniform size; Feature extraction: input the RGB image or thermal image into the network architecture to extract features and obtain multi-layer feature output; Convolve each layer of feature output respectively to adjust the channel number of each layer of feature to obtain the encoding output result.

7. The attention mechanism based adaptive multi-modal scene segmentation system in dark and weak environments as claimed in claim 5, wherein, The reliability of the thermal map can be obtained using the adaptive module, so as to obtain the reliability of the RGB image, specifically: The reliability of the heat map is , and the reliability of the RGB map is .

8. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory and loadable on the processor, characterized in that, The processor executes the program to realize the steps of the method of any one of claims 1-4.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to execute the steps of the method of any one of claims 1-4.

Images