Multimodal image semantic segmentation method and system based on cross-level guided fusion

Abstract

The present invention discloses a multimodal image semantic segmentation method and system based on cross-level guided fusion, which utilizes multimodal data of RGB image information and additional modal image information such as depth information, thermal imaging information and spectral information, fuses multimodal information and uses it for image semantic segmentation tasks. The method first converts the additional modal image into the same size and visual range as the RGB image, and then uses a dual-branch encoder based on the Transformer basic block to extract RGB modal and additional modal features, and uses a cross-level guided fusion module to guide the multimodal fusion of high-level features in shallow, middle and high-level features using low-level features, and finally uses a decoder to decode the multi-level fusion features, and outputs the image category pixel by pixel, which solves the problem of poor semantic segmentation performance in complex scenes; the technical solution of the present invention can be applied to complex scenes such as robot intelligent sorting and drone intelligent inspection.

Description

Technical Field

[0001] The present invention belongs to machine vision multimodal processing technology, and specifically, relates to a multimodal image semantic segmentation method and system based on cross-level guided fusion. Background Art

[0002] The ability to accurately understand complex scenes is critical to enabling advanced automated applications. Image semantic segmentation, as a core task in computer vision, aims to assign each pixel in an image to a predefined category, thereby achieving a deep understanding of the scene. However, due to the diversity and complexity of the natural environment, it is often difficult to achieve satisfactory results by relying solely on traditional single modality (such as RGB images) for semantic segmentation. To address this challenge, researchers have begun to explore the use of additional modal information to enhance the performance of semantic segmentation. In particular, in complex scenes, such as in environments with severe weather conditions, severe occlusion, or extreme changes in lighting, the limitations of RGB images when used alone become particularly apparent. Therefore, exploring multi-modal semantic segmentation based on RGB-D (visible light + depth images) and RGB-T (visible light + infrared thermal images) provides new possibilities for improving accuracy and robustness.

[0003] Although RGB-D and RGB-T image semantic segmentation techniques have proven their effectiveness in many scenarios, they show obvious limitations when faced with complex natural environments. This is mainly because existing multimodal fusion methods usually adopt simple hierarchical fusion strategies and fail to fully explore the complementary information between different modalities. In particular, in deep network structures, simple fusion strategies cannot effectively integrate multi-level features with different semantic richness. In addition, problems such as extreme lighting changes, occlusions, and dramatic changes in scene depth in complex scenes further limit the performance of these methods, mainly because they fail to effectively identify and utilize the deep correlation and complementarity between modalities. For example, in low-light or over-exposed environments, RGB image information is severely lost, and existing methods fail to fully utilize depth or thermal imaging information to compensate. Summary of the invention

[0004] In view of the shortcomings of the above-mentioned prior art in complex natural environments, the present invention proposes a multimodal image semantic segmentation method and system based on cross-level guided fusion, aiming to fully explore the multi-level consistency and complementarity of RGB modality and additional modality (depth or infrared thermal image) through cross-level guided fusion, so as to solve the problem of poor semantic segmentation performance of multimodal images such as RGB-D and RGB-T in complex scenes. The core improvement of the technical solution of the present invention is that on the basis of fusion at the same level, cross-level fusion is proposed, and the underlying feature details are used to guide the integration of high-level semantic features. At the same time, object edge information is introduced, which optimizes the model's ability to handle complex scene details and improves the model's semantic segmentation model accuracy.

[0005] To this end, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a multimodal image semantic segmentation method based on cross-level guided fusion, comprising the following steps:

[0007] S1: preprocessing the RGB image of the target scene and the additional modality image to align the spatial information of the RGB and additional modality images;

[0008] S2: Pseudo-color coding of preprocessed additional modality images;

[0009] S3: construct a dual-branch encoder based on the Transformer basic block and a cross-level fusion module, and use the dual-branch encoder to extract RGB modality features and additional modality features, and use the fusion module to perform feature fusion to obtain fusion features at each level;

[0010] The dual-branch encoder includes two branch encoders, which correspond to the preprocessed RGB image and the additional modality image after pseudo-color encoding, respectively. The branch encoders and the fusion module are both multi-level structures. The RGB modality features and the additional modality features extracted from the same encoding layer of the two branch encoders are fused through the fusion module, and the fused features of the previous level are input into the next fusion layer.

[0011] S4: constructing a decoder, wherein the decoder performs multi-level fusion feature decoding based on the fusion features of each level to obtain a semantic segmentation result;

[0012] Among them, the RGB image, the extra-modal image and the semantic segmentation mark of the target scene are used as sample data, and the RGB image and the extra-modal image are processed according to the process of steps S1-S2; then the dual-branch encoder, the cross-level fusion module and the decoder constructed according to steps S3-S4 are trained to obtain the semantic segmentation network for image semantic segmentation.

[0013] The multimodal image semantic segmentation method provided by the technical solution of the present invention fully exploits the multi-level consistency and complementarity of the RGB modality and the additional modality (depth or infrared thermal image) through the constructed dual-branch encoder, cross-level fusion module and decoder, and solves the problem of poor semantic segmentation performance of multimodal images such as RGB-D and RGB-T in complex scenes. It should be understood that after using the above-constructed dual-branch encoder, cross-level fusion module and decoder and model training, the model can meet the image semantic segmentation requirements of specific application fields.

[0014] Further optionally, each branch encoder is respectively provided with M encoding layers connected in sequence, and the fusion module is provided with M fusion layers connected in sequence;

[0015] Among them, the output of the same encoding layer of the two branch encoders is used as the input data of the fusion layer of the corresponding layer, and the output of the previous fusion layer and the object edge information are also used as the input data of the next fusion layer. M is a positive integer.

[0016] Further optionally, the fusion layer of the fusion module adopts cross-level guided multimodal fusion, and the fusion process is as follows:

[0017] First, the linear representation is calculated using the RGB modality features and additional modality features extracted by the encoder of the same layer:

[0018] , , ,

[0019] , , ,

[0020] In the formula, , , Represents RGB modality features The corresponding query, key and value, , , Represents additional modal features Corresponding query, key and value, W is the weight matrix, , , , , , Corresponding to , , , , , The weight matrix of , n represents the level;

[0021] Then, the two modalities are fused by calculating the attention to obtain the modal fusion feature , , and then splice to get the splicing features :

[0022] ,

[0023] ,

[0024] ,

[0025] In the formula, , They correspond to the fusion features of RGB modality features and additional modality features respectively, and T is the matrix transposition symbol. Indicates the number of channels; Represents a splicing operation, Indicates converting a set of real values ​​into a probability distribution, where the sum of the probabilities of all parts is 1; if level n is not the first level, the concatenated features are fused with the lower-level features by a multi-layer perceptron MLP. Perform matrix multiplication to obtain cross-level guided splicing features :

[0026] ,

[0027] The function is defined as:

[0028] Let A and B be two matrices, the dimension of A is , the dimension of B is ,but:

[0029] ,

[0030] In the formula, , are the element values ​​of matrices A and B.

[0031] Further optionally, if each fusion layer also fuses object edge information, the method further includes:

[0032] Extract object edge information from RGB images;

[0033] If the fusion layer is not the first layer, the object edge information and the cross-level guided splicing features are combined. The cross-level guided fusion edge features are obtained by matrix multiplication as the fusion features of the corresponding level;

[0034] ,

[0035] In the formula, Guide the fusion of edge features across levels for the nth layer;

[0036] If the fusion layer is the first layer, the edge information of the object and the splicing features The cross-level guided fusion edge features are obtained by matrix multiplication as the fusion features of the corresponding level.

[0037] Further optionally, the process of the decoder decoding the multi-level fusion features based on the fusion features of each level to obtain the semantic segmentation result is:

[0038] First, the fusion features of the last level are upsampled by the multi-layer perceptron MLP to obtain the feature map;

[0039] Then, the feature map is spliced ​​with the fusion features of the previous level, and then the multi-layer perceptron MLP upsampling is performed; wherein, this step is performed in sequence until the feature map is spliced ​​with the fusion features of the first level to obtain the feature map;

[0040] Finally, the number of output channels is set to the number of segmented categories C, and then the feature map is upsampled by a multi-layer perceptron MLP to obtain a size of One-hot encoding of is the original image size, where the position of each pixel corresponding to the unique hot encoding is the category number represented by the corresponding pixel.

[0041] Further optionally, the additional modality image is a depth image, an infrared image, or a spectral image.

[0042] Further optionally, the target scenario is robot intelligent sorting and drone intelligent inspection.

[0043] In robot intelligent sorting scenarios, there are overlapping objects, complex backgrounds, or uneven lighting conditions, and a single RGB image is difficult to accurately identify and locate objects. The technical solution of the present invention can use depth images as an additional modality to execute a multimodal image semantic segmentation algorithm, greatly enhancing the ability to resolve details in semantic segmentation, especially in fine operations and object segmentation, showing significant technical advantages and practical value, improving the robot's recognition rate of object position and shape, and achieving efficient intelligent sorting. The specific steps include: 1) using an RGB camera and a depth sensor to simultaneously capture visible light images and depth images of the target object; 2) aligning the captured RGB images and depth images to ensure the consistency of the spatial information of the two images; 3) using a dual-branch encoder to extract feature information of the RGB image and the depth image respectively; 4) using a cross-level guided fusion module to perform multi-level feature fusion of the extracted RGB and depth features, combining low-level details and high-level semantic information; 5) using a decoder to decode the fused features and output the semantic segmentation results of the image pixel by pixel; 6) based on the segmentation results, the robot accurately identifies and locates the shape and position of different objects; 7) based on the identification and positioning results, the robot performs grasping and sorting operations to achieve efficient intelligent sorting.

[0044] In the application of UAV intelligent inspection, RGB images can accurately identify ground objects and facility details, such as cracks and corrosion, with their rich color and texture features. However, they perform poorly in low-light or obstructed vision environments such as at night, and cannot detect the internal structure of power facilities. The technical solution of the present invention can use infrared images as an additional modality, accurately integrate complementary information from the two modalities, achieve efficient scene monitoring and facility health status assessment, and enhance application performance in complex environments. The specific steps include: 1) using the RGB camera and infrared camera carried by the drone to simultaneously collect RGB images and infrared images of the power lines; 2) aligning the collected RGB images and infrared images to ensure the consistency of the spatial information of the two images; 3) using a dual-branch encoder to extract the feature information of the RGB image and the infrared image respectively; 4) through the cross-level guided fusion module, the extracted RGB and infrared features are multi-level feature fused, combining low-level details and high-level semantic information; 5) using a decoder to decode the fused features and output the semantic segmentation results of the image pixel by pixel; 6) based on the segmentation results, the drone can accurately identify the details of the power lines and facilities; 7) combining the segmentation results with the temperature distribution information provided by the infrared image to detect the hot spots inside the equipment; 8) generating an inspection report to record the potential problems and abnormal conditions identified.

[0045] It should be understood that in actual applications, by training the network according to sample labels, semantic segmentation results corresponding to label classifications can be achieved.

[0046] In a second aspect, the present invention provides a system based on the above-mentioned multimodal image semantic segmentation method, comprising:

[0047] A preprocessing module is used to preprocess the RGB image of the target scene and the additional modality image to align the spatial information of the RGB and additional modality images;

[0048] A pseudo color coding processing module, used for performing pseudo color coding processing on the pre-processed additional modality image;

[0049] The encoding fusion module is used to construct a dual-branch encoder based on the Transformer basic block and a cross-level fusion module, and use the dual-branch encoder to extract RGB modality features and additional modality features, and use the fusion module to perform feature fusion to obtain fusion features at each level;

[0050] The dual-branch encoder includes two branch encoders, which correspond to the preprocessed RGB image and the additional modality image after pseudo-color encoding, respectively. The branch encoders and the fusion module are both multi-level structures. The RGB modality features and the additional modality features extracted from the same encoding layer of the two branch encoders are fused through the fusion module, and the fused features of the previous level are input into the next fusion layer.

[0051] The decoding module is used to construct a decoder, and the decoder performs multi-level fusion feature decoding based on the fusion features of each level to obtain a semantic segmentation result.

[0052] In a third aspect, the present invention provides an electronic device, comprising:

[0053] one or more processors;

[0054] a memory storing one or more computer programs;

[0055] The processor calls the computer program to execute:

[0056] The steps of the multimodal image semantic segmentation method.

[0057] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program programmed or configured to implement the multimodal image semantic segmentation method.

[0058] Beneficial Effects

[0059] Compared with the existing method, the advantages of the present invention are:

[0060] The multimodal image semantic segmentation method based on cross-level guided fusion provided by the technical solution of the present invention ensures that the RGB image is completely aligned with the additional modal image in space, and unifies all modal data into the same data format and size through pseudo-color coding processing, which facilitates the algorithm to process multimodal data and reduces the complexity of the preprocessing stage; it uses a dual-branch encoder and Transformer basic blocks to extract rich features from multiple levels, and combines self-attention to optimize the capture of local and global information, so that the model can understand and process image content more accurately; the cross-level guided fusion method is adopted, which not only improves the feature The flexibility and efficiency of fusion also ensure the effective integration of details and semantic information through the guidance of low-level features on the fusion of high-level features, and enhance the model's recognition ability for complex scenes; through the decoding of multi-level fusion features, combined with the multi-scale decoding strategy, the accuracy of pixel-by-pixel classification is significantly improved. The present invention preferably combines details such as edges to improve the fineness of semantic segmentation; the technical solution of the present invention is not only technologically innovative, but also has the potential to be applied to various fields due to its efficient and accurate feature fusion and scene understanding capabilities, such as autonomous driving, drone intelligent inspection, robot intelligent sorting and other scenarios, showing broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of a complex scene multimodal image semantic segmentation method based on cross-level guided fusion of the present invention;

[0062] Figure 2 A diagram showing the structure of a multimodal image fusion network for cross-level guided fusion according to the present invention;

[0063] Figure 3 This is a diagram of the specific implementation process of the fusion layer of the present invention;

[0064] Figure 4 This is a diagram of the layer-by-layer decoding process of the present invention. DETAILED DESCRIPTION

[0065] The present invention will be further described below with reference to the embodiments.

[0066] Embodiment 1:

[0067] The embodiment of the present invention provides a multimodal image semantic segmentation method based on cross-level guided fusion, comprising the following steps:

[0068] S1: Preprocessing: preprocessing the acquired RGB image and the additional modality image to align the spatial information of the RGB and additional modality images.

[0069] Keep the center of the two images consistent, and crop and scale the high-resolution RGB image to the same image size as the additional modality image. Specifically, the SIFT algorithm is used to detect N1 feature points in the two images and generate corresponding descriptors; then a set of high-quality matching pairs is obtained through matching and screening, each of which consists of the coordinates of feature points in the two images; finally, a high-quality scaling method such as bicubic interpolation is used to minimize information loss during the scaling process and ensure alignment with the size of the additional modality image. Since this process is achievable with existing technology, it will not be elaborated in detail, but simply stated as follows:

[0070] This embodiment calculates and selects three pairs (or more pairs) of high-quality matching feature points from a large number of feature points, such as and , and as well as and , x, y represent the image coordinates of the feature points. After obtaining three pairs of matching feature points, this embodiment constructs a 2×3 affine transformation matrix A1, which can be expressed as:

[0071]

[0072] For each point in the image , the transformed coordinates can be calculated as:

[0073] ,

[0074] ,

[0075] Where a, b, c are the coefficients controlling translation and linear transformation, and d, e, f are the coefficients controlling translation and linear transformation. No matter how many pairs of matching feature points there are, the affine transformation matrix A is in the form of 2×3 and contains six parameters.

[0076] In this embodiment, the three pairs of matching feature points are substituted to solve the value of matrix A1 (for example, using the minimum multiplication method to find an approximate solution), and then for each pixel position in the RGB image , using the above affine transformation formula to calculate its new position in the extra modality image , because the interpolated coordinates may not fall at integer locations in the extra modality image, bicubic interpolation is used to calculate the values ​​of the new pixel locations to obtain an image that is aligned with the dimensions of the extra modality image.

[0077] S2: Perform pseudo color encoding on the additional modality image so that the data format and size of the additional modality image match those of the RGB image.

[0078] Existing feature extraction modules are often designed for three-channel RGB images, so it is necessary to ensure that the extra modal images have the same data format and size as the RGB images. For example, infrared thermal images and depth images are both single-channel, and they are both three-channel images after encoding. In order to better extract features from extra modal images and visually observe the spatial information of extra modal images, the technical solution of the present invention performs pseudo-color encoding on extra modal images. The process is:

[0079] First, the single-channel value g of the additional modality image is normalized. The single-channel value g represents the depth value in the depth image and the temperature value in the thermal imaging image. The normalization formula is:

[0080] ,

[0081] in, and is the single channel value in the extra modality image The minimum and maximum values ​​of .

[0082] For simplicity, this embodiment directly converts the normalized single channel value Map to the same color, select the starting color and end color , , , Corresponding to the starting colors of the red, green, and blue channels respectively, , , Corresponding to the end colors of the red, green, and blue channels respectively. For the sake of intuition, the technical solution of the present invention sets the starting color to 0 and the ending color to 255, indicating the movement from light color to dark color. For each color channel, the linear interpolation formula is defined as follows:

[0083] ,

[0084] ,

[0085]

[0086] Finally, the single channel of the additional modality image is mapped to three channels, which is consistent with RGB, and the data size is .

[0087] In other feasible embodiments, the settings of the starting color and the ending color can be adaptively adjusted according to application requirements and accuracy.

[0088] S3: Construct a dual-branch encoder based on the Transformer basic block and a cross-level fusion module, use the dual-branch encoder to extract RGB modal features and additional modal features, and use the fusion module to perform feature fusion to obtain fusion features at each level.

[0089] The dual-branch encoder provided by the present invention includes two branch encoders, corresponding to RGB images and additional modal images respectively, and each branch encoder has an M-layer structure, that is, M sequentially connected encoding layers are respectively provided, and the output of the same encoding layer of the two branch encoders is used as a type of input data of the fusion layer of the corresponding layer. The cross-level fusion module includes M sequentially connected fusion layers, and the output of the previous fusion layer and the object edge information are used as another type of input data of the fusion layer, that is, the input data of the non-first fusion layer includes: the RGB modal features and additional modal features extracted by the two branch encoders corresponding to the same layer, the output of the previous fusion layer, and the object edge information.

[0090] In this embodiment, M is set to 4. Figure 2 As shown in FIG. 1 , the aligned RGB images and the encoded extra modality images in the first two steps are respectively input to four encoding layers for feature extraction. Each encoding layer in this embodiment uses MixTransformer as a basic block and uses downsampling and reshaping operations. and Respectively represent the RGB image and additional modality features extracted at the nth encoding level. In this embodiment, n=1, 2, 3, 4. Assume that the input RGB and additional modality image sizes are , H and W represent the height and width of the input image respectively. Then and The size is , Indicates the number of channels of the feature map, and its value can be set according to the size of the network. This embodiment provides a set of arrive They are .

[0091] After using the coding layer to extract the four layers of features of the image and the extra modality image, the fusion module of this embodiment uses four fusion layers to fuse the RGB features and extra modality image features of each layer respectively. Each fusion layer accepts the RGB features and extra modality image features of the corresponding layer as input, and also extracts the object edge information from the RGB image as auxiliary information through the existing Canny edge detector. In addition to fusion layer 1, fusion layer n also receives information from the previous layer n-1 to guide the fusion of RGB and extra modality images at this level.

[0092] In summary, the dual-branch encoder architecture constructed by the present invention uses the basic coding layer, and novelly connects the object edge information and the fusion feature information of this layer to the next layer of fusion for guidance. Through this framework, in addition to the traditional fusion of the features of the two modalities at this level, the information of other levels is also used to enable the fusion at this level to select the most useful features. At this point, the dual-branch encoder can efficiently and accurately fuse RGB and additional modality features.

[0093] Regarding the cross-level guided multimodal fusion of the fusion layer, the specific fusion process is as follows: Figure 3 As shown, the RGB features and the additional modality features are represented as and First, the linear representation is calculated using RGB features and additional modality features Q, K, V :

[0094] , , ,

[0095] , , ,

[0096] As shown in the above formula, Q, K, V Represents the query, key and value of the feature respectively, both represented by and Multiply it by the corresponding weight matrix. , , Represents RGB modality features The corresponding query, key and value, , , Represents additional modal features The corresponding query, key and value are For example, to generate The weight of . And here we get Q, K, V The dimensions are , Indicates the number of channels, and then the fusion feature is obtained by calculating the attention between the two modalities. , :

[0097] ,

[0098] ,

[0099] After splicing, we get the splicing features :

[0100] ,

[0101] The size of the stitching feature is , if n>1, it is necessary to fuse features with the lower level Matrix multiplication is performed through the MLP (multi-layer perceptron) layer to obtain cross-level guided splicing features :

[0102] ,

[0103] Among them, the low-level fusion features It comes from the fusion layer of the previous layer. For example, now it is fused in the third layer, that is, fusion layer 3, that is, n=3, then the low-level fusion feature here refers to the output of the previous fusion layer 2, that is, .

[0104] Low-level fusion features often have more detailed features than this level. When fused at this level, there may be a problem of loss of details in some locations in complex scenes. For example, when visible light is partially blocked, the fusion features of the previous level compensate for the lack of visible light through infrared thermal imaging. However, when fused at this level, there may be a problem of infrared details being lost with downsampling, making it impossible to achieve accurate complementarity. Therefore, the fusion information of the low level can guide the current level to perform more accurate fusion.

[0105] After achieving cross-level guidance, edge information is then integrated. The introduction of edge information is to solve the problem of unclear target contours in complex scenes. It is difficult to directly extract the clear contours of targets in complex scenes by directly fusing features. Therefore, edge information is used as prior knowledge to provide high-level guidance for multimodal features during cross-level fusion. Edge information is obtained by edge detection of the input RGB image and is expressed as , whose size is , extracted by MLP and pooled into the same size as the cross-level guided feature, and then the cross-level guided fusion edge feature is obtained by matrix multiplication :

[0106] ,

[0107] The MLP of this embodiment includes multi-layer perception calculation and pooling operation, and the specific parameter setting depends on the feature size of each layer. Among them, the cross-level guided fusion edge features are processed by the residual block to obtain the cross-level guided fusion features of this layer. .

[0108] Finally, the four fusion layers obtain the fusion features , , and .

[0109] S4: Decode multi-level fusion features. Figure 4 As shown in the figure, this method decodes the multi-level fusion features by decoding them layer by layer. , in cross-level fusion, its size is , use a layer of MLP to map it to 256 channels, and upsample it to get a feature map size of , feature map size and fusion features The size is the same, completing the first stage of decoding. In the next level, the decoded feature map is After splicing, the size is In stages 2, 3, and 4, according to Figure 4 Perform the same decoding operation and concatenate it with the feature maps of other levels. After splicing, the size is In the last MLP, the present invention sets the number of output channels to the number of categories that need to be segmented, and then obtains a size of One-hot encoding of . For an original image of the same size, each pixel corresponds to the position of the one-hot encoding, which is the category number represented by the pixel, thereby achieving multimodal semantic segmentation.

[0110] Embodiment 2:

[0111] An embodiment of the present invention provides a system based on the multimodal image semantic segmentation method or the application method, including:

[0112] A preprocessing module is used to preprocess the RGB image of the target scene and the additional modality image to align the spatial information of the RGB and additional modality images;

[0113] A pseudo color coding processing module, used for performing pseudo color coding processing on the pre-processed additional modality image;

[0114] The encoding fusion module is used to construct a dual-branch encoder based on the Transformer basic block and a cross-level fusion module, and use the dual-branch encoder to extract RGB modality features and additional modality features, and use the fusion module to perform feature fusion to obtain fusion features at each level;

[0115] The dual-branch encoder includes two branch encoders, which correspond to the preprocessed RGB image and the additional modality image after pseudo-color encoding, respectively. The branch encoders and the fusion module are both multi-level structures. The RGB modality features and the additional modality features extracted from the same encoding layer of the two branch encoders are fused through the fusion module, and the fused features of the previous level are input into the next fusion layer.

[0116] A decoding module is used to construct a decoder, and the decoder performs multi-level fusion feature decoding based on the fusion features of each level to obtain a semantic segmentation result;

[0117] The sample construction module is used to obtain the RGB image of the target scene and the additional modality image as training samples, and perform semantic segmentation labeling (corresponding to the number of segmentation categories C set);

[0118] The training module is also used to use the training samples and labels to train the constructed dual-branch encoder, cross-level fusion module and decoder to obtain a semantic segmentation network, and then the constructed semantic segmentation network is used to perform semantic segmentation on the RGB images to be processed and the additional modality images.

[0119] Among them, please refer to the above method content for the specific implementation process of each module, the present invention will not be repeated here, and the division of the above functional modules is only for example illustration. In some embodiments, some functional modules can be merged, and some functional modules can be split. Each functional module can be implemented in software or hardware or a combination of software and hardware. Among them, the hardware and software equipment includes but is not limited to general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0120] Embodiment 3:

[0121] An embodiment of the present invention provides an electronic device, comprising: one or more processors; and a memory storing one or more computer programs; wherein the processor calls the computer program to execute:

[0122] The multimodal image semantic segmentation method or the steps of the application method.

[0123] The process of executing the multimodal image semantic segmentation method is as follows:

[0124] S1: preprocessing the RGB image of the target scene and the additional modality image to align the spatial information of the RGB and additional modality images;

[0125] S2: Pseudo-color coding of preprocessed additional modality images;

[0126] S3: construct a dual-branch encoder based on the Transformer basic block and a cross-level fusion module, and use the dual-branch encoder to extract RGB modality features and additional modality features, and use the fusion module to perform feature fusion through the fusion module to obtain fusion features of each level;

[0127] S4: constructing a decoder, wherein the decoder performs multi-level fusion feature decoding based on the fusion features of each level to obtain a semantic segmentation result;

[0128] Among them, the RGB image, the extra-modal image and the semantic segmentation mark of the target scene are used as sample data, and the RGB image and the extra-modal image are processed according to the process of steps S1-S2; then the dual-branch encoder, the cross-level fusion module and the decoder constructed according to steps S3-S4 are trained to obtain the semantic segmentation network for image semantic segmentation.

[0129] For the specific implementation process of each step, please refer to the description of the above method.

[0130] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.

[0131] Embodiment 4:

[0132] The present invention provides a computer-readable storage medium, which stores a computer program programmed or configured to implement the multimodal image semantic segmentation method or the application method.

[0133] The process of executing the multimodal image semantic segmentation method is as follows:

[0134] S1: preprocessing the RGB image of the target scene and the additional modality image to align the spatial information of the RGB and additional modality images;

[0135] S2: Pseudo-color coding of preprocessed additional modality images;

[0136] S3: construct a dual-branch encoder based on the Transformer basic block and a cross-level fusion module, and use the dual-branch encoder to extract RGB modality features and additional modality features, and use the fusion module to perform feature fusion through the fusion module to obtain fusion features of each level;

[0137] S4: constructing a decoder, wherein the decoder performs multi-level fusion feature decoding based on the fusion features of each level to obtain a semantic segmentation result;

[0138] Among them, the RGB image, the extra-modal image and the semantic segmentation mark of the target scene are used as sample data, and the RGB image and the extra-modal image are processed according to the process of steps S1-S2; then the dual-branch encoder, the cross-level fusion module and the decoder constructed according to steps S3-S4 are trained to obtain the semantic segmentation network for image semantic segmentation.

[0139] For the specific implementation process of each step, please refer to the description of the above method.

[0140] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the software and hardware device described in any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0141] Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0142] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is a device for realizing the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram with reference to the instructions executed by the processor according to the method, device (system) and computer program product of the embodiment of the present application. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which realizes the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0143] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A multimodal image semantic segmentation method based on cross-level guided fusion, characterized by: The following steps are involved: S1: preprocessing the RGB image of the target scene and the additional modality image to align the spatial information of the RGB and additional modality images; S2: Pseudo-color coding of preprocessed additional modality images; S3: construct a dual-branch encoder based on the Transformer basic block and a cross-level fusion module, and use the dual-branch encoder to extract RGB modality features and additional modality features, and use the fusion module to perform feature fusion to obtain fusion features at each level; The dual-branch encoder includes two branch encoders, which correspond to the preprocessed RGB image and the additional modality image after pseudo-color encoding, respectively. The branch encoders and the fusion module are both multi-level structures; the RGB modality features and the additional modality features extracted from the same encoding layer on the two branch encoders are fused through the fusion module, and the fused features of the previous level are input into the next fusion layer; S4: constructing a decoder, wherein the decoder performs multi-level fusion feature decoding based on the fusion features of each level to obtain a semantic segmentation result; The RGB image, the extra-modal image and the semantic segmentation mark of the target scene are used as sample data, and the RGB image and the extra-modal image are processed according to the process of steps S1-S2; then the dual-branch encoder, the cross-level fusion module and the decoder constructed according to steps S3-S4 are trained to obtain a semantic segmentation network for image semantic segmentation; The fusion layer of the fusion module adopts cross-level guided multimodal fusion, and the fusion process is as follows: First, the linear representation is calculated using the RGB modality features and additional modality features extracted by the encoder of the same layer: ， ， ， ， ， ， In the formula, , , Represents RGB modality features The corresponding query, key and value, , , Represents additional modal features Corresponding query, key and value, W is the weight matrix, , , , , , Corresponding to , , , , , The weight matrix of , n represents the level; Then, the two modalities are fused by calculating the attention to obtain the modal fusion feature , , and then splice to get the splicing features : ， ， ， In the formula, , They correspond to the fusion features of RGB modality features and additional modality features respectively, and T is the matrix transposition symbol. Indicates the number of channels; Represents a splicing operation, Indicates that a set of real values ​​is converted into a probability distribution, and the sum of the probabilities of all parts is 1; if level n is not the first level, the concatenated features are combined with the low-level fusion features of the multi-layer perceptron MLP. Perform matrix multiplication to obtain cross-level guided splicing features : ， The function is defined as: Let A and B be two matrices, the dimension of A is , the dimension of B is ,but: ， In the formula, , is the element value of matrix A and B; The process of the decoder performing multi-level fusion feature decoding based on the fusion features of each level to obtain the semantic segmentation result is as follows: First, the fusion features of the last level are upsampled by the multi-layer perceptron MLP to obtain the feature map; Then, the feature map is spliced ​​with the fusion features of the previous level, and then the multi-layer perceptron MLP upsampling is performed; wherein, this step is performed in sequence until the feature map is spliced ​​with the fusion features of the first level to obtain the feature map; Finally, the number of output channels is set to the number of segmented categories C, and then the feature map is upsampled by a multi-layer perceptron MLP to obtain a size of One-hot encoding of is the original image size, where the position of each pixel corresponding to the unique hot encoding is the category number represented by the corresponding pixel.

2. The method according to claim 1, characterized in that: Each branch encoder is respectively provided with M encoding layers connected in sequence, and the fusion module is provided with M fusion layers connected in sequence; Among them, the output of the same encoding layer of the two branch encoders is used as the input data of the fusion layer of the corresponding layer, and the output of the previous fusion layer and the object edge information are also used as the input data of the next fusion layer. M is a positive integer.

3. The method according to claim 1, characterized in that: If each fusion layer also fuses object edge information, the method further includes: Extract object edge information from RGB images; If the fusion layer is not the first layer, the object edge information and the cross-level guided splicing features are combined. The cross-level guided fusion edge features are obtained by matrix multiplication as the fusion features of the corresponding level; ， In the formula, Guide the fusion of edge features across levels for the nth layer; Represents the edge information obtained by edge detection of the input RGB image; If the fusion layer is the first layer, the edge information of the object and the splicing features The cross-level guided fusion edge features are obtained by matrix multiplication as the fusion features of the corresponding level.

4. The method according to claim 1, characterized in that: The additional modality image is a depth image, an infrared image, or a spectral image.

5. The method according to claim 1, characterized in that: The target scenarios are robot intelligent sorting and drone intelligent inspection.

6. A system based on the multimodal image semantic segmentation method according to any one of claims 1 to 5, characterized in that: include: A preprocessing module is used to preprocess the RGB image of the target scene and the additional modality image to align the spatial information of the RGB and additional modality images; A pseudo color coding processing module, used for performing pseudo color coding processing on an additional modality image; The encoding fusion module is used to construct a dual-branch encoder based on the Transformer basic block and a cross-level fusion module, and use the dual-branch encoder to extract RGB modality features and additional modality features, and use the fusion module to perform feature fusion to obtain fusion features at each level; The dual-branch encoder includes two branch encoders, corresponding to the RGB image and the encoded additional modality image respectively. The branch encoders and the fusion module are both multi-level structures, and the RGB modality features and the additional modality features extracted from the same encoding layer on the two branch encoders are feature fused, and the fused features of the previous level are input into the next fusion layer; The decoding module is used to construct a decoder, and the decoder performs multi-level fusion feature decoding based on the fusion features of each level to obtain a semantic segmentation result.

7. An electronic device, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to execute: The steps of the multimodal image semantic segmentation method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to implement the multimodal image semantic segmentation method according to any one of claims 1 to 5.

Images