A geometric self-attention semantic segmentation method and device and a storage medium

Abstract

The application discloses a geometric self-attention semantic segmentation method and device and a storage medium, and comprises the following steps: acquiring a color texture image and a depth topography image in a multi-modal image; obtaining a color texture feature map and a depth topography feature map according to the color texture image and the depth topography image; guiding the depth topography feature map to perform geometric correction through the color texture feature map to obtain a corrected geometric feature; performing depth uncertainty evaluation according to the depth topography image to obtain an uncertainty mask; dynamically fusing the color texture feature map and the corrected geometric feature according to the uncertainty mask to obtain a fusion feature map of n scales; and inputting the fusion feature map into a classification module to obtain a semantic segmentation result. The geometric correction of the depth topography feature map is used to eliminate the cross-modal boundary misplacement; the reliability evaluation of the depth data is used to dynamically adjust the weight of the geometric information in the attention calculation, so that the dynamic fusion between different modalities is realized, and the segmentation accuracy of the semantic segmentation result is ensured.

Description

Technical Field

[0001] This invention relates to the field of computer vision application technology, specifically to a geometric self-attention semantic segmentation method, apparatus, and storage medium. Background Technology

[0002] In computer vision applications, RGB-D semantic segmentation is widely used in industrial vision inspection tasks. However, current RGB-D semantic segmentation technology still faces the following core technical challenges in high-standard industrial vision inspection and automation scenarios:

[0003] On the one hand, under extremely high precision requirements, the robustness is insufficient due to the unreliability of depth sensor data. Depth sensors (such as LiDAR, ToF cameras, structured light cameras, laser profilometers, etc.) rely on light reflection time or structured light encoding. However, in industrial manufacturing processes, the objects being inspected often involve high-gloss metals, mirrors, transparent glass, or complex curved surfaces. These materials can cause holes, multiple reflections, random noise, or invalid pixels in the depth sensor data, resulting in the use of incorrect geometric information during semantic segmentation, which makes it impossible to meet the requirements of zero tolerance and high reliability for industrial quality control.

[0004] On the other hand, insufficient edge segmentation accuracy is caused by defects in the fusion strategy of semantic segmentation models and misalignment of cross-modal information. Existing fusion techniques simply splice or add features, and cannot enable the model to automatically switch modal data for fusion based on the accuracy of depth information. That is, for areas with accurate depth data, the model should highly rely on geometric information; while in areas where depth information is ineffective, the model should automatically switch to highly rely on RGB texture. Moreover, due to the low resolution and blurred edges of depth maps, existing models directly calculate geometric biases from depth maps, which can lead to "boundary overflow" or "jawing" in the segmentation results, directly affecting the positioning and measurement accuracy of automated systems. Summary of the Invention

[0005] The main technical problem solved by this invention is to address the shortcomings of existing RGB-D semantic segmentation methods in terms of data reliability judgment, modality fusion, and edge accuracy.

[0006] According to the first aspect, one embodiment provides a geometric self-attention semantic segmentation method based on uncertainty awareness, including:

[0007] Acquire multimodal images, which include color texture images and depth topography images;

[0008] The color texture image is input into the first feature extraction module for feature extraction to obtain a color texture feature map; the depth topography image is input into the second feature extraction module for feature extraction to obtain a depth topography feature map.

[0009] The depth topography feature map and the color texture feature map are input into the cross-modal boundary correction module. The cross-modal boundary correction module guides the depth topography feature map to perform geometric correction through the color texture feature map to obtain corrected geometric features.

[0010] The depth topography image is input into the depth uncertainty evaluation module to obtain an uncertainty mask; wherein, the value of each pixel in the uncertainty mask is used to characterize the depth confidence of that pixel;

[0011] The color texture feature map, the corrected geometric feature, and the uncertainty mask are input into the uncertainty perception fusion module. The uncertainty perception fusion module dynamically fuses the color texture feature map and the corrected geometric feature according to the uncertainty mask to obtain a fused feature map at n scales, where n is a positive integer.

[0012] The fused feature maps at n scales are input into the classification module to obtain the semantic segmentation result of the multimodal image.

[0013] According to a second aspect, one embodiment provides a semantic segmentation apparatus, comprising:

[0014] An image acquisition device is used to acquire multimodal images, the multimodal images including color texture images and depth topography images;

[0015] The processor, connected to the image acquisition device, is used to process the multimodal image according to the above-described geometric self-attention semantic segmentation method based on uncertainty perception, and obtain the semantic segmentation result of the multimodal image;

[0016] A display, connected to the processor, is used to display the semantic segmentation results of the multimodal image.

[0017] According to a third aspect, one embodiment provides a storage medium storing a computer program that can be executed by a processor to implement the above-described geometric self-attention semantic segmentation method based on uncertainty perception.

[0018] According to the geometric self-attention semantic segmentation method, apparatus, and storage medium of the above embodiments, by constructing a depth uncertainty assessment module, the model is endowed with the ability to "self-check" depth quality, enabling it to perform pixel-level self-assessment of depth information quality. This allows for the prediction of the confidence level of the depth information corresponding to each pixel with pixel-level accuracy, thereby achieving intelligent judgment of the reliability of depth data. This breaks the idealized assumptions of existing Transformer models regarding depth data and avoids "blind trust" in depth information. Through an uncertainty perception fusion module, an uncertainty mask is used as a gating signal, and the contribution of geometric information in attention calculation is dynamically adjusted based on depth reliability, ensuring that the semantic segmentation model can function even when depth information is unreliable. By automatically suppressing the influence of geometric bias and effectively relying on RGB texture features for compensation, dynamic fusion between different modalities is achieved, significantly improving the robustness of the model in complex environments. Through the cross-modal boundary correction module, a cross-modal boundary correction mechanism is introduced to address edge misalignment with low height map resolution. The high-frequency gradient information of the RGB image guides the geometric edges of depth features for physical alignment and sharpening, thereby forcing the edges of geometric features to align with the clear edges of the RGB texture, achieving fine-tuning of geometric boundaries, eliminating cross-modal boundary misalignment, and ensuring that the contour of the final semantic segmentation result matches the real texture edge of the object precisely at the pixel level, meeting the needs of industrial inspection and high-precision automated operation. Attached Figure Description

[0019] Figure 1 This is a flowchart of a geometric self-attention semantic segmentation method based on uncertainty perception;

[0020] Figure 2 This is a schematic diagram of the network architecture of a geometric self-attention semantic segmentation model;

[0021] Figure 3 This is a schematic diagram of the network architecture of each sub-network module of the uncertainty-aware fusion module;

[0022] Figure 4 This is the structural flowchart of the fusion module R;

[0023] Figure 5 This is a schematic diagram of a color texture image;

[0024] Figure 6 This is a schematic diagram of the depth topography image;

[0025] Figure 7 This is a schematic diagram of the semantic segmentation results;

[0026] Figure 8 This is a flowchart of the training process for a geometric self-attention semantic segmentation model;

[0027] Figure 9This is a schematic diagram of a semantic segmentation device. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0029] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0030] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0031] RGB-D semantic segmentation technology addresses the shortcomings of RGB images in semantic segmentation by introducing depth information. These shortcomings include extreme sensitivity to lighting and surface characteristics, inability to perceive geometric deformation and 3D defects, difficulty in segmenting occluded and stacked scenes, lack of spatial positioning accuracy, and weak generalization ability. This technology aims to meet the high-precision and high-reliability 3D quality inspection requirements in smart manufacturing.

[0032] Before the Transformer model became mainstream, RGB-D segmentation based on Convolutional Neural Networks (CNNs) (such as Fully Convolutional Networks (FCN) and U-Nets) was the preferred method for industrial inspection. However, under the requirements of high precision and robustness in industrial applications, the inherent defects of the CNN architecture were amplified: due to the limited local receptive field of this architecture, it is difficult to capture the global structure and shape of large-sized industrial parts (such as automotive parts, large PCB circuit boards, etc.), resulting in insufficient contextual understanding of defects and easy misidentification of local areas; furthermore, the architecture has weak long-distance dependency modeling and cannot effectively associate feature points spanning large distances, resulting in the inability to infer the missing parts from distant features when it is necessary to identify incomplete or occluded parts; in addition, the modal fusion capability of this architecture is also insufficient, as it only fuses RGB information and height information through simple channel splicing, lacking a refined fusion mechanism, which makes it impossible to accurately segment objects with complex textures and geometric bumps.

[0033] The Transformer, with its core self-attention mechanism, successfully overcame the aforementioned limitations of CNNs, thus dominating the RGB-D semantic segmentation field. However, it still has the following shortcomings, which are particularly prominent in industrial inspection scenarios where reliability and accuracy are extremely important: one is the "blind trust" of data. Existing Transformer models typically assume the reliability of depth data when fusing depth information, lacking an intrinsic mechanism to evaluate and quantify noise, holes, or invalid pixels in the input heightmap (depth map), resulting in poor model robustness. Secondly, they suffer from non-adaptive modal dependence, using static or global fusion mechanisms that fail to achieve pixel-level dynamic weighting. Consequently, when fusion fails in depth data regions, the model cannot automatically shift its focus to more reliable RGB texture features for segmentation, leading to extremely poor robustness when facing complex industrial materials or sensor failures, contributing to insufficient generalization. Thirdly, edge accuracy alignment is inadequate. To reduce computation, heightmap resolution is typically lower than RGB image resolution, and the edges of heightmaps are often blurred compared to RGB texture edges due to smoothing filters or sensor limitations. Existing models often directly calculate geometric biases from the heightmap, but due to the ambiguity of depth edges, the boundaries of the resulting semantic segmentation results often "overflow" or become "jawed." Therefore, existing Transformer models still have shortcomings in data reliability assessment, modal fusion, and edge accuracy.

[0034] To address the robustness bottlenecks and insufficient edge precision issues in existing RGB-D semantic segmentation methods, this invention proposes a geometric self-attention semantic segmentation model G. This model G constructs a depth uncertainty assessment module, endowing the model with the ability to "self-check" depth quality. This allows for pixel-level self-assessment of depth information quality, enabling intelligent judgment of depth data reliability and avoiding "blind trust" in depth information. Through an uncertainty-aware fusion module, the weight of geometric prior information in Transformer attention is dynamically adjusted based on depth reliability, ensuring the model effectively relies on RGB texture features for compensation. This achieves dynamic fusion between different modalities, significantly improving the model's robustness in complex environments. Furthermore, a cross-modal boundary correction module addresses edge misalignment due to low height map resolution by introducing a cross-modal boundary correction mechanism. This forces the edges of geometric features to align with the clear edges of the RGB texture, refining geometric boundaries and eliminating cross-modal boundary misalignment. Ultimately, this ensures that the contour of the final semantic segmentation result precisely matches the true texture edges of the object at the pixel level, meeting the needs of industrial inspection and high-precision automated operations. Furthermore, by introducing high-frequency texture gradient maps, the high-frequency changing regions in the feature space are explicitly labeled, guiding the parameters of the semantic segmentation model to quickly focus on edge and texture information. This increases the prior information of the semantic segmentation model G during the training process and reduces the training time of the semantic segmentation model.

[0035] Please refer to Figure 1 Some embodiments provide a geometric self-attention semantic segmentation method based on uncertainty awareness, which includes the following steps:

[0036] Step S100: Obtain a multimodal image, which includes a color texture image and a depth topography image.

[0037] To acquire multimodal images (RGB-D images) of products to be inspected in industrial vision inspection tasks, such as surface defect detection and welding defect detection of parts, the obtained multimodal images include color texture images (RGB images) and depth topography images. The color texture images contain rich color and texture information, while the depth information corresponding to each pixel in the depth topography image reflects the three-dimensional shape of the object to be inspected. It is not affected by changes in lighting, similar colors, or confused textures and contains stable geometric information.

[0038] The acquired multimodal images are then input into the pre-trained geometric self-attention semantic segmentation model G, which outputs the corresponding semantic segmentation results in an end-to-end manner.

[0039] The network architecture of the geometric self-attention semantic segmentation model G in this embodiment is as follows: Figure 2As shown, it includes a feature extraction module B, a cross-modal boundary correction module C, a depth uncertainty assessment module D, an uncertainty-aware fusion module U, and a classification module M. The feature extraction module B is mainly used to extract features from the color texture image and depth topography image in the multimodal image, obtaining a color texture feature map and a depth topography feature map. The cross-modal boundary correction module C is mainly used to guide the depth topography feature map to perform geometric correction by using high-frequency texture gradient information from the color texture feature map as guiding information, ensuring that the geometric edges are aligned with the RGB texture edges, thus obtaining corrected geometric features. The depth uncertainty assessment module D is mainly used to predict and quantify the reliability of the depth information corresponding to each pixel based on the depth topography image, thereby obtaining an uncertainty mask. The values ​​of each pixel in the obtained uncertainty mask are then used for... The depth confidence of the pixel is represented; the uncertainty-aware fusion module U is responsible for fusing 3D spatial information into the attention mechanism, and adjusting the geometric information from static to dynamic variable through the gating mechanism, so as to intelligently weight according to the reliability of the depth information, overcome the rigidity defects of traditional modal fusion, and realize the dynamic fusion of color texture feature map and corrected geometric features according to uncertainty mask to obtain multi-scale (such as n scales) fused feature map; the classification module M upsamples and aggregates the multi-scale fused feature map, and transforms the aggregated feature map (aggregated feature map) into pixel-level classification prediction, thereby obtaining the final semantic segmentation result of the multimodal image.

[0040] Step S110: The color texture image and depth topography image in the multimodal image are extracted by the first feature extraction module and the second feature extraction module to obtain the color texture feature map and the depth topography feature map; and the uncertainty mask is obtained by the depth uncertainty evaluation module based on the depth topography image.

[0041] After inputting the multimodal image into the pre-trained geometric self-attention semantic segmentation model G, feature extraction is first performed on the multimodal image data through feature extraction module B. Specifically, corresponding feature extraction modules can be set for the color texture image and depth topography image in the multimodal image. In this embodiment, feature extraction module B includes a first feature extraction module and a second feature extraction module. The first feature extraction module extracts features from the color texture image in the multimodal image to obtain a color texture feature map; the second feature extraction module extracts features from the depth topography image in the multimodal image to obtain a depth topography feature map.

[0042] It should be noted that the network modules used for feature extraction from color texture images and depth topography images, namely the first feature extraction module and the second feature extraction module, can be the same feature extraction module, or the first feature extraction module and the second feature extraction module can be two feature extraction modules with shared weights.

[0043] For example, to balance performance and speed, the backbone network of the ViT model (such as ViT-Base / 16) can be used as the network structure of the entire feature extraction module B. This backbone network has advantages such as natural global modeling capability, strong feature representation capability, and simple and unified structure, and is also compatible with the subsequent uncertainty-aware fusion module. In this case, the first feature extraction module and the second feature extraction module use the same backbone network and share weights, which are pre-trained weights. That is, when the received color texture image and depth shape image are input into the first feature extraction module and the second feature extraction module respectively, feature extraction is performed by directly loading the pre-trained weights of ViT-Base / 16 to obtain the corresponding color texture feature map and depth shape feature map for use by subsequent modules.

[0044] It should be noted that this embodiment uses a backbone network to extract initial features from the input multimodal image. Here, the backbone network of the ViT model is used as an example for illustration. However, other network models can also be used to extract features from multimodal images, such as the backbone networks of MobileNet, ShuffleNet (lightweight convolutional neural network), and MobileVit (lightweight Transformer).

[0045] Simultaneously, the depth topography image is input into the depth uncertainty evaluation module D to obtain an uncertainty mask. In this embodiment, the depth uncertainty evaluation module D is a key module introduced to address the problem of depth data quality fluctuations. It is used to predict in real time the reliability (confidence level) of the depth information corresponding to each pixel in the depth topography image and encodes it as an uncertainty mask m at the pixel level. uncFurthermore, the uncertainty mask is used as a "smart switch" and provided to the uncertainty perception fusion module U as a dynamic weighting term. This guides the semantic segmentation model G in this embodiment to suppress unreliable geometric features during attention calculation, preventing them from misleading the semantic segmentation model G. At the same time, in scenarios where poor-quality depth data is obtained in complex environments, it ensures that the semantic segmentation model G can adaptively and robustly switch its attention focus to more reliable RGB texture features (color texture features). This allows the semantic segmentation model G to highly rely on geometric information in areas where the depth data is accurate, while automatically switching to highly relying on RGB texture information in areas where the depth information is unreliable or invalid. This enables dynamic fusion between different modalities and improves the generalization performance of the semantic segmentation model G.

[0046] The depth uncertainty assessment module D identifies the uncertainty of depth information based on self-supervised feature learning. Since depth uncertainty is often related to low-level features of depth topography images (such as high-frequency noise, gradient abrupt changes, and depth holes), the feature extraction layer of this module first extracts features from the depth topography image to obtain low-level depth features. In this embodiment, a lightweight convolutional sub-network is used to learn, identify, and extract low-level features from the depth topography image to ensure that the overall computational cost of the model is not significantly increased. The feature extraction layer in this module is composed of multiple stacked first convolutional layers, and each first convolutional layer is followed by a batch normalization (BN) layer and an activation function (ReLU) layer. The width and / or height of the convolutional kernel of the first convolutional layer is an odd number greater than 1. For example, the feature extraction layer in this module can be composed of three stacked first convolutional layers.

[0047] Then, the low-level depth features are input into the second convolutional layer for channel compression to reduce the number of channels of the high-dimensional features to 1, resulting in a channel compression result. For example, the second convolutional layer is a convolutional layer with a kernel size of 1×1. Finally, based on the obtained channel compression result, the value corresponding to each pixel in the extracted features (channel compression result) is converted into a confidence value of [0,1] using the Sigmoid activation function to obtain an uncertainty mask. At this time, the value of each pixel in the uncertainty mask is used to represent the depth confidence of that pixel. When the depth confidence (or mask value) corresponding to a pixel tends to 0, the depth feature corresponding to that pixel is highly uncertain (corresponding to a flat, reliable surface), and the depth information corresponding to that pixel is less reliable. When the depth confidence corresponding to that pixel tends to 1, the depth feature corresponding to that pixel is highly certain (corresponding to a surface with high noise or holes), and the depth information corresponding to that pixel is less reliable.

[0048] It should be noted that the internal structure of the depth uncertainty assessment module D in this embodiment of the invention can be adjusted according to actual needs, including but not limited to: adjusting the number of layers, the size of the convolutional kernel, and the type of activation function in the first convolutional layer of its feature extraction layer, or removing the batch normalization layer; and adjusting the number of layers, the size of the convolutional kernel, and the type of activation function used when obtaining the uncertainty mask based on the channel compression result in the second convolutional layer.

[0049] Step S120: Input the depth topography feature map and the color texture feature map into the cross-modal boundary correction module. Use the color texture feature map to guide the depth topography feature map to perform geometric correction and obtain the corrected geometric features.

[0050] The cross-modal boundary correction module C mainly solves the problem of "inconsistency between geometric edges and texture edges" caused by the difference in resolution and quality between RGB images (color texture images) and depth images (depth topography images). Its core logic is to use high-resolution, high-frequency information-rich modal to guide and reshape low-resolution, blurred-edge modal. By using texture information in the color texture feature map as a guide, the geometric edges are aligned with the texture edges, thereby outputting a geometrically corrected depth topography feature map, which is to say, corrected geometric features.

[0051] In this embodiment, a high-frequency texture gradient map is extracted from the color texture feature map. For example, a first-order high-pass filtering algorithm, such as the Sobel operator, Prewitt operator, Roberts operator, and other edge detection algorithms, is used to extract the high-frequency texture gradient map from the color texture feature map. The obtained high-frequency texture gradient map highlights the clear outline and texture boundary of the object under test, and uses it as a "calibration template" for geometric edge correction.

[0052] It should be noted that high-frequency texture gradient maps can also be obtained in other ways, including but not limited to obtaining them through second-order high-pass filtering algorithms (such as the Laplacian operator), or by obtaining them based on the numerical differences in the spatial dimensions of color texture feature maps, such as by using the central difference method.

[0053] Then, the high-frequency texture gradient map and the depth shape feature map are concatenated along the channel dimension to obtain the concatenated result. Finally, the concatenated result is convolved by deformable convolution to obtain the corrected geometric features. The kernel size of the deformable convolution is 3×3, which learns autonomously during the training of the semantic segmentation model G. This allows the semantic segmentation model G to "learn" to focus the sampling points of the convolution kernel on more relevant locations (such as edge locations), thereby "pulling" the edges of the depth features back to the real physical boundaries to eliminate edge misalignment and ensure that the geometric bias of the subsequent calculation is based on the accurate object contour. This prevents incorrect geometric information from misleading the subsequent attention mechanism, thus ensuring the accuracy of the semantic segmentation result at the object contour.

[0054] It should be noted that the internal structure of the cross-modal boundary correction module C in this embodiment of the invention can be adjusted according to actual needs, including but not limited to: adjusting the number of deformable convolutions and the size of the convolution kernels used in the convolution operation.

[0055] Step S130: Input the color texture feature map, the corrected geometric features, and the uncertainty mask into the uncertainty-aware fusion module to dynamically fuse the color texture feature map and the corrected geometric features according to the uncertainty mask, so as to obtain a fused feature map at n scales.

[0056] The main function of the uncertainty-aware fusion module U in this embodiment is to achieve dynamic fusion between color texture information and geometric information during the feature aggregation stage. It also relies on uncertainty mask and geometric bias matrix obtained based on depth topography feature map to achieve mode switching. This allows for dynamic fusion of multiple modes based on the depth reliability of different regions. Specifically, in depth-reliable regions, geometric information is used to improve segmentation accuracy, while in depth-unreliable or even failed regions, the corresponding geometric bias is actively masked, and RGB texture features are selected to ensure that the automated system does not crash.

[0057] In this embodiment, the uncertainty-aware fusion module U may include multiple sub-network modules, each corresponding to a scale, used to obtain the fusion feature map of the corresponding scale of the sub-network module; wherein, for the i-th sub-network module, 1≤i≤n, n≥3, and i and n are positive integers, the larger the value of i, the smaller the corresponding scale, and the value of n can be adjusted according to actual needs. The fusion feature map of the i-th sub-network module at the corresponding scale is obtained by the following method:

[0058] Obtain the first, second, and third inputs corresponding to the scale of the i-th sub-network module; where, when i=1, the first input is the color texture feature map; when 2≤i≤n, the first input is the downsampled result of the fused feature map output by the (i-1)-th sub-network module; the second input corresponding to the i-th sub-network module is the corrected geometric feature after the (i-1)-th downsampling, then when i=1, the second input corresponding to the i-th sub-network module is the corrected geometric feature without downsampling (downsampling times are 0), which is the corrected geometric feature output by the cross-modal boundary correction module C; the third input corresponding to the i-th sub-network module is the uncertainty mask after the (i-1)-th downsampling, then when i=1, the third input corresponding to the i-th sub-network module is the uncertainty mask without downsampling (downsampling times are 0), which is the uncertainty mask output by the depth uncertainty evaluation module D.

[0059] Then, the first, second, and third inputs corresponding to the scale of the i-th sub-network module are input into the i-th sub-network module to obtain the fused feature map corresponding to the scale of the i-th sub-network module. In this embodiment, each sub-network module of the uncertainty-aware fusion module U includes a geometric coding module GE, a block coding module PE, and a fusion module R, as follows: Figure 3 As shown, the block encoding module PE is used to transform the first input (i.e., the color texture feature map or the fused feature map after downsampling at the previous scale) into a serialized feature representation suitable for Transformer processing. The geometric encoding module GE is used to obtain the corresponding geometric bias matrix based on the second input (the corrected geometric features after downsampling at the (i-1)th scale), thereby injecting prior knowledge of the spatial location and geometric structure of objects in the image into the semantic segmentation model G to compensate for the shortcomings of the global attention mechanism in local geometric perception. The fusion module R performs dynamic fusion between modalities based on the third input (the uncertainty mask after downsampling at the (i-1)th scale) and the geometric bias matrix with the serialized feature representation corresponding to the color texture feature map. It combines the calculated geometric bias with self-attention, using the mask value in the uncertainty mask as the gating signal, thereby adaptively adjusting the geometric bias matrix through the gating mechanism to generate dynamically weighted attention scores, and finally outputting a fused feature map that incorporates noise-resistant geometric information (such as the fused feature map corresponding to the i-th scale).

[0060] Taking the i-th sub-network module as an example, the first input of the i-th sub-network module at the corresponding scale, that is, the color texture feature map corresponding to the scale, is input into the block coding module PE of the sub-network module to obtain the serialized feature representation corresponding to the scale.

[0061] The second input corresponding to the scale of the i-th sub-network module, which is the corrected geometric feature corresponding to that scale, is input into the geometric encoding module (GE) of that sub-network module to obtain the geometric bias matrix corresponding to that scale. For example, after obtaining the second input corresponding to the i-th sub-network module, the geometric gradient map of the second input is obtained through a first-order high-pass filtering algorithm. Taking any two pixels as a pixel pair, for any pixel pair: according to the gradient values ​​corresponding to the two pixels in the pixel pair in the geometric gradient map, the gradient difference of the pixel pair is obtained. This process is repeated to obtain the gradient difference of all pixel pairs in the geometric gradient map. Then, the gradient difference of all pixel pairs is input into the fully connected layer for nonlinear mapping processing to obtain the geometric bias matrix corresponding to that scale.

[0062] The implementation process of the fusion module R is as follows: Figure 4 As shown, firstly, the serialized features corresponding to the scale of the i-th sub-network module are subjected to layer normalization to obtain its layer normalized feature representation. The layer normalized feature representation of the sub-network module at the corresponding scale, the third input, and the geometric bias matrix are then input into the geometric self-attention module (GSA) (not shown in the figure) to obtain the geometric prior fusion tensor corresponding to that scale. Next, based on the geometric prior fusion tensor and the serialized feature representation, the first summation result corresponding to that scale is obtained. In this embodiment, the two are residually connected (Add), and the result is recorded as the first summation result. Then, layer normalization is performed on the first summation result corresponding to that scale. The process involves processing the layers to obtain the normalized summation result corresponding to the scale. Based on this result, an enhanced feature representation corresponding to the scale is obtained. In this embodiment, the normalized summation result is input into a feedforward neural network (MLP) to improve feature representation capabilities and obtain the enhanced feature representation corresponding to the scale. Finally, the fusion feature map corresponding to the scale of the i-th sub-network module is obtained by combining the enhanced feature representation and the first summation result. In this embodiment, the enhanced feature representation and the first summation result are joined by a residual connection (Add), and the result obtained is the fusion feature map corresponding to the scale.

[0063] It should be noted that the internal structure of the uncertainty-aware fusion module U in this embodiment of the invention can be adjusted according to actual needs, including but not limited to: adjusting the number of normalization layers, the number of fully connected layers, the number of layers in the feedforward neural network and the number of its internal hidden layers, the size of the convolutional kernel used, the type of activation function, and removing the batch normalization layer, etc.

[0064] In the Geometric Self-Attention (GSA) module, the query matrix Q, key matrix K, and value matrix V are obtained through linear projection based on the layer-normalized feature representation corresponding to this scale, in order to calculate the basic attention score reflecting texture similarity. Then, the corrected high-precision geometric features (i.e., corrected geometric features) are received relative to the encoded geometric bias matrix, and an uncertainty mask is introduced as a gating signal. Based on the reliability of the quantized depth information corresponding to each pixel, the contribution of geometric information in the attention calculation is dynamically adjusted to achieve the core multimodal dynamic fusion, thereby obtaining the corresponding normalized attention score matrix, and subsequently the corresponding fused feature map. Then, based on the third input corresponding to this scale, the geometric bias matrix, and the query matrix and key matrix, the core multimodal dynamic fusion operation is performed. The resulting normalized attention score matrix can be expressed as:

[0065]

[0066] Where Score represents the normalized attention score matrix; Softmax represents the normalization operation; Q represents the query matrix, K represents the key matrix; T represents the transpose sign; d represents the number of channels in the key matrix; m unc This represents the third input corresponding to this scale; λ is a constant representing the scaling factor; b geo Represents the geometric bias matrix, " represents matrix dot product.

[0067] This represents the texture relevance term, corresponding to the base attention score; The dynamically modulated geometric bias term ensures that when the mask value in the uncertainty mask approaches 0, i.e., the depth information is unreliable, the corresponding geometric bias is automatically masked, causing the semantic segmentation model G to degenerate into relying on more reliable RGB texture features for semantic segmentation. Conversely, when the mask value in the uncertainty mask approaches 1, i.e., the depth information is reliable, the high-precision geometric information can be fully utilized to enhance the consistency of semantic segmentation.

[0068] Finally, based on the obtained normalized attention score matrix Score and value matrix V, the geometric prior fusion tensor corresponding to this scale is obtained; in this embodiment, the obtained normalized attention score matrix is ​​multiplied by the value matrix, and the result of the multiplication is the fusion feature map of the scale corresponding to this sub-network module.

[0069] In some embodiments, multiple sub-network modules in the uncertainty-aware fusion module U can be the same network module. In this case, the network module is also the uncertainty-aware module U. By adjusting the input of the network module, fusion feature maps corresponding to different scales can be obtained. At this time, the initial input of the network module is a color texture feature map, a corrected geometric feature output by the cross-modal boundary correction module C, and an uncertainty mask output by the depth uncertainty evaluation module D. The output is the fusion feature map corresponding to the first scale. For the i-th scale (2≤i≤n), the fusion feature map corresponding to the previous scale (i-1-th scale), the corrected geometric feature, and the uncertainty mask are downsampled and re-input into the network module, thereby obtaining the fusion feature map corresponding to the i-th scale.

[0070] For example, the fusion feature map corresponding to the first scale, the corrected geometric features output by the cross-modal boundary correction module C, and the uncertainty mask output by the depth uncertainty assessment module D are all downsampled by a factor of 2 (the first downsampling) and then re-inputted into the uncertainty-aware fusion module U to obtain the fusion feature map corresponding to the second scale. Then, the fusion feature map corresponding to the second scale, the corrected geometric features after the first downsampling, and the uncertainty mask are downsampled by a factor of 2 again and re-inputted into the uncertainty-aware fusion module U to obtain the fusion feature map corresponding to the third scale, and so on, until the fusion feature maps of n scales are finally obtained.

[0071] Step S140: Input the fused feature maps of n scales into the classification module to obtain the semantic segmentation results of the multimodal image.

[0072] In this embodiment, the classification module M primarily uses upsampling to gradually restore the low-resolution, high-semantic feature map encoded by the uncertainty-aware fusion module U to the same resolution as the original input image, and outputs a probability distribution map (i.e., a semantic segmentation probability map) of each pixel belonging to various semantic categories. For example, in a welding defect detection task, for any pixel on the surface of the product under test, the probability value corresponding to the different welding defect types of that pixel is output, thus obtaining the final semantic segmentation result. Its core principle is to complete the mapping from features to the pixel level through a simple multilayer perceptron and upsampling, significantly reducing the amount of computation and the number of parameters.

[0073] After receiving the multi-scale fused feature maps, the classification module M first preprocesses the fused feature maps corresponding to each scale to obtain the preprocessed fused feature maps for each scale. The preprocessing operation includes channel compression and upsampling. In this embodiment, each fused feature map is first compressed through a convolutional layer with a kernel of 1×1, and then upsampled to the resolution size corresponding to the fused feature map of the first scale through bilinear interpolation. This makes the resolution size of the preprocessed fused feature maps at each scale the same as that of the fused feature map corresponding to the first scale.

[0074] Then, an aggregated feature map is obtained based on the fused feature maps preprocessed at each scale. In this embodiment, all fused feature maps preprocessed at each scale are concatenated along the channel dimension to achieve multi-scale feature aggregation and obtain an aggregated feature map. The aggregated feature map is then input into a feedforward network block for nonlinear refinement to obtain a refined feature map. The feedforward network block includes a 3×3 convolutional layer, followed by a batch normalization (BN) layer and an activation function (ReLU) layer. The internal structure of the feedforward network block can be adjusted according to actual needs, including but not limited to: adjusting the number of convolutional layers, the size of the convolutional kernel, the type of activation function, and removing the batch normalization layer.

[0075] Finally, a semantic segmentation probability map is obtained based on the refined feature map. This map includes the probability value of each pixel for each category. In this embodiment, after obtaining the refined feature map, the number of channels in the refined feature map is projected onto the final number of semantic categories using a 1×1 convolutional layer to obtain a projected feature map. The number of channels in the projected feature map equals the total number of categories. A Softmax function operation and upsampling are then performed on the projected feature map to obtain a pixel-level semantic segmentation probability map. Finally, the semantic segmentation result of the multimodal image is obtained based on the semantic segmentation probability map, where the category corresponding to the largest probability value among all probability values ​​for each pixel is taken as the category of that pixel. During the acquisition of the semantic segmentation probability map, the number of convolutional layers, the size of the convolutional kernel, and the type of activation function can be adjusted according to actual needs.

[0076] For example, during welding defect detection, an image acquisition device performs multimodal image acquisition on the component under test, and the resulting color texture image is as follows: Figure 5 As shown, the depth topography image is as follows Figure 6 As shown, the obtained multimodal images are input into the geometric self-attention semantic segmentation model G for semantic segmentation, and the resulting semantic segmentation results are as follows. Figure 7 As shown.

[0077] The training process of the geometric self-attention semantic segmentation model G in this embodiment is as follows: Figure 8As shown, it includes the following steps:

[0078] Step S141: Obtain multimodal training images and their annotation data, wherein the multimodal training images include color texture images and depth topography images; the annotation data includes the true annotation value of each pixel in each category.

[0079] Step S142: Based on the color texture image and the depth topography image, obtain the color texture feature map, the depth topography feature map, and the uncertainty mask.

[0080] In this embodiment, features are extracted from the color texture image and the depth topography image by feature extraction module B. Specifically, the color texture image in the multimodal training image is input into the first feature extraction module for feature extraction to obtain a color texture feature map; the depth topography image in the multimodal training image is input into the second feature extraction module for feature extraction to obtain a depth topography feature map.

[0081] Simultaneously, the depth topography image is input into the depth uncertainty evaluation module D to obtain the uncertainty mask; the value of each pixel in the obtained uncertainty mask is used to characterize the depth confidence of that pixel.

[0082] Step S143: Obtain the corrected geometric features based on the depth topography feature map and the color texture feature map.

[0083] The depth topography feature map and the color texture feature map are input into the cross-modal boundary correction module, so that the depth topography feature map is guided by the color texture feature map to perform geometric correction and obtain the corrected geometric features.

[0084] Step S144: Obtain a fused feature map of n scales based on the color texture feature map, corrected geometric features, and uncertainty mask.

[0085] The color texture feature map, the corrected geometric features, and the uncertainty mask are input into the uncertainty-aware fusion module to dynamically fuse the color texture feature map and the corrected geometric features according to the uncertainty mask, so as to obtain a fused feature map at n scales.

[0086] Step S145: Obtain the semantic segmentation probability map of the multimodal training image based on the fused feature maps of n scales.

[0087] The fused feature map of n scales is input into the classification module to obtain the semantic segmentation probability map of the multimodal training image. The obtained semantic segmentation probability map includes the probability value of each pixel in each category.

[0088] It should be noted that steps S142 to S145 in the training process of the geometric self-attention semantic segmentation model G in this embodiment correspond to steps S110 to S140 above. The specific implementation method has been described in the above steps and will not be repeated here.

[0089] Step S146: Calculate the total loss function based on the uncertainty mask, the semantic segmentation probability map of the multimodal training images, and the labeled data.

[0090] In this embodiment, the total loss function includes a first loss function and a second loss function. The uncertainty prediction loss is obtained based on the uncertainty mask output by the deep uncertainty assessment module D and is used as the first loss function. The semantic segmentation loss is obtained based on the semantic segmentation probability map and corresponding annotation information of the multimodal training image output by the classification module M and is used as the second loss function.

[0091] By imposing a sparsity constraint on the uncertainty mask through uncertainty prediction loss, the semantic segmentation model G in this embodiment is forced to assume that the depth data is reliable by default (i.e., the mask value tends to 1). When the input depth data contains noise that causes incorrect geometric bias and increases the semantic segmentation error, the gradient flow generated by the semantic segmentation loss will drive the mask value of the corresponding pixel to decrease in order to shield the geometric noise, thereby reducing the total loss. This dynamic game mechanism based on the multi-task loss function enables the network parameters to automatically learn the pixel-level depth data confidence distribution during training, realizing intelligent judgment of the quality of depth information. In this embodiment, regularization is achieved by calculating the L1 norm of the uncertainty value (mask value) of all pixels in the image. Taking the current multimodal training image as an example, the first loss function can be expressed as:

[0092]

[0093] Among them, L unc Let m represent the first loss function. unc This represents the uncertainty mask corresponding to the multimodal training image; N represents the total number of pixels; m unc (j) represents the pixel value corresponding to the j-th pixel in the uncertainty mask.

[0094] The second loss function is obtained based on the semantic segmentation probability map and labeled data of the multimodal training images. In this embodiment, the semantic segmentation loss is calculated based on the probability value of each pixel in each category and the true labeled value corresponding to that category in each multimodal training image. Taking the current multimodal training image as an example, the second loss function can be expressed as:

[0095]

[0096] Among them, L seg This represents the second loss function, where N represents the total number of pixels; C represents the total number of semantic categories; and G represents the second loss function. jk P represents the true labeled value of the j-th pixel in the k-th category; jk This represents the probability value of the j-th pixel in the semantic segmentation probability map of the multimodal training image corresponding to the k-th category.

[0097] The total loss function is obtained by combining the first and second loss functions. The semantic segmentation model G is jointly trained using uncertainty prediction loss and semantic segmentation loss. This allows the semantic segmentation model G to automatically suppress the influence of geometric bias when depth information is unreliable, and instead rely on RGB texture features for compensation, thereby improving its robustness in complex environments.

[0098] The total loss function in this embodiment can be expressed as: Loss = L seg +αL unc Where α is a hyperparameter used to balance the importance of the two losses, for example, α=0.6.

[0099] Step S147: Train the geometric self-attention semantic segmentation model according to the total loss function to obtain the trained geometric self-attention semantic segmentation model.

[0100] By calculating the total loss function, the backpropagation algorithm is used to iteratively update all network parameters in the cross-modal boundary correction module C, the deep uncertainty assessment module D, the uncertainty perception fusion module U, and the classification module M until the model converges, resulting in the trained geometric self-attention semantic segmentation model G.

[0101] This embodiment constructs a depth uncertainty assessment module, endowing the model with the ability to "self-check" depth quality. This allows for pixel-level self-assessment of depth information quality, predicting the confidence level of depth information corresponding to each pixel with pixel-level precision. This enables intelligent judgment of depth data reliability, breaking the idealized assumptions of existing Transformer models regarding depth data and avoiding "blind trust" in depth information. Through an uncertainty-aware fusion module, an uncertainty mask is used as a gating signal, dynamically adjusting the contribution of geometric information to attention calculation based on depth reliability. This ensures that the semantic segmentation model can automatically... The model suppresses the influence of geometric bias and effectively relies on RGB texture features for compensation, thereby achieving dynamic fusion between different modalities and significantly improving the robustness of the model in complex environments. Through the cross-modal boundary correction module, a cross-modal boundary correction mechanism is introduced to address edge misalignment with low height map resolution. The high-frequency gradient information of the RGB image guides the geometric edges of the depth features for physical alignment and sharpening, thereby forcing the edges of the geometric features to align with the clear edges of the RGB texture, achieving fine-tuning of the geometric boundaries, eliminating cross-modal boundary misalignment, and ensuring that the contour of the final semantic segmentation result matches the real texture edges of the object precisely at the pixel level.

[0102] This invention addresses the problem of blindly trusting depth noise in existing methods by evaluating the depth uncertainty of each pixel in a depth topography image. It also solves the rigidity problem in multimodal fusion through a geometric self-attention mechanism based on uncertainty perception. Furthermore, by introducing a cross-modal boundary correction mechanism, it resolves the edge alignment inaccuracy caused by differences in modal resolutions. This solves the technical problems of "depth sensor failure in reflective / transparent areas" and "insufficient model fusion capability and edge alignment accuracy" in existing methods, resulting in more accurate semantic segmentation results and better meeting the needs of high-precision industrial inspection tasks such as those for precision metal parts and glass products.

[0103] Please refer to Figure 9 Some embodiments provide a semantic segmentation apparatus, including an image acquisition device 200, a processor 210, and a display 220, wherein:

[0104] Image acquisition device 200 is used to acquire multimodal images, the multimodal images including color texture images and depth topography images;

[0105] Processor 210, connected to image acquisition device 200, is used to process multimodal images according to the above-described geometric self-attention semantic segmentation method to obtain semantic segmentation results of the multimodal images; wherein, processor 210 includes:

[0106] The first feature extraction module is used to extract features from the color texture image in the multimodal image to obtain a color texture feature map;

[0107] The second feature extraction module is used to extract features from the depth topography image of the multimodal image to obtain a depth topography feature map.

[0108] The cross-modal boundary correction module is used to guide the depth topography feature map through the color texture feature map for geometric correction, so as to obtain the corrected geometric features.

[0109] The depth uncertainty assessment module is used to obtain an uncertainty mask based on the depth topography image; the value of each pixel in the uncertainty mask is used to characterize the depth confidence of that pixel.

[0110] The uncertainty-aware fusion module is used to dynamically fuse the color texture feature map and the corrected geometric features according to the uncertainty mask to obtain a fused feature map at n scales, where n is a positive integer;

[0111] The classification module is used to obtain the semantic segmentation results of multimodal images based on the fused feature maps of n scales.

[0112] It should be noted that the various modules of the processor in this embodiment correspond to the method steps in the geometric self-attention semantic segmentation method based on uncertainty perception described above. The specific implementation method has been described in the above method and will not be repeated here.

[0113] The display 220 is connected to the processor 210 and is used to display the semantic segmentation results of the multimodal image. The semantic segmentation results are visualized by adding the obtained semantic segmentation results to the original input image (RGB image).

[0114] Some embodiments of the present invention also disclose a storage medium storing a computer program that can be executed by a processor to implement the methods described in any of the embodiments herein.

[0115] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0116] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A geometric self-attention semantic segmentation method based on uncertainty perception, characterized in that, include: Acquire multimodal images, which include color texture images and depth topography images; The color texture image is input into the first feature extraction module for feature extraction to obtain a color texture feature map; the depth topography image is input into the second feature extraction module for feature extraction to obtain a depth topography feature map. The depth topography feature map and the color texture feature map are input into the cross-modal boundary correction module. The cross-modal boundary correction module guides the depth topography feature map to perform geometric correction through the color texture feature map to obtain corrected geometric features. The depth topography image is input into the depth uncertainty evaluation module to obtain an uncertainty mask; wherein, the value of each pixel in the uncertainty mask is used to characterize the depth confidence of that pixel; The color texture feature map, the corrected geometric feature, and the uncertainty mask are input into the uncertainty perception fusion module. The uncertainty perception fusion module dynamically fuses the color texture feature map and the corrected geometric feature according to the uncertainty mask to obtain a fused feature map at n scales, where n is a positive integer. The fused feature maps at n scales are input into the classification module to obtain the semantic segmentation result of the multimodal image.

2. The geometric self-attention semantic segmentation method as described in claim 1, characterized in that, The step of guiding the depth topography feature map through the color texture feature map to perform geometric correction, thereby obtaining corrected geometric features, includes: Extract high-frequency texture gradient maps from the color texture feature maps; The high-frequency texture gradient map and the depth topography feature map are concatenated along the channel dimension to obtain the concatenation result; The splicing result is convolved using deformable convolution to obtain corrected geometric features.

3. The geometric self-attention semantic segmentation method as described in claim 1, characterized in that, The step of inputting the depth topography image into the depth uncertainty evaluation module to obtain the uncertainty mask includes: The feature extraction layer in the depth uncertainty assessment module extracts features from the depth topography image to obtain low-level depth features. The feature extraction layer is composed of multiple stacked first convolutional layers, and each first convolutional layer is followed by a batch normalization layer and an activation function layer. The width and / or height of the convolutional kernel of the first convolutional layer is an odd number greater than 1. The low-level depth features are input into the second convolutional layer for channel compression to obtain the channel compression result; wherein, the kernel size of the second convolutional layer is 1×1; An uncertainty mask is obtained based on the channel compression result.

4. The geometric self-attention semantic segmentation method as described in claim 1, characterized in that, The step of dynamically fusing the color texture feature map and the corrected geometric features according to the uncertainty mask to obtain a fused feature map at n scales includes: The uncertainty-aware fusion module includes multiple sub-network modules, each of which is used to obtain a fusion feature map at the corresponding scale of the sub-network module; For the i-th sub-network module, 1≤i≤n, n≥3, and i is a positive integer, the fused feature map of the corresponding scale of this sub-network module is obtained by the following method: Obtain the first, second, and third inputs corresponding to the scale of the i-th sub-network module; where, when i=1, the first input is a color texture feature map; when 2≤i≤n, the first input is the downsampled result of the fused feature map output by the (i-1)-th sub-network module; the second input corresponding to the i-th sub-network module is the corrected geometric feature after the (i-1)-th downsampling; and the third input corresponding to the i-th sub-network module is the uncertainty mask after the (i-1)-th downsampling. The first, second, and third inputs corresponding to the scale of the i-th sub-network module are input into the i-th sub-network module to obtain the fused feature map corresponding to the scale of the i-th sub-network module.

5. The geometric self-attention semantic segmentation method as described in claim 4, characterized in that, The step of inputting the first, second, and third inputs corresponding to the scale of the i-th sub-network module into the i-th sub-network module to obtain the fused feature map corresponding to the scale of the i-th sub-network module includes: Each sub-network module of the uncertainty-aware fusion module includes a geometric coding module, a block coding module, and a fusion module; The first input of the scale corresponding to the i-th sub-network module is input into the block encoding module of the sub-network module to obtain the serialized feature representation corresponding to the scale; The second input corresponding to the scale of the i-th sub-network module is input into the geometric encoding module of the sub-network module to obtain the geometric bias matrix corresponding to the scale; The third input, serialized feature representation, and geometric bias matrix of the corresponding scale of the sub-network module are input into the fusion module of the sub-network module to obtain the fusion feature map of the corresponding scale of the i-th sub-network module.

6. The geometric self-attention semantic segmentation method as described in claim 5, characterized in that, The step of inputting the second input corresponding to the scale of the i-th sub-network module into the geometric encoding module of the sub-network module to obtain the geometric bias matrix corresponding to that scale includes: Obtain the second input corresponding to the i-th sub-network module, and obtain the geometric gradient map of the second input through a first-order high-pass filtering algorithm; Take any two pixels as a pixel pair. For any pixel pair: based on the gradient values ​​of the two pixels in the pixel pair in the geometric gradient map, obtain the gradient difference of the pixel pair. The gradient difference of all pixel pairs is input into a fully connected layer for nonlinear mapping to obtain the geometric bias matrix corresponding to that scale.

7. The geometric self-attention semantic segmentation method as described in claim 5, characterized in that, The step of inputting the third input, serialized feature representation, and geometric bias matrix of the sub-network module at the corresponding scale into the fusion module of the sub-network module to obtain the fused feature map at the corresponding scale of the i-th sub-network module includes: The serialized features corresponding to the scale of the i-th sub-network module are subjected to layer normalization to obtain its layer normalized feature representation; The layer-normalized feature representation, third input, and geometric bias matrix of the corresponding scale of the sub-network module are input into the geometric self-attention module to obtain the geometric prior fusion tensor corresponding to that scale. The first addition result corresponding to this scale is obtained based on the geometric prior fusion tensor and the serialized feature representation. The first summation result corresponding to this scale is subjected to layer normalization to obtain the layer normalized summation result corresponding to this scale. The enhanced feature representation corresponding to this scale is obtained by summing the layer normalization results corresponding to this scale. The fused feature map of the i-th sub-network module at the corresponding scale is obtained by adding the enhanced feature representation corresponding to the scale to the first sum.

8. The geometric self-attention semantic segmentation method as described in claim 7, characterized in that, The step of inputting the layer-normalized feature representation, the third input, and the geometric bias matrix of the corresponding scale of the sub-network module into the geometric self-attention module to obtain the geometric prior fusion tensor corresponding to that scale includes: The query matrix, key matrix, and value matrix are obtained based on the layer-normalized feature representation corresponding to this scale. The normalized attention score matrix is ​​obtained based on the third input, geometric bias matrix, query matrix, and key matrix corresponding to this scale. Based on the normalized attention score matrix and the value matrix, the geometric prior fusion tensor corresponding to this scale is obtained.

9. The geometric self-attention semantic segmentation method as described in claim 8, characterized in that, The process of obtaining the normalized attention score matrix based on the third input, geometric bias matrix, query matrix, and key matrix corresponding to this scale includes: Where Score represents the normalized attention score matrix; Softmax represents the normalization operation; Q represents the query matrix, K represents the key matrix; T represents the transpose sign; d represents the number of channels in the key matrix; m unc This represents the third input corresponding to this scale; λ is a constant representing the scaling factor; b geo Represents the geometric bias matrix, " represents matrix dot product.

10. The geometric self-attention semantic segmentation method as described in claim 1, characterized in that, The step of inputting the fused feature maps at n scales into the classification module to obtain the semantic segmentation result of the multimodal image includes: The fusion feature map corresponding to each scale is preprocessed to obtain the preprocessed fusion feature map of each scale; the preprocessing operation includes channel compression operation and upsampling operation, and the resolution of the preprocessed fusion feature map of each scale is the same as that of the fusion feature map corresponding to the first scale. The aggregated feature map is obtained based on the fused feature maps after preprocessing at each scale; The aggregated feature map is input into the feedforward network block to obtain the refined feature map; A semantic segmentation probability map is obtained from the refined feature map, which includes the probability value of each pixel in each category; wherein, the refined feature map is used to obtain a projection feature map through convolution operation, the number of channels of the projection feature map is equal to the total number of categories, and the projection feature map is subjected to Softmax function operation and upsampling operation to obtain the semantic segmentation probability map; The semantic segmentation result of the multimodal image is obtained based on the semantic segmentation probability map.

11. The geometric self-attention semantic segmentation method as described in claim 10, characterized in that, The geometric self-attention semantic segmentation method is implemented based on a geometric self-attention semantic segmentation model, which includes: The first feature extraction module is used to extract features from the color texture image in the multimodal image to obtain a color texture feature map; The second feature extraction module is used to extract features from the depth topography image in the multimodal image to obtain a depth topography feature map; The cross-modal boundary correction module is used to guide the depth topography feature map to perform geometric correction through the color texture features, so as to obtain corrected geometric features; A depth uncertainty assessment module is used to obtain an uncertainty mask based on the depth topography image; wherein, the value of each pixel in the uncertainty mask is used to characterize the depth confidence of that pixel; An uncertainty-aware fusion module is used to dynamically fuse the color texture feature map and the corrected geometric features according to the uncertainty mask to obtain a fused feature map at n scales. The classification module is used to obtain a semantic segmentation probability map based on the fused feature map at n scales, and to obtain the semantic segmentation result of the multimodal image based on the semantic segmentation probability map.

12. The geometric self-attention semantic segmentation method as described in claim 11, characterized in that, The geometric self-attention semantic segmentation model is trained using the following methods: Acquire multimodal training images and their annotation data, wherein the multimodal training images include color texture images and depth topography images; the annotation data includes the true annotation value of each pixel for each category; The color texture image is input into the first feature extraction module for feature extraction to obtain a color texture feature map; The depth topography image is input into the second feature extraction module for feature extraction to obtain a depth topography feature map; The depth topography feature map and the color texture feature map are input into the cross-modal boundary correction module so that the depth topography feature map is guided by the color texture feature map to perform geometric correction and obtain corrected geometric features. The depth topography image is input into the depth uncertainty evaluation module to obtain an uncertainty mask; wherein, the value of each pixel in the uncertainty mask is used to characterize the depth confidence of that pixel; The color texture feature map, the corrected geometric features, and the uncertainty mask are input into the uncertainty-aware fusion module to dynamically fuse the color texture feature map and the corrected geometric features according to the uncertainty mask, so as to obtain a fused feature map of n scales. The fused feature maps at n scales are input into the classification module to obtain the semantic segmentation probability map of the multimodal training image. The semantic segmentation probability map includes the probability value of each pixel in each category. The total loss function is calculated based on the uncertainty mask, the semantic segmentation probability map of the multimodal training image, and the labeled data; The geometric self-attention semantic segmentation model is trained based on the total loss function to obtain the trained geometric self-attention semantic segmentation model.

13. The geometric self-attention semantic segmentation method as described in claim 12, characterized in that, The step of calculating the total loss function based on the uncertainty mask, the semantic segmentation probability map of the multimodal training image, and the labeled data includes: The first loss function is obtained based on the uncertainty mask; A second loss function is obtained based on the semantic segmentation probability map of the multimodal training images and the labeled data; The total loss function is obtained by combining the first loss function and the second loss function.

14. The geometric self-attention semantic segmentation method as described in claim 13, characterized in that, The step of obtaining the first loss function based on the uncertainty mask includes: Among them, L unc Let m represent the first loss function. unc Indicates an uncertainty mask; N represents the total number of pixels; m unc (j) represents the pixel value corresponding to the j-th pixel in the uncertainty mask.

15. The geometric self-attention semantic segmentation method as described in claim 13, characterized in that, The step of obtaining the second loss function based on the semantic segmentation probability map of the multimodal training image and the labeled data includes: Among them, L seg This represents the second loss function, where N represents the total number of pixels; C represents the total number of semantic categories; and G represents the second loss function. jk P represents the true labeled value of the j-th pixel in the k-th category; jk This represents the probability value of the j-th pixel in the semantic segmentation probability map of the multimodal training image corresponding to the k-th category.

16. The geometric self-attention semantic segmentation method as described in claim 1, characterized in that, The first feature extraction module and the second feature extraction module are feature extraction modules with shared weights, and the weights are pre-trained weights.

17. A semantic segmentation device, characterized in that, include: An image acquisition device is used to acquire multimodal images, the multimodal images including color texture images and depth topography images; A processor, connected to the image acquisition device, is used to process the multimodal image according to the geometric self-attention semantic segmentation method as described in any one of claims 1-16, to obtain the semantic segmentation result of the multimodal image; A display, connected to the processor, is used to display the semantic segmentation results of the multimodal image.

18. The semantic segmentation apparatus as described in claim 17, characterized in that, The processor includes: The first feature extraction module is used to extract features from the color texture image to obtain a color texture feature map; The second feature extraction module is used to extract features from the depth topography image to obtain a depth topography feature map; A cross-modal boundary correction module is used to guide the depth topography feature map to perform geometric correction through the color texture feature map, so as to obtain corrected geometric features; A depth uncertainty assessment module is used to obtain an uncertainty mask based on the depth topography image; wherein, the value of each pixel in the uncertainty mask is used to characterize the depth confidence of that pixel; An uncertainty-aware fusion module is used to dynamically fuse the color texture feature map and the corrected geometric features according to the uncertainty mask to obtain a fused feature map at n scales, where n is a positive integer; The classification module is used to obtain the semantic segmentation result of the multimodal image based on the fused feature map at n scales.

19. A storage medium, characterized in that, The storage medium stores a computer program that can be executed by a processor to implement the method as described in any one of claims 1-16.

Images