Workpiece multi-dimensional feature segmentation method and system based on cross-modal mamba network

Abstract

The application discloses a workpiece multi-dimensional feature segmentation method and system based on a cross-modal Mamba network, and relates to the technical field of image segmentation. Firstly, a two-dimensional texture image and a three-dimensional depth image that are pixel-level aligned in spatial coordinates are acquired, and texture feature sequences and geometric feature sequences are formed by mapping; texture coding features and geometric coding features are extracted through a parameter-shared visual state space module branch; a long-range topological correlation between local edge details and global contour structures is established by using a two-dimensional selective scanning mechanism; dynamic control parameters are generated according to geometric state features, to guide texture state feature reconstruction and compensation; and a workpiece segmentation mask is generated according to feature signal-to-noise ratio adjustment fusion weights. The application can reduce edge fracture and area loss in a complex industrial environment, and reduce the calculation overhead of global feature modeling of a high-resolution image.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a method and system for multidimensional feature segmentation of workpieces based on cross-modal Mamba networks. Background Technology

[0002] Industrial robots have been applied to tasks such as casting grinding, sorting of scattered parts, and precision assembly. In these tasks, it is typically necessary to first determine the location and boundaries of the workpiece from images collected at the industrial site. This provides a data foundation for subsequent workpiece pose estimation, robot path planning, and force control. Existing workpiece segmentation methods mainly utilize two-dimensional texture images to extract the workpiece's color, contrast, and surface texture features, and then generate pixel-level segmentation results based on the extracted two-dimensional texture features.

[0003] With the application of industrial 3D cameras, image acquisition equipment can not only acquire two-dimensional texture images, but also simultaneously acquire three-dimensional depth images spatially registered with the two-dimensional texture images, providing information on surface height, spatial slope, and geometric shape for workpiece segmentation. Existing multimodal segmentation methods typically cascade or sum two-dimensional texture features with three-dimensional geometric features, and then use convolutional neural networks or Transformer networks to complete feature extraction and segmentation prediction. To balance global feature modeling capabilities with high-resolution image processing efficiency, selective scanning state-space models have also begun to be used for the serialization of visual features.

[0004] However, in industrial environments with metallic reflections, oil stains, shadow interference, or uneven lighting distribution, local features in 2D texture images are prone to distortion or loss. Relying solely on 2D texture features can easily lead to blurred workpiece edges or incomplete segmentation. Existing methods simply cascade or sum 2D texture features with 3D geometric features, making it difficult to compensate for damaged texture features using stable geometric topography information based on the quality of local features. Furthermore, convolutional neural networks are limited by their local receptive fields, making it difficult to establish long-range associations between local workpiece edges and the global contour; the self-attention computation of Transformer networks increases significantly with increasing input image resolution, requiring substantial computational resources when processing high-resolution industrial images. Therefore, how to achieve deep interaction between 2D texture features and 3D geometric features while controlling computational overhead, and how to dynamically guide and compensate for texture features in disturbed areas using 3D geometric information, remains a problem that existing workpiece segmentation methods need to solve. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-dimensional feature segmentation method and system for workpieces based on cross-modal Mamba networks. This method can solve the problems of easy distortion of two-dimensional texture features, insufficient fusion depth of heterogeneous features, and large computational overhead of global modeling of high-resolution images in complex industrial environments. It can achieve dynamic compensation of two-dimensional texture information by three-dimensional geometric information, improve the continuity of workpiece segmentation boundaries, and take into account the efficiency of global feature modeling of high-resolution images.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for multidimensional feature segmentation of workpieces based on cross-modal Mamba networks, comprising:

[0008] Obtain two-dimensional texture images and three-dimensional depth images with spatial coordinate pixel-level alignment, and map the two-dimensional texture images and three-dimensional depth images into spatially semantically aligned texture feature sequences and geometric feature sequences, respectively;

[0009] The texture feature sequence and geometric feature sequence are respectively input into two visual state space module branches with shared parameters to extract texture coding features and geometric coding features;

[0010] By using a two-dimensional selective scanning mechanism, texture coding features and geometric coding features are expanded into linear sequences along four orthogonal directions. Each linear sequence is scanned in parallel using a selective state space equation, and the system matrix is ​​dynamically adjusted according to the input features of each linear sequence to obtain texture state features and geometric state features that contain long-range topological associations between the local edge details of the workpiece and the global contour structure.

[0011] By using the cross-modal feature extraction module, dynamic control parameters are generated based on geometric state features. Using geometric state features as prior signals, texture state features are reconstructed and compensated to obtain enhanced texture features.

[0012] Using the modality-guided fusion module, the fusion weights of enhanced texture features and geometric features are adjusted according to the feature signal-to-noise ratio of enhanced texture features and geometric features in local regions. The enhanced texture features and geometric features are then weighted and fused to generate a workpiece segmentation mask.

[0013] Preferably, a two-dimensional texture image and a three-dimensional depth image with spatial coordinate pixel-level alignment are obtained, and the two-dimensional texture image and the three-dimensional depth image are mapped to a spatially semantically aligned texture feature sequence and a geometric feature sequence, respectively, including:

[0014] A 3D industrial camera is used to simultaneously acquire two-dimensional texture images and three-dimensional depth images. Based on the pixel-level alignment relationship between the two-dimensional texture images and the three-dimensional depth images, an overlap sampling mechanism is used to divide the two-dimensional texture images and the three-dimensional depth images into blocks, forming a texture image block sequence and a depth image block sequence.

[0015] The Laplacian operator is used to extract the depth edge auxiliary features of the depth image patch sequence, and the depth edge auxiliary features are used as the auxiliary embedding features of the depth image patch sequence.

[0016] Linear embedding is performed on the texture image patch sequence and the depth image patch sequence with auxiliary embedding features, respectively, and then projected onto a feature space of the same dimension to form a spatially semantically aligned texture feature sequence and geometric feature sequence.

[0017] Preferably, the texture feature sequence and geometric feature sequence are respectively input into two parameter-shared visual state space module branches to extract texture coding features and geometric coding features, including:

[0018] The two visual state space module branches are set as texture branch and geometry branch respectively. The texture branch and geometry branch share network parameters and are set with dynamic residual connection structure respectively.

[0019] The texture feature sequence is input into the texture branch to extract the color features, contrast features and texture detail features of the workpiece surface to form texture coding features, and an activation response map is generated based on the texture coding features;

[0020] The geometric feature sequence is input into the geometric branch to extract the normal vector change features, spatial slope features and geometric shape features of the workpiece surface, forming intermediate geometric features;

[0021] The spatial attention weights of the geometric branches are determined based on the activation response map. These spatial attention weights are then used to weight the intermediate geometric features to form geometric coding features.

[0022] Preferably, a two-dimensional selective scanning mechanism is used to expand the texture encoding features and geometric encoding features into linear sequences along four orthogonal directions, respectively. Each linear sequence is scanned in parallel using a selective state-space equation, and the system matrix is ​​dynamically adjusted based on the input features of each linear sequence. This yields texture state features and geometric state features that contain long-range topological associations between the local edge details of the workpiece and the global contour structure, including:

[0023] The texture coding features and geometric coding features are expanded into four sets of linear sequences along the forward and reverse directions of the horizontal direction and the forward and reverse directions of the vertical direction, respectively.

[0024] The discretization step size is determined based on the input characteristics of each linear sequence. A zero-order preservation strategy is adopted to discretize the state transition matrix and input control matrix in the system matrix according to the discretization step size. The discretized state transition matrix and input control matrix are dynamically adjusted according to the input characteristics.

[0025] The hidden states corresponding to each linear sequence are updated using the selective state-space equation, forming scan features corresponding to four orthogonal directions. These scan features are then fused to form texture state features and geometric state features.

[0026] Preferably, a two-dimensional selective scanning mechanism is used to expand the texture encoding features and geometric encoding features into linear sequences along four orthogonal directions, respectively. Each linear sequence is scanned in parallel using a selective state-space equation, and the system matrix is ​​dynamically adjusted based on the input features of each linear sequence. This yields texture state features and geometric state features that contain long-range topological associations between the local edge details of the workpiece and the global contour structure, including:

[0027] The texture coding features and geometric coding features are each expanded into linear sequences along four orthogonal directions;

[0028] A position-sensitive parameter predictor is used to determine whether the image patch corresponding to the input features of each linear sequence is located in the central region, edge region, or transition region of the workpiece contour, and a matrix attenuation parameter is generated based on the position of the image patch in the workpiece contour.

[0029] The system matrix is ​​adaptively attenuated using matrix attenuation parameters. Each linear sequence is scanned in parallel using selective state-space equations, and the scan features corresponding to the four orthogonal directions are fused to form texture state features and geometric state features.

[0030] Preferably, a cross-modal feature extraction module is used to generate dynamic control parameters based on geometric state features. These geometric state features are used as prior signals to guide the reconstruction and compensation of texture state features, resulting in enhanced texture features, including:

[0031] The cross-modal feature extraction module is used to perform feature mapping on texture state features and geometric state features respectively, forming texture mapping features and geometric mapping features;

[0032] The geometric mapping features are activated to form dynamic cross-modal guidance weights that characterize the guidance strength of geometric state features, and these dynamic cross-modal guidance weights are used as dynamic control parameters.

[0033] The geometric mapping features are weighted according to the dynamic control parameters. The weighted geometric mapping features are then used as prior signals and fused with the texture mapping features to guide the reconstruction and compensation of the texture state features, thereby forming enhanced texture features.

[0034] Preferably, a cross-modal feature extraction module is used to generate dynamic control parameters based on geometric state features. These geometric state features are used as prior signals to guide the reconstruction and compensation of texture state features, resulting in enhanced texture features, including:

[0035] Dynamic control parameters are generated based on geometric state features, and feature confidence levels corresponding to each local region in the texture state features are determined based on pixel saturation of the two-dimensional texture image.

[0036] The feature confidence level is compared with a preset confidence threshold. When the feature confidence level is lower than the preset confidence threshold, the weight of the geometric state feature in the dynamic control parameters is increased.

[0037] A potential field guidance function is constructed based on the height gradient of the 3D depth image. Geometric state features are used as prior signals. Dynamic control parameters and the potential field guidance function are used to guide the reconstruction and compensation of features in the texture state features that correspond to local regions below the preset confidence threshold, thereby forming enhanced texture features.

[0038] Preferably, a modality-guided fusion module is used to adjust the fusion weights of enhanced texture features and geometric features based on the feature signal-to-noise ratio in a local region, and to perform weighted fusion of enhanced texture features and geometric features to generate a workpiece segmentation mask, including:

[0039] Determine the feature signal-to-noise ratio of enhanced texture features and geometric state features in each local region, and determine the texture fusion weight and geometric fusion weight based on the feature signal-to-noise ratio;

[0040] Based on texture fusion weights and geometric fusion weights, enhanced texture features and geometric state features are weighted and fused, and features of different scales are aggregated through multi-scale cascading and skip connections to form fused features.

[0041] The fused features are reverse-scanned to restore the spatial structure. The restored fused features are mapped to pixel-level category distribution probabilities using a normalized exponential function, and a workpiece segmentation mask is generated based on the pixel-level category distribution probabilities.

[0042] Preferably, the training method for the modality-guided fusion module includes:

[0043] The corresponding fusion weights are determined based on the signal-to-noise ratio of the enhanced texture features and geometric state features in each local region corresponding to the training samples. The enhanced texture features and geometric state features are then weighted and fused according to the fusion weights. The fused features formed by the weighted fusion are then mapped to the training pixel-level category distribution probability and the training segmentation mask.

[0044] The cross-entropy loss is determined based on the training pixel-level class distribution probability and the corresponding pixel class label, and the curvature penalty term is determined based on the segmentation edge curvature of the training segmentation mask.

[0045] The cross-entropy loss is coupled with the curvature penalty term to form a geometric consistency loss function, and the network parameters of the cross-modal Mamba network are adjusted according to the geometric consistency loss function.

[0046] A multi-dimensional feature segmentation system for workpieces based on cross-modal Mamba networks, comprising:

[0047] A multimodal feature sequence construction unit is used to acquire two-dimensional texture images and three-dimensional depth images with spatial coordinate pixel-level alignment, and to map the two-dimensional texture images and three-dimensional depth images into spatially semantically aligned texture feature sequences and geometric feature sequences, respectively.

[0048] The twin feature coding unit is used to input the texture feature sequence and the geometric feature sequence into two parameter-shared visual state space module branches to extract texture coding features and geometric coding features;

[0049] The long-range dependency modeling unit is used to expand the texture coding features and geometric coding features into linear sequences along four orthogonal directions using a two-dimensional selective scanning mechanism. Each linear sequence is scanned in parallel through a selective state space equation, and the system matrix is ​​dynamically adjusted according to the input features of each linear sequence to obtain texture state features and geometric state features that contain long-range topological associations between the local edge details of the workpiece and the global contour structure.

[0050] The dynamic guidance enhancement unit is used to generate dynamic control parameters based on geometric state features using the cross-modal feature extraction module. The geometric state features are used as prior signals to guide the reconstruction and compensation of texture state features to obtain enhanced texture features.

[0051] The modal fusion segmentation unit is used to utilize the modal-guided fusion module to adjust the fusion weights of enhanced texture features and geometric features based on the feature signal-to-noise ratio of enhanced texture features and geometric features in a local region, and to perform weighted fusion of enhanced texture features and geometric features to generate a workpiece segmentation mask.

[0052] The present invention discloses the following beneficial effects:

[0053] (1) This invention obtains two-dimensional texture images and three-dimensional depth images with spatial coordinate pixel-level alignment, and maps the two types of images into spatial semantic alignment texture feature sequences and geometric feature sequences respectively, so that texture information and geometric information have a consistent spatial correspondence, reducing feature mismatch caused by modal position deviation, and providing corresponding accurate input data for subsequent cross-modal feature processing.

[0054] (2) This invention extracts texture coding features and geometric coding features respectively through two visual state space module branches with shared parameters, so that the two types of modal features form feature expressions with corresponding semantics under the same parameter constraints. Compared with directly concatenating or summing two-dimensional texture features and three-dimensional geometric features, this processing can improve the comparability between the two types of features and facilitate the use of geometric information to supplement texture information.

[0055] (3) This invention expands texture coding features and geometric coding features along four orthogonal directions through a two-dimensional selective scanning mechanism, and dynamically adjusts the system matrix according to the input features, so that the details of the workpiece edge can establish a long-range topological association with the global contour structure. The four orthogonal directions participate in feature scanning together, which can reduce the omission of structural information caused by scanning in a single direction and improve the continuous representation of the contour and local edges of large workpieces.

[0056] (4) The present invention generates dynamic control parameters through geometric state features, and uses geometric state features as prior signals to guide the reconstruction and compensation of texture state features. When metal reflection, shadow or occlusion causes local texture information to weaken, geometric state features can participate in the enhancement of texture features in the affected area, reducing the possibility of workpiece edge breakage, area missing or background missegmentation.

[0057] (5) The present invention adjusts the fusion weights based on the signal-to-noise ratio of enhanced texture features and geometric features in local regions, so that different local regions adopt a fusion ratio adapted to the quality of modal features. Regions with clear texture information can retain texture details, while regions with disturbed texture information can increase the participation of geometric information, thereby generating a workpiece segmentation mask with continuous boundaries and corresponding to the physical contour of the workpiece. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of the system structure provided in an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this embodiment provides a method for multi-dimensional feature segmentation of workpieces based on cross-modal Mamba networks, including:

[0064] Step 100: Obtain a two-dimensional texture image and a three-dimensional depth image with spatial coordinate pixel-level alignment, and map the two-dimensional texture image and the three-dimensional depth image into a spatially semantically aligned texture feature sequence and geometric feature sequence, respectively;

[0065] Step 200: Input the texture feature sequence and geometric feature sequence into two parameter-shared visual state space module branches respectively, and extract the texture coding features and geometric coding features;

[0066] Step 300: Using a two-dimensional selective scanning mechanism, the texture coding features and geometric coding features are expanded into linear sequences along four orthogonal directions respectively. Each linear sequence is scanned in parallel through the selective state space equation, and the system matrix is ​​dynamically adjusted according to the input features of each linear sequence to obtain texture state features and geometric state features that contain long-range topological associations between the local edge details of the workpiece and the global contour structure.

[0067] Step 400: Using the cross-modal feature extraction module, dynamic control parameters are generated based on the geometric state features. The geometric state features are used as prior signals to guide the reconstruction and compensation of the texture state features, thereby obtaining enhanced texture features.

[0068] Step 500: Using the modal-guided fusion module, adjust the fusion weights of enhanced texture features and geometric features according to the feature signal-to-noise ratio of enhanced texture features and geometric features in the local area, perform weighted fusion of enhanced texture features and geometric features, and generate a workpiece segmentation mask.

[0069] This embodiment uses workpiece segmentation in casting grinding as an application scenario. The workpiece to be processed is placed in the industrial operation area, and a 3D vision sensing terminal faces the workpiece surface to simultaneously acquire two-dimensional texture images and three-dimensional depth images. The workpiece surface may have metallic reflections, oil stains, shadow interference, or uneven lighting distribution, which may distort local colors, contrast, or texture details in the two-dimensional texture image. The three-dimensional depth image is used to characterize the spatial height, spatial slope, and geometric shape of the workpiece surface, providing geometric information for areas where texture information is damaged. The final workpiece segmentation mask can be used for subsequent workpiece pose estimation, robot grinding path planning, vision-guided grasping, or automated quality inspection.

[0070] This embodiment constructs a processing procedure consisting of dual-modal data representation, parameter-shared feature encoding, multi-directional selective scanning, cross-modal dynamic guidance, and modal weighted fusion. Two-dimensional texture images and three-dimensional depth images are pixel-level aligned according to the same spatial coordinates. Data generated in each processing stage is passed sequentially, ensuring that the texture modality and geometric modality maintain a correspondence within the same workpiece region.

[0071] In step 100, a 3D visual sensing terminal with pixel-level hardware alignment is used to simultaneously acquire a two-dimensional texture image and a three-dimensional depth image of the same resolution. The two-dimensional texture image can be a color image or a grayscale image; taking a color image as an example, the two-dimensional texture image is denoted as... The three-dimensional depth image is denoted as The dimensions of the two types of images are as follows:

[0072]

[0073] In the formula, Indicates the image height. The width of the image is represented by three channels; the three channels of a 2D texture image are used to carry color information, while the single channel of a 3D depth image is used to carry depth information corresponding to each pixel position. The 2D texture image and the 3D depth image have consistent pixel coordinates, and data at the same pixel position correspond to the same spatial region on the workpiece surface.

[0074] This embodiment employs block-based linear embedding technology to process two-dimensional texture images and three-dimensional depth images. In a basic implementation, the input image is divided into segments of size [missing information]. Non-overlapping image patches, and flatten each image patch and map it as A dimensional vector. The corresponding processing procedure is represented as follows:

[0075]

[0076] In the formula, This represents the input two-dimensional texture image or three-dimensional depth image; This represents an image block operator, used to divide the input image into blocks. Image blocks; This represents the flattening operator, used to unfold the spatial dimensions of an image patch into a sequence length. ; This represents the linear projection operator, used to map pixel data to a high-dimensional feature space; This represents the normalization operator, used to maintain the stability of characteristic values; This represents a sequence of feature vectors that contain local spatial information.

[0077] The above processing is performed on the two-dimensional texture image to form a texture feature sequence. Perform the above processing on the 3D depth image to form a geometric feature sequence. When using a non-overlapping block partitioning method, the texture feature sequence and geometric feature sequence are represented as follows:

[0078]

[0079] In the formula, Indicates the length of the image patch sequence. This represents the feature dimension after mapping each image patch. The 2D texture image and 3D depth image have already undergone pixel-level alignment before patching. By using the corresponding patch positions and a unified feature dimension, the texture feature sequence and geometric feature sequence maintain spatial semantic correspondence, providing a unified input format for subsequent streaming processing in the Mamba architecture.

[0080] In another implementation, the linear embedding layer employs an overlap sampling mechanism. Overlapping regions are preserved between adjacent image blocks, ensuring that each image block contains local contextual information from its neighbors, reducing feature fragmentation when workpiece edges are located at image block boundaries. The two-dimensional texture image and the three-dimensional depth image are segmented at corresponding locations, ensuring that the texture feature sequence and geometric feature sequence formed after overlap sampling maintain spatial semantic alignment.

[0081] When segmenting a 3D depth image, the Laplacian operator is used to extract auxiliary depth edge features, which are then used as auxiliary embedding information for the depth image block sequence. These auxiliary features reflect abrupt depth changes between adjacent regions, enabling the geometric feature sequence to characterize not only the absolute depth information of the workpiece surface but also the geometric variations corresponding to the parting line, step surfaces, and surface irregularities. After the auxiliary depth edge features and the depth image block sequence are jointly mapped to a high-dimensional feature space, a geometric feature sequence sensitive to abrupt geometric changes in the workpiece is formed.

[0082] After processing in step 100, the two-dimensional texture image forms a texture feature sequence, and the three-dimensional depth image forms a geometric feature sequence. The two types of feature sequences have the same or corresponding sequence positions and feature dimensions, and are respectively passed to the two visual state space module branches in step 200.

[0083] In step 200, two parameter-shared Visual State Space Block (VSSBlock) branches are constructed to form a twin feature encoding structure. The two VSSBlock branches are a texture branch and a geometry branch, respectively, and the same network parameters are used to encode the texture feature sequence and the geometric feature sequence.

[0084] The texture branch receives the texture feature sequence formed in step 100, extracts the color features, contrast features, and texture detail features of the workpiece surface, and forms texture coding features. The geometry branch receives the geometric feature sequence formed in step 100, extracts the normal vector change features, spatial slope features, and geometric shape features of the workpiece surface, and forms geometric coding features. By sharing network parameters, the two vision state space module branches use consistent feature transformation rules to process data from different modalities, enabling the texture coding features and geometric coding features to represent the semantic information corresponding to the same workpiece region.

[0085] The visual state space processing in each visual state space module branch follows a continuous linear dynamic system, and the continuous state equation and feature formation equation are expressed as follows:

[0086]

[0087]

[0088] In the formula, Indicates a hidden state; Represents the state transition matrix; Indicates the input control matrix; This represents the output path matrix; Represents the input Token vector; Indicates the characteristics of formation; The dimension representing the hidden state.

[0089] To process the model in the digital computing engine, this embodiment employs a zero-order hold strategy for discretization and introduces a discretization step size. The discretized equation is expressed as:

[0090]

[0091]

[0092] In the formula, Indicates the first The hidden state corresponding to each sequence position This represents the hidden state corresponding to the position in the previous sequence. Indicates the first Input features corresponding to each sequence position Indicates the first Features formed by the sequence positions; and The discretized state transition matrix and input control matrix are represented as follows:

[0093]

[0094] In the formula, This represents the identity matrix. Unlike state-space models that use fixed parameters, this embodiment uses the input control matrix. Output path matrix and discretization step size Set as input feature The function allows the corresponding parameters to vary with the input features. When the input features correspond to unclear metal edges or locations of geometric abrupt changes, the discretization step size and the corresponding system parameters change with the input features to selectively capture features at key locations.

[0095] The two visual state space module branches each employ a dynamic residual connection structure. During the encoding process of the texture branch, the texture feature sequence undergoes residual transfer with the corresponding intermediate texture features, and during the encoding process of the geometric feature sequence, it undergoes residual transfer with the corresponding intermediate geometric features, thus preserving the input features during the deep encoding process.

[0096] The texture branch generates an activation response map based on the feature responses formed during the encoding process, and uses the activation response map to determine the spatial attention weights corresponding to the geometric branch. The spatial attention weights are used to weight the geometric features in the geometric branch, ensuring that workpiece regions with significant responses in the texture branch are associated with their corresponding spatial regions in the geometric branch. Through parameter sharing and spatial attention weighting, texture-encoded features are used to characterize the visual consistency of the workpiece, while geometric-encoded features are used to characterize the spatial continuity of the workpiece, providing a common feature basis for subsequent use of geometric information to compensate for texture information.

[0097] In step 300, the texture encoding features and geometric encoding features formed in step 200 are arranged into two-dimensional feature maps, and multi-directional long-range dependency modeling is performed using a two-dimensional selective scan (SS2D) mechanism. Any two-dimensional feature map is denoted as... Its dimensions are represented as:

[0098]

[0099] In the formula, Represents the height of the two-dimensional feature map. Represents the width of the two-dimensional feature map. This represents the number of channels in a two-dimensional feature map.

[0100] For any two-dimensional feature map, it is expanded into four sets of linear token sequences according to the forward direction of the horizontal direction, the reverse direction of the horizontal direction, the forward direction of the vertical direction, and the reverse direction of the vertical direction. The four sets of linear token sequences traverse the same two-dimensional feature map from different directions, so that the workpiece edge details located at different rows and columns can establish a sequence relationship with adjacent areas and distant contour areas in at least one scanning direction.

[0101] Each linear token sequence is scanned in parallel using the corresponding selective state space equation (SSE). During the scan, the hidden state is updated using the aforementioned discrete state equation, and the discretization step size is adjusted accordingly. The input features of each linear token sequence change dynamically. When the linear token sequence passes through a position where the pixel gradient of the workpiece edge changes significantly, the discretization step size is reduced accordingly to enhance the sampling accuracy of the corresponding position features; when the linear token sequence passes through the relatively gentle change of the workpiece interior region, the state space model completes the state evolution according to the corresponding input features.

[0102] After the scanning features corresponding to the four orthogonal directions complete their state evolution, they are rearranged and fused according to the correspondence between each scanning direction and the two-dimensional spatial position to form texture state features and geometric state features. Texture state features retain the local color, contrast, and texture details in the two-dimensional texture image, while establishing the long-range relationship between the local texture region and the overall contour of the workpiece; geometric state features retain the spatial height, spatial slope, and geometric shape in the three-dimensional depth image, while establishing the long-range relationship between the local geometric abrupt change region and the overall shape of the workpiece.

[0103] When the selective state-space equation scans a sequence, the computational cost increases linearly with the sequence length. Therefore, while maintaining global feature propagation, the computational cost and memory usage for high-resolution industrial images can be controlled.

[0104] For strip-shaped or elongated workpieces, the two-dimensional selective scanning mechanism employs an asymmetric scanning intensity configuration. A higher scanning sampling frequency is set along the long axis of the workpiece image, while step size contraction is used along the short axis, so that the scanning parameters are adapted to the directional shape of the workpiece.

[0105] The two-dimensional selective scanning mechanism also incorporates a position-sensitive parameter predictor. Based on the position of the image patch corresponding to the current input feature within the workpiece contour, the position-sensitive parameter predictor classifies the image patch into a central region, edge region, or transition region, and generates a matrix attenuation parameter matching the corresponding position. This matrix attenuation parameter is used to adjust the system matrix in the selective state-space equation, enabling the system matrix to adaptively attenuate according to the image patch's location, thus reducing the possibility of losing local edge information during long-sequence state propagation.

[0106] After step 300, the texture encoding features form texture state features containing long-range topological relationships between local texture details and global contour structure of the workpiece, and the geometric encoding features form geometric state features containing long-range topological relationships between local geometric details and global contour structure of the workpiece. Both types of state features are then passed to step 400.

[0107] In step 400, a cross-mamba feature extraction module (CMFEB) is constructed. The cross-mamba feature extraction module generates dynamic control parameters based on geometric state features and uses geometric state features to guide and enhance texture state features.

[0108] The modal state features corresponding to the texture branch and the geometric branch are processed by the modal feature mapping function, and the enhanced texture features are represented as follows:

[0109]

[0110] In the formula, This represents the enhanced texture feature vector; and These represent the modal feature mapping functions for the texture branch and the geometric branch, respectively; This represents the dynamic cross-modal steering weights. The dynamic cross-modal steering weights are generated based on the modal feature mapping results corresponding to the geometric branches, and are represented as:

[0111]

[0112] In the formula, Represents a linear transformation. This represents the activation function. The dynamic cross-modal guided weights change with the geometric state features, and the features corresponding to the texture branches and geometric branches are fused according to the dynamic cross-modal guided weights, thereby using geometric shape information to guide texture feature reconstruction.

[0113] The cross-modal feature extraction module employs confidence-driven enhancement logic. It evaluates the pixel saturation of each local region in the 2D texture image and determines the feature confidence of the corresponding texture state feature based on pixel saturation. When strong reflections from the metal surface cause local pixel oversaturation, or shadow occlusion weakens local texture information, causing the feature confidence to fall below a preset confidence threshold, the dynamic control parameters are adjusted to give geometric state features a higher weight in the enhancement processing of the corresponding local region. Corresponding to the enhancement formula above, increasing the participation weight of the geometric branch reduces the dynamic cross-modal guidance weight. Make the weights corresponding to the geometric branches The corresponding increase.

[0114] When geometric state features are involved in texture feature compensation, a potential field guidance function is constructed based on the height gradient of the 3D depth image. This function reflects the geometric unevenness of the workpiece surface and guides the edge positions corresponding to low-confidence regions in the 2D texture image. The cross-modal feature extraction module utilizes dynamic control parameters, geometric state features, and the potential field guidance function to reconstruct and compensate for the texture state features corresponding to low-confidence regions, ensuring that areas lacking visual information maintain consistent edge orientation and regional connectivity with the surrounding geometry.

[0115] After processing in step 400, the texture state features are transformed into enhanced texture features. The enhanced texture features preserve the effective color, contrast, and texture details in the two-dimensional texture image, and include compensation information provided by the geometric state features for reflected, shadowed, or occluded areas.

[0116] In step 500, a Modality-guided Mamba Fusion Block (MMFB) is constructed. The MMFB receives the enhanced texture features formed in step 400 and the geometric state features formed in step 300, and determines the corresponding fusion weights based on the feature signal-to-noise ratio of the two types of features in each local region.

[0117] For local areas with clear texture information and minimal interference, enhanced texture features provide color, contrast, and surface texture details; for local areas where texture information is affected by metallic reflections, shadows, or occlusion, geometric state features provide relatively stable spatial height and geometric shape. The modality-guided fusion module adjusts the fusion ratio according to the feature signal-to-noise ratio of the two types of features in each local area, so that different spatial regions adopt fusion weights that match the quality of modal information.

[0118] After weighted fusion of enhanced texture and geometric features, semantic features at different scales are aggregated using multi-scale cascading and skip connection structures. Shallower features are used to preserve workpiece edge details and local geometric abrupt changes, while deeper features are used to characterize workpiece category and global contour. Skip connections pass features of corresponding scales to the fusion processing stage, forming a fused feature tensor. .

[0119] After the fused features complete multi-scale aggregation, the two-dimensional spatial structure is recovered through a reverse scan corresponding to the multi-directional unfolding process in step 300, and the class distribution probability corresponding to each pixel position is calculated using the Softmax function. The pixel-level class distribution probability is expressed as:

[0120]

[0121] In the formula, This indicates that the pixel at a given position belongs to the first... The probability of each category; Represents the fused feature tensor; Represents the segmentation weight matrix; This represents the bias term. The Softmax function is expressed as:

[0122]

[0123] In the formula, The first element to be normalized is... Each category component This represents the total number of categories. The normalized category distribution probability is used to determine the category corresponding to each pixel position and form a workpiece segmentation mask.

[0124] During the training of the cross-modal Mamba network, pixel-level segmentation is constrained by a geometric consistency-based loss function. The geometric consistency loss function consists of cross-entropy loss and a curvature penalty term. The cross-entropy loss constrains the difference between the pixel-level class distribution probability and the corresponding pixel class label; the curvature penalty term, formed based on the curvature of the predicted segmentation edges, limits excessive smoothing or irregular jagged edges. The coupled cross-entropy loss and curvature penalty term are used to adjust the network parameters of the cross-modal Mamba network.

[0125] In precision assembly scenarios, the pixel-level category distribution probability is calibrated in real time by comparing the geometric centroid deviation between the workpiece segmentation mask and the workpiece CAD model, ensuring that the edges of the generated workpiece segmentation mask coincide with the actual physical contour of the workpiece at the sub-pixel level. Geometric centroid deviation calibration is an optional process in precision assembly scenarios and does not change the basic segmentation process formed in steps 100 to 500.

[0126] Through the above processing, a pixel-level segmentation mask corresponding to the workpiece area is finally formed. The workpiece segmentation mask is used to distinguish the workpiece area from the background area and maintain the continuity of the workpiece edges, providing workpiece area data for subsequent pose estimation, path planning, vision-guided grasping, or automated quality inspection.

[0127] In another embodiment, the workpiece multi-dimensional feature segmentation method is used in dynamic operation scenarios on an assembly line. A 3D vision sensing terminal continuously acquires RGB-D image streams, and an online real-time segmentation processing terminal employs a multi-threaded asynchronous computing mechanism to perform feature extraction on the two-dimensional texture images and three-dimensional depth images of consecutive frames in parallel. Different frames complete steps 100 to 500 according to their corresponding acquisition sequence, forming workpiece segmentation masks corresponding to the consecutive frames.

[0128] For workpieces of different sizes and shapes, the state dimension or image block size of the state space model can be adjusted by changing the corresponding configuration file, making the same segmentation framework applicable to the quality inspection of tiny precision parts and the grinding of large and complex castings. The adjustment of the configuration file does not change the basic processing relationships of bimodal feature sequence construction, parameter sharing encoding, two-dimensional selective scanning, cross-modal dynamic guidance, and modal weighted fusion.

[0129] like Figure 2 As shown, this embodiment also provides a workpiece multi-dimensional feature segmentation system based on a cross-modal Mamba network, including a multi-modal feature sequence construction unit, a twin feature encoding unit, a long-range dependency modeling unit, a dynamic guidance enhancement unit, and a modal fusion segmentation unit.

[0130] The multimodal feature sequence construction unit acquires two-dimensional texture images and three-dimensional depth images with pixel-level alignment in spatial coordinates, and maps the two-dimensional texture images and three-dimensional depth images into spatially semantically aligned texture feature sequences and geometric feature sequences, respectively. The multimodal feature sequence construction unit includes a 3D vision sensing terminal, pixel-level hardware alignment processing, and feature mapping processing. The 3D vision sensing terminal simultaneously acquires two-dimensional texture images and three-dimensional depth images. Pixel-level hardware alignment processing maintains consistent spatial coordinates for the two types of images, and feature mapping processing performs segmentation, flattening, linear projection, and normalization.

[0131] The twin feature encoding unit is used to input the texture feature sequence and the geometric feature sequence into two visual state space module branches with shared parameters, respectively, to extract texture encoding features and geometric encoding features. The two visual state space module branches form texture branches and geometric branches, respectively, and establish a semantic correspondence between texture information and geometric information using shared network parameters.

[0132] The long-range dependency modeling unit utilizes a two-dimensional selective scanning mechanism to expand texture-encoded features and geometric-encoded features into linear sequences along four orthogonal directions. It then scans each linear sequence in parallel using selective state-space equations and dynamically adjusts the system matrix based on the input features of each linear sequence to form texture and geometric state features. The long-range dependency modeling unit performs two-dimensional selective scanning processing, using state evolution in four orthogonal directions to establish long-range topological relationships between local edge details and the global contour structure.

[0133] The dynamic guided enhancement unit utilizes the cross-modal feature extraction module to generate dynamic control parameters based on geometric state features. Using these geometric state features as prior signals, it guides the reconstruction and compensation of texture state features, resulting in enhanced texture features. Internally, the dynamic guided enhancement unit incorporates a dynamic parameter calculation operator. This operator generates dynamic control parameters based on the geometric state features and then reweights and compensates the texture state features according to these parameters.

[0134] The modal fusion segmentation unit utilizes the modal-guided fusion module to adjust the fusion weights of enhanced texture features and geometric features based on the signal-to-noise ratio of the enhanced texture features and geometric features in a local region. It then performs weighted fusion of the enhanced texture features and geometric features to generate a workpiece segmentation mask. The modal fusion segmentation unit performs modal-guided fusion, multi-scale concatenation, skip connections, inverse scanning, and pixel-level class probability mapping.

[0135] In an optional implementation, the workpiece multi-dimensional feature segmentation system also includes an industrial collaboration interface module. This module converts the workpiece segmentation mask into industry-standard protocol messages; when the system is configured with workpiece pose estimation processing, it also converts the workpiece segmentation mask and the workpiece pose data generated by the workpiece pose estimation processing into ROS2 messages or TCP / IP motion commands, providing guidance data for the robot's grinding, gripping, or quality inspection actuators.

[0136] Those skilled in the art will understand that the two-dimensional texture image format, image block size, state space model state dimension, and preset confidence threshold involved in this embodiment can be configured according to the workpiece size, image resolution, and work scenario. The above configuration does not change the pixel-level correspondence between the two-dimensional texture image and the three-dimensional depth image, nor does it change the data transmission relationship between the texture feature sequence, geometric feature sequence, texture coding feature, geometric coding feature, texture state feature, geometric state feature, and enhanced texture feature.

[0137] This embodiment uses parameter-sharing dual-branch visual state space processing and four-directional selective scanning to form a global feature expression of texture and geometric information under a unified spatial semantics; it dynamically guides texture state features through geometric state features to compensate for reflective, shadow, or occluded areas with geometric information; and it adjusts the dual-modal fusion weights through the signal-to-noise ratio of local features to form a workpiece segmentation mask with continuous edges.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0139] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for multidimensional feature segmentation of workpieces based on cross-modal Mamba networks, characterized in that, include: Obtain a two-dimensional texture image and a three-dimensional depth image with spatial coordinate pixel-level alignment, and map the two-dimensional texture image and the three-dimensional depth image into a spatially semantically aligned texture feature sequence and geometric feature sequence, respectively; The texture feature sequence and the geometric feature sequence are respectively input into two visual state space module branches with shared parameters to extract texture coding features and geometric coding features; The texture coding features and the geometric coding features are expanded into linear sequences along four orthogonal directions using a two-dimensional selective scanning mechanism. Each linear sequence is scanned in parallel using a selective state space equation, and the system matrix is ​​dynamically adjusted according to the input features of each linear sequence to obtain texture state features and geometric state features that contain long-range topological associations between the local edge details of the workpiece and the global contour structure. Using a cross-modal feature extraction module, dynamic control parameters are generated based on the geometric state features. The geometric state features are used as prior signals to guide the reconstruction and compensation of the texture state features, thereby obtaining enhanced texture features. Using a modal-guided fusion module, the fusion weights of the enhanced texture features and the geometric state features are adjusted according to the feature signal-to-noise ratio in a local region. The enhanced texture features and the geometric state features are then weighted and fused to generate a workpiece segmentation mask.

2. The workpiece multidimensional feature segmentation method based on cross-modal Mamba network according to claim 1, characterized in that, Obtain a two-dimensional texture image and a three-dimensional depth image with pixel-level alignment in spatial coordinates, and map the two-dimensional texture image and the three-dimensional depth image into a spatially semantically aligned texture feature sequence and geometric feature sequence, respectively, including: The two-dimensional texture image and the three-dimensional depth image are acquired simultaneously using a 3D industrial camera. Based on the pixel-level alignment relationship between the two-dimensional texture image and the three-dimensional depth image, an overlap sampling mechanism is used to divide the two-dimensional texture image and the three-dimensional depth image into blocks, forming a texture image block sequence and a depth image block sequence. The Laplacian operator is used to extract the depth edge auxiliary features of the depth image patch sequence, and the depth edge auxiliary features are used as the auxiliary embedding features of the depth image patch sequence. The texture image patch sequence and the depth image patch sequence with the auxiliary embedding features are linearly embedded and projected onto a feature space of the same dimension to form the spatially semantically aligned texture feature sequence and geometric feature sequence.

3. The workpiece multidimensional feature segmentation method based on cross-modal Mamba network according to claim 1, characterized in that, The texture feature sequence and the geometric feature sequence are respectively input into two parameter-shared visual state space module branches to extract texture coding features and geometric coding features, including: The two visual state space module branches are respectively set as texture branches and geometric branches. The texture branches and geometric branches share network parameters and are respectively set with dynamic residual connection structures. The texture feature sequence is input into the texture branch to extract the color features, contrast features, and texture detail features of the workpiece surface, forming the texture encoding features, and an activation response map is generated based on the texture encoding features; The geometric feature sequence is input into the geometric branch to extract the normal vector change features, spatial slope features and geometric shape features of the workpiece surface, forming intermediate geometric features; The spatial attention weights of the geometric branches are determined based on the activation response map, and the geometric intermediate features are weighted using the spatial attention weights to form the geometric coding features.

4. The workpiece multidimensional feature segmentation method based on cross-modal Mamba network according to claim 1, characterized in that, The texture encoding features and geometric encoding features are expanded into linear sequences along four orthogonal directions using a two-dimensional selective scanning mechanism. Each linear sequence is scanned in parallel using a selective state-space equation, and the system matrix is ​​dynamically adjusted based on the input features of each linear sequence. This yields texture state features and geometric state features that include long-range topological relationships between local edge details and the global contour structure of the workpiece, including: The texture coding features and the geometric coding features are expanded into four sets of linear sequences along the forward and reverse directions of the horizontal direction and the forward and reverse directions of the vertical direction, respectively. The discretization step size is determined based on the input characteristics of each linear sequence. A zero-order preservation strategy is adopted to discretize the state transition matrix and the input control matrix in the system matrix according to the discretization step size. The discretized state transition matrix and the input control matrix are dynamically adjusted according to the input characteristics. The hidden states corresponding to each linear sequence are updated using the selective state space equation to form scan features corresponding to four orthogonal directions. The scan features corresponding to the four orthogonal directions are then fused to form the texture state features and the geometric state features.

5. The workpiece multidimensional feature segmentation method based on cross-modal Mamba network according to claim 1, characterized in that, The texture encoding features and geometric encoding features are expanded into linear sequences along four orthogonal directions using a two-dimensional selective scanning mechanism. Each linear sequence is scanned in parallel using a selective state-space equation, and the system matrix is ​​dynamically adjusted based on the input features of each linear sequence. This yields texture state features and geometric state features that include long-range topological relationships between local edge details and the global contour structure of the workpiece, including: The texture coding features and the geometric coding features are respectively expanded into linear sequences along the four orthogonal directions; A position-sensitive parameter predictor is used to determine whether the image patch corresponding to the input features of each linear sequence is located in the center region, edge region, or transition region of the workpiece contour, and a matrix attenuation parameter is generated based on the position of the image patch in the workpiece contour. The system matrix is ​​adaptively attenuated using the matrix attenuation parameter. Each linear sequence is scanned in parallel using the selective state space equation, and the scan features corresponding to the four orthogonal directions are fused to form the texture state features and the geometric state features.

6. The workpiece multidimensional feature segmentation method based on cross-modal Mamba network according to claim 1, characterized in that, Using a cross-modal feature extraction module, dynamic control parameters are generated based on the geometric state features. These geometric state features are then used as prior signals to guide the reconstruction and compensation of the texture state features, resulting in enhanced texture features, including: The cross-modal feature extraction module is used to perform feature mapping on the texture state features and the geometric state features respectively, forming texture mapping features and geometric mapping features; The geometric mapping features are activated to form dynamic cross-modal guidance weights that characterize the guidance strength of the geometric state features, and the dynamic cross-modal guidance weights are used as the dynamic control parameters. The geometric mapping features are weighted according to the dynamic control parameters, and the weighted geometric mapping features are used as the prior signal to be fused with the texture mapping features to guide the reconstruction and compensation of the texture state features, thereby forming the enhanced texture features.

7. The workpiece multidimensional feature segmentation method based on cross-modal Mamba network according to claim 1, characterized in that, Using a cross-modal feature extraction module, dynamic control parameters are generated based on the geometric state features. These geometric state features are then used as prior signals to guide the reconstruction and compensation of the texture state features, resulting in enhanced texture features, including: The dynamic control parameters are generated based on the geometric state features, and the feature confidence level corresponding to each local region in the texture state features is determined based on the pixel saturation of the two-dimensional texture image. The feature confidence level is compared with a preset confidence threshold. When the feature confidence level is lower than the preset confidence threshold, the weight of the geometric state feature in the dynamic control parameters is increased. A potential field guidance function is constructed based on the height gradient of the three-dimensional depth image. The geometric state features are used as the prior signal. The dynamic control parameters and the potential field guidance function are used to guide the reconstruction and compensation of features in the texture state features that correspond to local regions below the preset confidence threshold, thereby forming the enhanced texture features.

8. The workpiece multidimensional feature segmentation method based on cross-modal Mamba network according to claim 1, characterized in that, Using a modality-guided fusion module, the fusion weights of the enhanced texture features and the geometric state features are adjusted based on the feature signal-to-noise ratio in a local region. Weighted fusion of the enhanced texture features and the geometric state features is then performed to generate a workpiece segmentation mask, including: The feature signal-to-noise ratios of the enhanced texture features and the geometric state features in each of the local regions are determined respectively, and the texture fusion weights and geometric fusion weights are determined respectively based on the feature signal-to-noise ratios; According to the texture fusion weight and the geometric fusion weight, the enhanced texture features and the geometric state features are weighted and fused, and features of different scales are aggregated through multi-scale cascading and skip connections to form fused features; The fused features are reverse-scanned to restore the spatial structure. The restored fused features are mapped to pixel-level category distribution probabilities using a normalized exponential function. The workpiece segmentation mask is then generated based on the pixel-level category distribution probabilities.

9. The workpiece multidimensional feature segmentation method based on cross-modal Mamba network according to claim 8, characterized in that, The training methods for the modality-guided fusion module include: The corresponding fusion weights are determined based on the feature signal-to-noise ratio of the enhanced texture features and geometric state features corresponding to the training samples in each local region. The enhanced texture features and geometric state features are then weighted and fused according to the fusion weights. The fused features formed by the weighted fusion are then mapped to the training pixel-level category distribution probability and the training segmentation mask. The cross-entropy loss is determined based on the training pixel-level category distribution probability and the corresponding pixel category label, and the curvature penalty term is determined based on the segmentation edge curvature of the training segmentation mask. The cross-entropy loss is coupled with the curvature penalty term to form a geometric consistency loss function, and the network parameters of the cross-modal Mamba network are adjusted according to the geometric consistency loss function.

10. A workpiece multi-dimensional feature segmentation system based on a cross-modal Mamba network, characterized in that, include: A multimodal feature sequence construction unit is used to acquire two-dimensional texture images and three-dimensional depth images with spatial coordinate pixel-level alignment, and to map the two-dimensional texture images and the three-dimensional depth images into spatial semantically aligned texture feature sequences and geometric feature sequences, respectively. The twin feature encoding unit is used to input the texture feature sequence and the geometric feature sequence into two parameter-shared visual state space module branches to extract texture encoding features and geometric encoding features; The long-range dependency modeling unit is used to expand the texture coding features and the geometric coding features into linear sequences along four orthogonal directions using a two-dimensional selective scanning mechanism, scan each linear sequence in parallel through a selective state space equation, and dynamically adjust the system matrix according to the input features of each linear sequence to obtain texture state features and geometric state features that contain long-range topological associations between the local edge details of the workpiece and the global contour structure. The dynamic guidance enhancement unit is used to generate dynamic control parameters based on the geometric state features using the cross-modal feature extraction module, and use the geometric state features as a priori signals to guide the reconstruction and compensation of the texture state features to obtain enhanced texture features. The modal fusion segmentation unit is used to utilize the modal-guided fusion module to adjust the fusion weights of the enhanced texture features and the geometric state features according to the feature signal-to-noise ratio of the enhanced texture features and the geometric state features in a local region, and to perform weighted fusion of the enhanced texture features and the geometric state features to generate a workpiece segmentation mask.

Images