Surface anomaly detection method based on multi-modal dynamic fusion and double-path collaborative detection

Abstract

This invention discloses a surface anomaly detection method based on multimodal dynamic fusion and dual-path collaborative detection, comprising the following steps: Step 1, constructing an enhanced multimodal dataset; Step 2, establishing a geometric-semantic collaborative perception network; Step 3, establishing a multimodal fusion deep learning defect detection model with dual-path collaborative operation; Step 4, model deployment and detection. This invention, through a three-level architecture of geometric-semantic collaborative perception, residual sensitivity enhancement, and intelligent decision optimization, solves the problems of low efficiency in multimodal feature fusion, insufficient sensitivity in micro-defect detection, and difficulty in defining defect boundaries in existing methods, thus helping to improve the accuracy and efficiency of surface quality inspection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of industrial machine vision detection, and specifically relates to a surface anomaly detection method based on multi-modal dynamic fusion and double-path collaborative detection, a multi-modal dynamic fusion method fusing three-dimensional geometric features and two-dimensional texture features, a double-path detection mechanism combining feature matching and generative reconstruction, and precise detection and positioning of small defects on the surface of an object. BACKGROUND

[0002] With the upgrading of intelligent quality inspection demand of Industry 4.0, surface defect detection has become an important part of intelligent manufacturing. In the industrial scene, the types of defects to be detected on the surface of the product cover geometric deformation, texture anomaly, small defects and other features, which puts forward strict requirements on the perception accuracy and decision efficiency of the detection system.

[0003] In recent years, although the visual detection technology based on deep learning has made significant progress in surface defect recognition, it still faces multi-dimensional challenges in practical application: the two-dimensional visual method is limited by the planar perception characteristics, and there is a significant missed detection phenomenon in geometric defect detection, and it shows sensitivity to changes in light conditions; while the three-dimensional point cloud method has advantages in geometric anomaly detection, but it lacks a texture modeling mechanism, resulting in a significant lack of surface color anomaly recognition ability. More importantly, the existing multi-modal fusion strategy has essential limitations - direct feature splicing easily causes modal feature interference, and cross-modal alignment error further leads to low multi-source information collaboration efficiency. SUMMARY

[0004] The purpose of the application is to overcome the deficiencies of the prior art, and to solve the problems of inefficient multi-modal feature fusion, insufficient small defect recognition and limited real-time performance in industrial surface defect detection. A surface anomaly detection method based on multi-modal dynamic fusion and double-path collaborative detection is proposed, which significantly improves the detection accuracy and robustness in complex industrial scenes by constructing a geometric-semantic collaborative perception network and a dynamic residual sensitive system.

[0005] The technical problems of the application are solved by the following technical solutions:

[0006] The surface anomaly detection method based on multi-modal dynamic fusion and double-path collaborative detection comprises the following steps:

[0007] Step 1: Construct an enhanced multi-modal data set

[0008] The method adopts MVTec3D-AD dataset, which is a three-dimensional surface anomaly detection benchmark dataset commonly used in the industry field, and contains 10 types of industrial products, each type of product covers various defect types, including geometric deformation, surface scratch, hole missing, etc.; in order to enhance the diversity and robustness of the data, the method pre-processes the MVTec3D-AD dataset, including multi-scale filtering denoising and local curvature feature enhancement of three-dimensional point cloud data, and photometric disturbance of two-dimensional texture image; through these processes, a physically interpretable multi-modal dataset containing three-dimensional geometric features and two-dimensional texture features is generated;

[0009] Step 2: Establishing a geometric-semantic collaborative perception network

[0010] According to the structure of the enhanced multi-modal dataset constructed in step 1, a geometric-semantic collaborative perception network is established, which is used to jointly extract spatial structure and texture semantic information from three-dimensional point cloud and two-dimensional image, and improve the cross-modal feature perception ability; this method first uses the rotation-invariant PointNeXt network to extract the geometric features in the three-dimensional point cloud data, including curvature distribution, normal vector field, etc.; the rotation-invariant PointNeXt network can extract stable and robust geometric features from three-dimensional data without being affected by rotation or translation transformation; at the same time, the improved Swin-Transformer network is used to extract the local contrast, gradient direction and other texture features of the two-dimensional texture image; through multi-scale convolution and windowed self-attention mechanism, Swin-Transformer can capture local texture information in the image, thereby improving the ability to identify details; in order to effectively fuse three-dimensional geometric features and two-dimensional texture features, this paper designs a deformable attention mechanism, which can dynamically adjust the attention weight according to the spatial relationship between three-dimensional and two-dimensional features, ensuring that the features of different modalities can be adaptively aligned and fused in space; finally, the feature representation output in this step will provide rich information for the subsequent defect detection model;

[0011] Step 3: Establishing a dual-path collaborative multi-modal fusion deep learning defect detection model

[0012] Based on the three-dimensional geometric features, two-dimensional texture features and semantic features extracted and fused in step 2, this step constructs an efficient model integrating defect detection, residual enhancement and anomaly judgment; the model adopts a double-path parallel processing mechanism: the feature matching path and the generated reconstruction path process the input data simultaneously, respectively outputting a feature difference map and a generated residual map, and finally fusing through a residual sensitivity enhancement module; in the feature matching path, a three-dimensional geometric feature library and a two-dimensional texture feature library are constructed through a shape-guided double-memory library mechanism; the three-dimensional geometric features are encoded using an improved FPFH descriptor, and a two-dimensional texture feature library is constructed through the PatchCore algorithm to calculate the similarity and locate the potential abnormal area; this step can effectively filter out the areas with high matching degree in large-scale data sets and determine the preliminary position of the defects; in the generated reconstruction path, a diffusion-driven progressive repair network is used to reconstruct the input image to generate a defect-free reference image, and the difference between the original image and the reconstructed image is calculated to obtain the reconstruction residual; the residual is used to judge the potential defect area; the output results of the two paths are then sent to the residual sensitivity enhancement module for fusion, further improving the accuracy and robustness of defect detection. In the training phase, the MVTec3D-AD data set obtained in step 1 is divided into training set and test set, and the defect category and position label are used as a supervision signal for end-to-end training optimization; through training, the model can efficiently learn the features of object surface defects and achieve high-precision detection capability on the training data; through the cooperative work of the double-path, residual enhancement and dynamic threshold judgment, this step can efficiently identify and accurately locate various defects; specifically, the output includes a pixel-level defect heat map, which shows the abnormality of the detection area, and the color intensity of the abnormal area represents the confidence of the defect; at the same time, the model also outputs a positioning bounding box for each detected defect area, accurately marking the location of the defect; in addition, the last layer of the model output layer generates a feature vector of length N, representing the detected defect categories and their corresponding probabilities, where N represents the number of defect categories, and each element of the feature vector corresponds to the probability value of the object being each type of defect;

[0013] Step 4: Model deployment and detection

[0014] According to the trained model, input the multi-modal data of the object to be detected for defect detection, output the pixel-level defect heat map and the corresponding bounding box, and realize high-precision surface defect detection and positioning.

[0015] Further, the specific process of constructing the enhanced multimodal dataset in step 1 is as follows: Based on the MVTec3D-AD benchmark dataset, the original 3D point cloud is first subjected to multi-scale filtering for noise reduction and local curvature feature enhancement using the classic 3D feature space preservation criterion; then, a non-parametric distribution calibration method is used to eliminate the range bias of the sensor, and redundant geometric descriptors are removed using the feature entropy threshold; finally, a physically interpretable anomaly detection dataset is generated, which is dominated by classic 3D geometric features and supplemented by 2D texture features, as the enhanced multimodal dataset.

[0016] Furthermore, in the data augmentation process of step 1, for 3D point clouds, the Poisson surface reconstruction algorithm is used to retain more than 95% of the point cloud density of the target area, and the robustness of the geometric data is enhanced by random rotation, translation and Gaussian noise injection; for 2D images, adaptive ROI extraction technology is applied to remove background interference, and chromatic perturbation, local occlusion and multi-scale Gaussian blur are implemented to improve data diversity and enhance the robustness of the model.

[0017] Furthermore, the construction of the geometry-semantic collaborative perception network in step 2 includes: first, using a rotation-invariant PointNeXt network to extract geometric features of the 3D point cloud, including curvature distribution and normal vector field; second, using an improved Swin-Transformer network combined with gradient direction auxiliary features to extract 2D texture features; and then, designing a deformable attention mechanism to spatially adaptively align the 3D geometric features and 2D texture features, thereby achieving dynamic fusion of cross-modal features and completing the construction of the geometry-semantic collaborative perception network.

[0018] Furthermore, the calculation formula for the deformable attention mechanism described in step 2 is as follows:

[0019]

[0020] In the formula, Q For query vector, K For key vectors, AK This is a deformable offset obtained by mapping the 3D geometric features extracted by PointNet++ through a 1×1 convolution, used to achieve spatial adaptive alignment between 3D geometric features and 2D texture features. For the feature vector dimension, the Softmax function normalizes the dot product similarity in the formula, generating the attention weight distribution, which is one of the key components of this mechanism; the deformable attention mechanism... AKBy integrating 3D geometric information into 2D feature matching and combining it with the features of Softmax selective attention alignment, cross-modal spatial alignment is achieved. Compared with ordinary attention, it can dynamically "deform" the attention weight distribution to adapt to the spatial differences of heterogeneous data, and shows stronger alignment effect and robustness in visual tasks.

[0021] Furthermore, step 3 employs a multimodal defect detection model with dual-path parallel processing: the feature matching path uses an improved FPFH descriptor to construct a three-dimensional geometric feature library and combines it with the PatchCore algorithm to establish a two-dimensional texture feature library. Then, it calculates feature similarity through K-nearest neighbor search to locate potential abnormal regions. The reconstruction path uses a diffusion-driven denoising diffusion model to progressively repair the input image, generating a defect-free reference image. It then calculates the difference between the reference image and the original image to obtain the reconstruction residual, which is used to further determine the defect region. The outputs of the two paths are fused through a residual sensitivity enhancement module to achieve accurate detection of abnormal regions.

[0022] Furthermore, the denoising diffusion model is as follows:

[0023]

[0024] In the formula, For noise dispatch coefficient, For noise terms predicted by neural networks, and These are the control parameters for the diffusion process. For resampling noise intensity, This is standard Gaussian noise. The formula applies at each time step... Regarding the current state The process of noise correction and reconstruction, which gradually generates target data from the noise space, can effectively preserve key structural features and improve the fidelity and stability of the reconstructed image. It is one of the core components of generating the reconstruction path in the defect detection model of this invention.

[0025] Furthermore, the residual sensitivity enhancement module designed in step 3 combines multi-scale sliding windows to extract local anomaly features. The window size gradually increases from 0.5mm×0.5mm to 5mm×5mm, and is converted to a window range of approximately 5×5 to 50×50 pixels based on the registration resolution of 0.1mm / pixel between the point cloud and the image. An adaptive weighted fusion strategy is used to fuse the anomaly features extracted at different scales to improve the sensitivity of defect detection.

[0026] Furthermore, the residual sensitivity enhancement module employs an adaptive weighted fusion strategy to integrate the feature maps output from the feature matching path in step 3. FThe defect heatmap H output by the generated reconstruction path is fused to obtain the fused feature map. The fusion calculation formula is as follows:

[0027]

[0028] In the formula, F Feature maps output for feature matching paths ,H To generate the defect heatmap output by the reconstruction path, To integrate weights, The parameters are dynamically adjusted based on local anomaly density, residual intensity, and spatial consistency, thereby improving the accuracy and sensitivity of anomaly detection. In addition, a three-dimensional shape constraint loss is introduced into the generation of the reconstruction path to enhance the detection capability for geometric defects.

[0029] The beneficial effects of this invention are as follows:

[0030] This invention provides a surface anomaly detection method based on multimodal dynamic fusion and dual-path collaborative detection. The method employs a three-level collaborative optimization model: first, a geometric-semantic collaborative perception network is constructed to achieve the fusion of features from different modalities; second, a dual-path detection mechanism is designed to improve detection robustness; and third, a dynamic threshold module is introduced to enhance anomaly detection sensitivity, while combining local and global features to improve the identification capability of minute defects. This method improves the accuracy and stability of detecting complex surface defects to a certain extent and has good adaptability and generalization ability. Compared with traditional single-modal detection methods, it has certain application potential and promotional value in practical industrial inspection scenarios. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall model of the detection method of the present invention, illustrating the cross-modal feature alignment and dual-path collaborative reconstruction process;

[0032] Figure 2 This is a roadmap for the multimodal feature fusion technology of the detection method of the present invention, which includes a geometric-semantic collaborative perception network and a dual-path detection model. Detailed Implementation

[0033] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0034] A surface anomaly detection method based on multimodal dynamic fusion and dual-path collaborative detection includes the following steps:

[0035] See Figure 1 Here is an overall flowchart of an embodiment of the present invention, which includes the following steps:

[0036] Step 1: Construct an enhanced multimodal dataset

[0037] The enhanced multimodal dataset is generated by constructing a joint enhancement module based on random geometric transformation of 3D point clouds and photometric perturbation of 2D images. The enhanced multimodal dataset is constructed based on MVTec3D-AD benchmark data. Specifically, by adopting the classic 3D feature space preservation criterion, multi-scale filtering denoising and local curvature feature enhancement are performed on the original 3D point cloud. At the same time, nonparametric distribution calibration is used to eliminate sensor range bias, and feature entropy thresholding is used to remove redundant geometric descriptors. After the above processing, a physically interpretable anomaly detection dataset with classic 3D geometric features as the main feature and 2D texture features as a supplement is finally generated, which is the enhanced multimodal dataset.

[0038] Step 2: Establish a geometric-semantic collaborative sensing network

[0039] For the enhanced multimodal dataset generated in step 1, three-dimensional geometric features are extracted to obtain a three-dimensional feature map, two-dimensional geometric features and two-dimensional texture feature maps are extracted, and then fused with semantic features. A deformable attention mechanism is designed to establish a geometric-semantic collaborative perception network and perform cross-modal feature alignment.

[0040] To achieve robust perception of defects on complex industrial surfaces, this invention proposes a geometry-semantic collaborative perception network architecture for deep fusion of 3D geometric and 2D texture features from an enhanced multimodal dataset. This module consists of a 3D geometric encoder, a 2D semantic encoder, and a deformable attention fusion mechanism, with the overall structure as follows: Figure 2 As shown, the 3D geometric encoder is constructed based on the rotationally invariant PointNeXt network, which consists of multiple rotationally invariant convolutional modules, point set abstraction layers, and feature aggregation modules. Each convolutional module extracts orientation-sensitive features of the point cloud, such as local curvature and normal vector distribution, through rotationally invariant operators, and achieves hierarchical abstraction of the point cloud by combining farthest point sampling with spherical clustering. In this process, the number of points in the point cloud is sampled to 2048, 1024, and 512 points respectively, and the feature dimension is expanded layer by layer from 64 dimensions to 256 dimensions. Finally, a fixed-length 3D geometric feature vector is generated through mean pooling to describe the geometric distribution of the input point cloud.

[0041] The two-dimensional semantic encoder adopts an improved Swin-Transformer network structure, which consists of two backbone modules that perform multi-head self-attention computation based on 8×8 and 4×4 windows, respectively. This structure first performs edge preprocessing on the input image to enhance the boundary gradient characteristics, and then extracts key features such as texture contrast and orientation gradient through a local attention mechanism. Each Transformer module consists of window partitioning, attention interaction, and MLP. The output feature map resolution is gradually downsampled from 56×56 to 28×28, and the dimension is increased from 96 to 192.

[0042] After the 3D and 2D feature encoding is completed, the cross-modal feature fusion stage begins. To address the challenges of spatial alignment and semantic inconsistency between point cloud features and image features, this invention designs a deformable attention fusion mechanism. This mechanism introduces an offset guided by 3D geometry into the Transformer attention path to achieve spatial resampling and matching of 2D texture features at key geometric locations. The offset is generated by 1×1 convolution mapping of the 3D features output by PointNeXt, controlling the dynamic offset of key value positions in attention calculation, thereby improving the accuracy and adaptability of cross-modal feature alignment.

[0043] Finally, the geometry-semantic collaborative perception module outputs a set of fused cross-modal feature maps, providing a robust multi-source information representation foundation for the subsequent dual-path detection network. Experiments show that the module can significantly improve the response consistency and feature representation integrity of abnormal regions, especially when dealing with workpieces with complex surface structures and texture variations, it exhibits good generalization ability.

[0044] Step 3: Establish a dual-path collaborative multimodal fusion deep learning defect detection model

[0045] The defect detection model used in this embodiment consists of a feature matching path and a generation and reconstruction path. The input consists of object surface data in two modalities: image and point cloud. The two paths process data in parallel, outputting a feature difference map and a generated residual map, respectively. The feature matching path uses Fast Point Feature Histogram (FPFH) to extract local geometric features from the point cloud and combines this with image surface texture features. The PatchCore algorithm is then used to match and compare the extracted multimodal features with a pre-built normal sample feature library to obtain a feature difference map highlighting abnormal areas. The generation and reconstruction path uses a diffusion model to reconstruct the object surface data without defects, achieving high fidelity through noise injection and denoising processes. The normal reconstruction result is obtained by performing a difference calculation between the original input and the reconstruction result to obtain a generated residual map containing potential anomalies. The anomaly information obtained from the two paths is then fed into the residual sensitivity enhancement module for fusion. This module uses a density-guided weighting method to fuse the feature difference map and the generated residual map: the anomaly signal is weighted according to the local density of the point cloud, so that the anomaly response in the high-density data area is enhanced while the anomaly signal in the sparse area is appropriately suppressed, thereby highlighting the consistent anomaly signal of the two paths and suppressing noise, resulting in a more sensitive fused residual map. Finally, an adaptive dynamic threshold is calculated based on the statistical characteristics of the fused residual distribution to identify defect areas.

[0046] Step 4: Model Deployment and Detection

[0047] The defect detection model trained in step 3 is deployed to the hardware platform of the actual detection environment. The deployed detection system collects multimodal data of the object to be detected by sensors as model input and inputs the collected data into the loaded model for defect identification. The model uses the parameters obtained from training to perform real-time inference on the input data and generates a pixel-level heat map of the defect area. Then, the bounding box of the defect area is further extracted by methods such as threshold segmentation and connected component analysis, so as to accurately mark the defect location and achieve high-precision surface defect detection and localization.

[0048] This step focuses on the application and deployment of the model in a real-world engineering environment, fully considering the compatibility of data input formats with hardware platforms. The model runs on the target hardware with an optimized inference process, ensuring that the system has stable real-time detection capabilities. By integrating multimodal sensor data, the accuracy and robustness of identifying various defects are further improved. In summary, this defect detection system has advantages such as high detection accuracy, accurate positioning, fast response speed, and strong robustness, and can meet the stringent requirements for automatic surface defect detection in industrial settings.

[0049] In summary, the proposed industrial product surface anomaly detection method achieves high-precision detection through multimodal data collaborative enhancement and geometric-semantic dynamic fusion. First, an enhanced training dataset is constructed: Poisson surface reconstruction is performed on the original 3D point cloud to repair the geometric topology, retaining over 95% of the point cloud density, and random rigid body transformation and Gaussian noise are applied to enhance geometric robustness. Simultaneously, adaptive ROI extraction is performed on the registered 2D image, and nonlinear photometric perturbation and local occlusion are applied to improve illumination invariance. Subsequently, a geometric-semantic collaborative perception network is designed, employing a rotationally equivariant PointNeXt network to extract multi-scale curvature features and normal vector fields, combined with an improved Swin-Transformer to capture local contrast and gradient direction texture features, and finally, a deformable attention machine is used. A dynamic alignment cross-modal feature space is constructed, with its weight matrix generated by 3D coordinate guidance to achieve pixel-level spatial matching. A defect detection model is further constructed: in the feature matching path, a 3D geometric feature library is built based on an improved FPFH descriptor, and a 2D texture feature library is established using the PatchCore algorithm; feature similarity is calculated through k-nearest neighbor search. In the generation and reconstruction path, a denoising diffusion model is used to gradually repair abnormal regions, and shape constraint loss is introduced to ensure reconstruction consistency. Finally, a density-adaptive dynamic threshold system is developed, which dynamically adjusts the judgment boundary by combining local point cloud density estimation and multi-scale residual analysis; when the residual intensity exceeds the threshold, it is marked as a defect. This method significantly improves the detection accuracy and robustness of surface anomalies through cross-modal feature complementarity and bidirectional verification mechanisms.

[0050] The method proposed in this invention achieves efficient feature alignment and fusion across multiple modalities through a multimodal dynamic fusion strategy and generative reconstruction path, thereby forming a clearer classification decision boundary. Combined with dynamic threshold optimization of the residual sensitivity enhancement module, it ultimately achieves higher accuracy and robustness in surface anomaly detection in industrial scenarios.

[0051] Although the embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the scope of the invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A surface anomaly detection method based on multimodal dynamic fusion and dual-path collaborative detection, characterized in that, Includes the following steps: Step 1: Construct an enhanced multimodal dataset, which includes 3D point cloud data and 2D texture images; Step 2: Based on the structure of the enhanced multimodal dataset constructed in Step 1, establish a geometric-semantic collaborative perception network to extract and fuse cross-modal features; Step 3: Based on the cross-modal features extracted and fused in Step 2, establish and train a dual-path collaborative multimodal fusion deep learning defect detection model; The dual-path collaborative multimodal fusion deep learning defect detection model established in step 3 includes two parts: a feature matching path and a generation and reconstruction path. The feature matching path uses an improved FPFH descriptor to construct a 3D geometric feature library and combines it with the PatchCore algorithm to build a 2D texture feature library. Then, it calculates feature similarity using K-nearest neighbor search to locate potential abnormal regions. The generation and reconstruction path uses a diffusion-driven denoising diffusion model to progressively repair the input image, generating a defect-free reference image. It then calculates the difference between the reference image and the original image to obtain the reconstruction residual, further used to determine defect regions. The outputs of the two paths are fused through a residual sensitivity enhancement module to achieve accurate detection of abnormal regions. The denoising diffusion model is as follows: ； in, For noise dispatch coefficient, For noise terms predicted by neural networks, and These are the control parameters for the diffusion process. For resampling noise intensity, Standard Gaussian noise is used; a defect-free reference image is generated by denoising; the reference image is differiating from the original image, and the resulting reconstructed residual map is used to characterize potential defect areas; The dual-path collaborative multimodal fusion deep learning defect detection model in step 3 also includes a residual sensitivity enhancement module to improve the accuracy of anomaly detection. This module employs an adaptive weighted fusion strategy to combine the feature maps output by the feature matching path. F Defect heatmap output with generated reconstruction path H The fusion is performed to obtain a fused feature map. G The fusion calculation formula is as follows: ； Among them, the fusion weight By dynamically adjusting the density of local anomalous point clouds, the intensity of reconstructed residuals, and spatial consistency, this adaptive fusion enhances the representation of anomalous region features in the fusion result. Simultaneously, combined with a density-adaptive dynamic threshold mechanism, the system calculates a defect determination threshold in real time based on local point cloud density and residual distribution. When the residual value exceeds the threshold, the corresponding region is determined to be anomalous, resulting in the output of a pixel-level defect heatmap and its bounding box, achieving precise defect location. This density-adaptive threshold mechanism effectively optimizes the defect region determination criteria, further improving detection sensitivity and positioning accuracy. Step 4: Based on the multimodal data of the object to be detected input from the trained model, perform defect detection, and output pixel-level defect heatmaps and corresponding bounding boxes to achieve high-precision surface defect detection and localization.

2. The surface anomaly detection method based on multimodal dynamic fusion and dual-path collaborative detection according to claim 1, characterized in that, In step 1: during the construction of the enhanced multimodal dataset, based on the MVTec3D-AD benchmark data, multi-scale filtering and local curvature feature enhancement are performed on the original 3D point cloud using the classic 3D feature space preservation criterion to preserve the rich geometric information of the target region; simultaneously, a non-parametric distribution calibration method is used to eliminate sensor range bias, ensuring data consistency, and feature entropy thresholding technology is used to remove redundant geometric descriptors; for 2D images, adaptive ROI extraction technology is applied to remove background interference, while chromatic perturbation, local occlusion, and multi-scale Gaussian blur are implemented to enhance data diversity and improve model robustness; finally, the generated enhanced multimodal dataset is dominated by classic 3D geometric features, supplemented by 2D texture features.

3. The surface anomaly detection method based on multimodal dynamic fusion and dual-path collaborative detection according to claim 1, characterized in that, In step 2, the geometry-semantic collaborative perception network includes a rotationally equivariant PointNeXt network, an improved Swin-Transformer network, and a deformable attention mechanism. The outputs of the rotationally equivariant PointNeXt network and the improved Swin-Transformer network are connected to the deformable attention mechanism. The rotationally equivariant PointNeXt network is used to extract geometric features from the 3D point cloud data: curvature distribution and normal vector field. The improved Swin-Transformer network is used to extract local contrast and gradient direction texture features from the 2D texture image. The deformable attention mechanism dynamically adjusts the attention weights according to the spatial relationship between the 3D and 2D features to ensure that features of different modalities can be adaptively aligned and fused in space.

4. The surface anomaly detection method based on multimodal dynamic fusion and dual-path collaborative detection according to claim 3, characterized in that, The deformable attention mechanism is as follows: ； in, Q For query vector, K For key vectors, ΔK This is a deformable offset obtained by mapping the 3D geometric features extracted by PointNet++ through a 1×1 convolution, used to achieve spatial adaptive alignment between 3D geometric features and 2D texture features. The dimension of the feature vector.

5. The surface anomaly detection method based on multimodal dynamic fusion and dual-path collaborative detection according to claim 1, characterized in that, The specific implementation method of training in step 3 is as follows: the enhanced MVTec3D-AD multimodal dataset obtained in step 1 is divided into training set and test set, and the defect category and location label are used as supervision signals to perform end-to-end training and optimization of the dual-path collaborative multimodal fusion deep learning defect detection model. During the detection phase, through dual-path collaborative work, residual enhancement, and dynamic threshold determination, the location bounding box of each detected defect region is output accordingly. The final output layer of the model provides a vector of length N to represent the defect category and its probability distribution of the detection result, where N is the number of defect categories and each element of the vector corresponds to the probability value of the object belonging to the corresponding defect category.

Images