Multi-modal industrial defect detection method and system based on adaptive view fusion

Abstract

The invention discloses a multi-modal industrial defect detection method and system based on adaptive view fusion, and belongs to the technical field of industrial visual detection. The method comprises the following steps: collecting point cloud and RGB image data of an industrial casting; extracting point cloud three-dimensional local geometric features and RGB image two-dimensional features, and strengthening edge and small region features through an edge detail enhancement module; performing multi-view projection on the point cloud, and screening an optimal view by using an adaptive view combination pre-selection module; fusing the multi-view features through a global-local sensing fusion module, and weighting the multi-view features with RGB features to obtain final two-dimensional feature embedding; two-dimensional and three-dimensional feature memory libraries are constructed, and defect detection is carried out in the test stage; and finally, outputting an abnormal segmentation image and a score through a local gradient guide decision fusion module. The method can effectively improve the tiny defect detection capability and reasoning speed, and has the advantages of high precision, strong robustness and good engineering applicability.

Description

Technical Field

[0001] This invention belongs to the field of industrial defect detection technology, specifically relating to a multimodal industrial defect detection method and system based on adaptive view fusion. Background Technology

[0002] Defect detection is one of the core steps in industrial quality inspection. It aims to accurately locate defective areas on industrial castings and provide efficient and accurate guidance for tasks such as robotic intelligent welding, grinding, and cutting, thereby improving production efficiency and stability.

[0003] Manual inspection suffers from low efficiency, inconsistent evaluation standards, and high labor costs. In recent years, the rapid development of deep learning has provided a new solution for automated industrial defect detection, effectively improving detection efficiency and accuracy. Due to the scarcity and diversity of defect samples in real-world production scenarios, deep learning-based industrial defect detection algorithms primarily rely on unsupervised learning.

[0004] Deep learning-based methods for 2D industrial defect detection mainly fall into two categories: reconstruction methods and feature embedding methods. Reconstruction methods utilize architectures such as generative adversarial networks (GANs) and diffusion models to learn the distribution of normal samples and reconstruct test samples, then identify defects through pixel-level reconstruction errors. However, these methods are weak in identifying minute defects and are easily affected by environmental interference. Feature embedding methods use pre-trained networks to acquire sample features and then construct detection modules to locate defects. These methods use architectures such as normalized flow, teacher-student models, and memory banks for defect detection and still exhibit good robustness in real-world industrial scenarios. However, these methods have high requirements for the quality of feature extraction. Real-world industrial scenarios often involve complex conditions such as uneven lighting, excessively high or low contrast, significant differences between industrial product characteristics and publicly available datasets, and data scarcity. Directly using pre-trained 2D feature extraction networks has limited generalization ability and makes it difficult to obtain distinguishable features. The development of sensors such as RGB-D depth cameras and structured light cameras has created conditions for simultaneously acquiring point cloud and corresponding frame image data, promoting the development of multimodal technologies in the vision field. Point cloud data possesses rotation invariance and is unaffected by external conditions such as lighting and background, thus improving detection stability. However, existing multimodal defect detection methods still have room for improvement in inference speed. Therefore, researching the comprehensive and effective extraction of point cloud and image information, enhancing the distinguishability between abnormal and normal features, and thereby improving the accuracy and efficiency of workpiece defect detection is of significant research importance and practical value. Summary of the Invention

[0005] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a multimodal industrial defect detection method and system based on adaptive view fusion. In the feature extraction stage, the method uses a cross-attention mechanism to fuse image texture information with point cloud structure information and adopts a lightweight two-dimensional feature extraction network to improve detection efficiency and reduce memory consumption. This method is used to solve the technical problems of small defects being easily missed and slow inference speed in multimodal industrial defect detection, and meets the defect detection needs of multimodal data in industrial scenarios.

[0006] The present invention adopts the following technical solution: A multimodal industrial defect detection method based on adaptive view fusion includes the following steps: S1. Collect an industrial casting dataset containing defect-free and defective samples, the dataset including point cloud data and corresponding RGB images; S2. Extract the three-dimensional local geometric features of the point cloud data using fast point feature histogram. The two-dimensional features of the RGB image are extracted using a lightweight MobileViTv3 backbone network. The two-dimensional features of the RGB image are enhanced through the edge detail enhancement module. Enhance; S3. Perform multi-view projection on the point cloud data to obtain multi-view features, and filter out the optimal view combination through the adaptive view combination pre-selection module. S4. Rotate and project the sample point cloud according to the rotation angle corresponding to the optimal viewpoint combination to obtain a target multi-view image. Use the feature extraction and enhancement method in step S2 to obtain target multi-view features. Fuse the target multi-view features through a global-local perception fusion module to obtain fused multi-view features. Combine the fused multi-view features with the two-dimensional features of the RGB image. Weighted fusion yields the final two-dimensional feature embedding. ; S5. Construct a two-dimensional feature memory and a three-dimensional feature memory to store the two-dimensional feature embeddings, respectively. and three-dimensional local geometric features The corresponding normal sample features are used to perform defect detection in two-dimensional and three-dimensional modes during the testing phase, and the abnormal scores and abnormal segmentation maps of the corresponding modes are obtained. S6. Input the anomaly segmentation map into the local gradient-guided decision fusion module to obtain the final anomaly segmentation map. and abnormal scores Complete industrial defect detection.

[0007] Preferably, step S1 further includes a preprocessing operation on the industrial casting dataset: The RGB image is smoothed by a joint bilateral filtering method to remove interfering textures; the disordered point cloud data is converted into an ordered format and aligned with the pixel positions of the corresponding RGB image; the position information of the point cloud data is stored as a three-channel tensor with x, y, and z coordinates; the three-channel tensor, the RGB image, and the reference ground truth map are simultaneously cropped to extract a square area covering the industrial casting and uniformly scaled to 224×224; data augmentation is performed through translation and rotation operations.

[0008] Preferably, the two-dimensional features of the RGB image enhanced by the edge detail enhancement module in step S2 are... The specific process of enhancement is as follows: Obtain the second-layer feature map output from the MobileViTv3 backbone network. and the third layer feature map For the second layer feature map Perform average pooling on the third layer feature map. After performing 1×1 convolution, the feature is added pixel-by-pixel to the pooled second-layer feature map; the resulting fused feature is upsampled to the resolution of the second-layer feature map, and then added to the second-layer feature map. By adding pixels one by one, the two-dimensional features of the enhanced RGB image are obtained. The expression for this process is:

[0009] in, Represents a two-dimensional modal feature enhancement function. This indicates a 2x upsampling. express convolution, Indicates average pooling. Indicates the first Layer feature map.

[0010] Preferably, in step S3, obtaining the multi-view features specifically includes: 10% of the sample point clouds were randomly selected from the training set of the industrial casting dataset. The sample point clouds were then rotated according to an initial rotation matrix set, wherein the initial rotation matrix was derived from the surrounding... Rotation angle of the shaft The rotation angles are respectively taken from... A total of 27 rotation combinations were formed; the rotated point cloud was projected and rendered to obtain 27 multi-view images from different angles; the feature extraction and enhancement methods in step S2 were used to process each multi-view image to obtain the corresponding multi-view features.

[0011] Preferably, in step S3, the process of selecting the optimal view combination through the adaptive view combination pre-selection module includes: Calculate the joint features of the current view combination The mean of the entropy is used to measure the amount of information contained in the current combination of perspectives; Calculate the currently selected Average ensemble mutual information from each perspective Used to measure the current view combination The degree of information redundancy; Calculate the variance of the feature entropy of the combined viewpoints This is used to evaluate the global consistency of the viewpoint combination; A multi-objective optimization model is constructed, and the optimal viewpoint combination is solved through an asymptotic optimization strategy. The expression of the multi-objective optimization model is as follows:

[0012]

[0013] in, Indicates the number of iterations. These are the weight coefficients of the objective function. This is the initial set of viewpoints. The number of viewpoints for the optimal viewpoint combination.

[0014] Preferably, in step S4, the process of fusing the target's multi-view features through the global-local perception fusion module to obtain fused multi-view features includes: Constructing a cosine similarity matrix Elements in the matrix For the first The first perspective and the first The cosine similarity of two perspectives represents the difference between them. Elements in the cosine similarity matrix Mapped to the [0,1] interval; Calculate the first Global consistency weights for multi-perspective features ; Calculate the first The standard deviation of the multi-view features is , No. Local detail weights of multi-view features ; Combined with global consistency weights and local detail weights Aggregating multi-view features ; Finally, the fused multi-view features Two-dimensional features of RGB images Weighted fusion yields the final two-dimensional feature embedding. .

[0015] Preferably, two-dimensional feature embedding The specific formula is as follows:

[0016] in, The number of viewpoints for the optimal viewpoint combination.

[0017] Preferably, in step S5, the calculation processes for obtaining the anomaly score and anomaly segmentation map of the corresponding mode are as follows: Anomaly scores are calculated using the following formula:

[0018] The anomaly segmentation map is calculated using the following formula:

[0019] in, , It is a two-dimensional feature memory. For three-dimensional feature memory, These are local block features of normal samples stored in the memory. To test sample features, To test the local block features of the sample, This is the function for calculating outlier scores. This is a function for calculating anomaly segmentation maps.

[0020] Preferably, step S6 specifically includes: Calculate the local gradients of the two-dimensional and three-dimensional modal anomaly segmentation maps, respectively, and denote them as follows: and ; Calculate the relative rate of change of the local gradient Based on relative rate of change Calculate the weights of the 2D modal anomaly segmentation map The final anomaly segmentation map is obtained by fusing the anomaly segmentation maps from the two-dimensional and three-dimensional modes. According to the abnormal heat map Calculate the pixel-level metrics P-AUROC and AUPRO; take the maximum value of the 2D modal anomaly score and the 3D modal anomaly score as the final anomaly score. According to abnormal scores Calculate the image-level index I-AUROC; analyze the defect detection results using quantitative indices I-AUROC, P-AUROC, AUPRO, and visualization results. Anomaly Segmentation Map and abnormal scores They are respectively:

[0021]

[0022] in, and It is a hyperparameter used to control the proportion of 2D modal decisions. For the sigmoid function, This is a 3D modal anomaly segmentation diagram. This is a two-dimensional modal anomaly segmentation diagram. For three-dimensional modal anomaly scoring, The score is for two-dimensional modal anomalies.

[0023] Secondly, embodiments of the present invention provide a multimodal industrial defect detection system based on adaptive view fusion, comprising: The data module is used to collect industrial casting datasets containing defect-free and defective samples, the datasets including point cloud data and corresponding RGB images; The feature module is used to extract the three-dimensional local geometric features of the point cloud data using fast point feature histograms. The two-dimensional features of the RGB image are extracted using a lightweight MobileViTv3 backbone network. The two-dimensional features of the RGB image are enhanced through the edge detail enhancement module. Enhance; The filtering module is used to perform multi-view projection on point cloud data, obtain multi-view features, and filter out the optimal view combination through the adaptive view combination pre-selection module. The fusion module rotates and projects the sample point cloud according to the rotation angle corresponding to the optimal viewpoint combination to obtain a target multi-view image. It then uses feature extraction and enhancement methods to obtain target multi-view features. A global-local perception fusion module fuses these target multi-view features to obtain fused multi-view features. Finally, it combines these fused multi-view features with the two-dimensional features of the RGB image. Weighted fusion yields the final two-dimensional feature embedding. ; The detection module is used to construct a two-dimensional feature memory and a three-dimensional feature memory, respectively storing the two-dimensional feature embeddings. and three-dimensional local geometric features The corresponding normal sample features are used to perform defect detection in two-dimensional and three-dimensional modes during the testing phase, and the abnormal scores and abnormal segmentation maps of the corresponding modes are obtained. The decision module is used to input the anomaly segmentation map into the local gradient-guided decision fusion module to obtain the final anomaly segmentation map. and abnormal scores Complete industrial defect detection.

[0024] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the above-described multimodal industrial defect detection method based on adaptive view fusion.

[0025] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described multimodal industrial defect detection method based on adaptive view fusion.

[0026] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the above-described multimodal industrial defect detection method based on adaptive view fusion.

[0027] In a sixth aspect, embodiments of the present invention provide an electronic device including a computer program, which, when executed by the electronic device, implements the steps of the above-described multimodal industrial defect detection method based on adaptive view fusion.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects: A multimodal industrial defect detection method based on adaptive view fusion integrates point cloud and RGB image multimodal data, forming a complete detection chain through six steps: feature extraction, viewpoint selection, feature fusion, memory construction, and decision fusion. It overcomes the limitations of single-modality methods by combining the rotation invariance of point clouds with the texture information of RGB images, solving the problems of illumination interference and poor generalization in existing methods. Through adaptive viewpoint selection and a lightweight network, it reduces memory consumption and improves inference speed while ensuring feature integrity, overcoming the pain point of slow inference in multimodal detection. The method connects multiple modules to work collaboratively, optimizing layer by layer from data to decision, achieving precise localization of minute defects, compensating for the shortcomings of traditional methods in missing small defects, and fully meeting the dual requirements of detection accuracy and efficiency in industrial scenarios.

[0029] Furthermore, by using a combined bilateral filter to smooth out RGB image interference textures, the impact of environmental noise on feature extraction is reduced, improving data purity. Unordered point clouds are converted into an ordered format and aligned with image pixels, solving the matching problem between point clouds and image data and laying a data foundation for multimodal fusion. The data size is unified to 224×224 to adapt to the input requirements of the feature extraction network and avoid feature distortion caused by inconsistent sizes. Data diversity is enhanced through translation and rotation, improving the model's generalization ability and enabling the model to perform stably in different industrial scenarios, providing reliable data support for subsequent high-precision detection.

[0030] Furthermore, by combining the feature maps of the second and third layers of the MobileViTv3 backbone network, multi-scale feature fusion is achieved using 1×1 convolution, average pooling, and upsampling operations to solve the problem of losing details of minor defects in two-dimensional feature extraction. Edge features are enhanced by pixel-by-pixel addition, making the features of minor defects such as burrs and flash more prominent, significantly improving the detection sensitivity of small defects. The operation process is simple and efficient, and feature enhancement can be achieved without complex calculations. While improving detection accuracy, it does not add too much extra computational burden, thus balancing accuracy and efficiency.

[0031] Furthermore, based on three rotation angle combinations around the x, y, and z axes, 27 images with different viewpoints are generated to comprehensively cover the spatial information of the point cloud and avoid information blind spots caused by a single viewpoint. 10% of the samples are randomly selected from the training set for viewpoint generation to control the amount of computation while ensuring viewpoint diversity, and to balance data representativeness and efficiency. The feature extraction and enhancement methods in step S2 are reused to ensure that the extraction standards of multi-view features and RGB image features are consistent, reducing fusion interference caused by feature differences, and providing high-quality, homogeneous multi-view feature data for subsequent optimal viewpoint selection.

[0032] Furthermore, a multi-objective optimization model is used to achieve viewpoint selection. A multi-objective optimization model that includes information content, information complementarity, and global uniformity is constructed to comprehensively consider viewpoint quality. The optimal viewpoint combination selected takes into account both feature completeness and redundancy. An incremental optimization strategy is adopted to dynamically balance the weights of each indicator, improve the rationality of the selection results, and make the optimal viewpoint combination suitable for different types of industrial castings. The number of multiple views is reduced to 18, which significantly reduces memory usage and computation while ensuring that the detection accuracy is not reduced, effectively improving the model inference speed and solving the efficiency problem caused by too many views in multimodal detection.

[0033] Furthermore, a cosine similarity matrix is ​​constructed to measure perspective differences. By combining global consistency weights and local detail weights, the commonalities and characteristics of multi-perspective features are fully explored, avoiding the limitations of a single weight mechanism. Cosine similarity is mapped to the [0, 1] interval, making weight calculation more intuitive and reasonable, and improving the accuracy of feature fusion. Through weighted aggregation, the effective fusion of multi-perspective features is achieved, enhancing the distinguishability of features and making the differences between defective and normal features more significant. This provides high-quality fused features for subsequent defect detection and improves the reliability of detection results.

[0034] Furthermore, weights are assigned based on the number of viewpoints K in the optimal viewpoint combination, making the weight allocation more targeted and fully reflecting the importance difference between multi-viewpoint features and RGB features. By weighted fusion, the spatial structure information of multi-viewpoint features and the texture information of RGB images are integrated to improve the comprehensiveness and richness of 2D feature embedding. Without increasing computational complexity, the ability of 2D features to represent defects is enhanced, making the distinction between defects and normal features more obvious, and providing better 2D feature support for subsequent detection.

[0035] Furthermore, the L2 distance metric is used to measure the difference between the features of the test samples and those of normal samples. This approach is logically rigorous, accurately reflects the similarity of features, and generates highly reliable anomaly scores. Anomaly segmentation maps are generated by calculating the minimum distance, enabling pixel-level defect localization and providing refined detection results for subsequent decision fusion. This approach is suitable for real-time detection needs in industrial scenarios, and the high accuracy of the detection results provides accurate and reliable basic data for the decision fusion module, ensuring the accuracy of the final detection results.

[0036] Furthermore, a decision fusion mechanism based on local gradient consistency is proposed, which achieves optimized fusion of 2D and 3D detection results through dynamic weight adjustment. This strategy can effectively combine 2D sensitivity and 3D robustness, improving the accuracy and stability of the final detection results.

[0037] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0038] In summary, this invention constructs an efficient, accurate, and robust industrial defect detection scheme through multimodal feature fusion, adaptive perspective optimization, and gradient-guided decision-making, which significantly outperforms existing methods in terms of accuracy, speed, and adaptability.

[0039] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0040] Figure 1 This is a flowchart of the process framework of the present invention; Figure 2 This is a structural diagram of the Edge-Detail Enhancement Block (EDE-Block). Figure 3 The results of defect detection on industrial castings using M3DM, CPMF, and the present invention; Figure 4 The image shows a magnified view of the local details of M3DM, CPMF, and the present invention for defect detection in industrial castings. Figure 5 The image shows the defect detection results of M3DM, CPMF, and the present invention on MVTec 3D-AD. Figure 6 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 7 This is a block diagram of a chip provided according to an embodiment of the present invention.

[0041] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0044] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0045] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0046] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0047] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0048] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0049] This invention provides a high-speed multi-modal anomaly detection framework with adaptive view fusion for industrial scenes (FastMADet). First, it utilizes FPFH to extract 3D local geometric features from point clouds and solves for the optimal multi-view combination scheme through an adaptive view combination pre-screening module (AVCP-Block). Second, it introduces a lightweight MobileViTv3 backbone and edge detail enhancement module (EDE-Block) to simultaneously extract RGB and multi-view image features. A global-local perception fusion module (GLAF-Block) integrates global information and local details to generate the final 2D feature embedding. Finally, it uses local gradient guided decision fusion (LGDF-Block) to dynamically fuse the local gradient consistency of 2D and 3D modal prediction results, achieving pixel-level defect localization. The defect detection results of self-built industrial castings show that FastMADet can fully combine the detection advantages of two-dimensional and three-dimensional modes to accurately locate defects, while improving the inference speed compared with similar multimodal detection methods. In addition, the invention demonstrates excellent generalization performance on the MVTec-3D AD public dataset, significantly improving the accuracy, efficiency and robustness of small defect detection in industrial scenarios.

[0050] In the feature extraction stage, 3D local geometric features are extracted from point cloud data and projected to obtain multi-view images. An Adaptive View Combination Pre-selection Block (AVCP-Block) is used to reduce information redundancy and avoid introducing low-quality view features. A Global-Local Aware Fusion Block (GLAF-Block) is employed to mine the commonalities and characteristics of multi-view images and fuse multi-view features with RGB image features. Furthermore, a lightweight 2D feature extractor is introduced to improve detection efficiency and reduce memory overhead; and an Edge-Detail Enhancer Block (EDE-Block) is proposed to enhance the feature representation capabilities of small regions and edges in the 2D modality. In the defect detection stage, a Local Gradient-guided Decision Fusion Block (LGDF-Block) is proposed to combine the high sensitivity of 2D detection with the robustness of 3D detection for more accurate defect localization. Finally, the method of this invention was applied to the MVTec3D-AD public dataset and the self-built industrial casting data. The results of both the public dataset and the self-built industrial casting data demonstrated the good defect detection performance of the FastMADet network.

[0051] Please see Figure 1 and Figure 2 This invention discloses a multimodal industrial defect detection method based on adaptive view fusion, comprising the following steps: S1. An industrial casting dataset was acquired using a structured light camera. Each sample in the dataset includes a point cloud obtained by sensor scanning and its corresponding RGB image. 90% of the defect-free workpiece data was randomly selected as the training set, and the remaining defect-free and defective workpiece data were mixed as the test set. Data for each category in the dataset were trained and tested independently. In addition, the MVTec 3D-AD public dataset was used for experimental evaluation to verify the generalization of the invention. The training set contains only defect-free normal samples, while the test set contains both normal and defective samples. The model was trained and tested on a self-built industrial casting dataset and the publicly available dataset MVTec 3D-AD to validate its defect detection and generalization performance. MVTec 3D-AD is the first publicly available 3D industrial anomaly detection dataset (https: / / www.mvtec.com / company / research / datasets / mvtec-3d-ad), covering multiple categories and defect types, including 10 categories such as bagels, carrots, potatoes, tires, cables, and foam, with a total of 2565 training samples and 1137 test samples. Each sample includes a high-resolution depth image and its corresponding RGB image. The self-built industrial casting dataset was obtained by sampling large castings using a Mech-Eye PRO structured light camera, directly acquiring high-density point clouds and images from corresponding viewpoints. It covers three categories of industrial castings, containing 275 training samples and 90 test samples. These industrial castings are approximately 1 meter in diameter, and their main defects include burrs, flash, and cutting residue. Each sample in the data includes a point cloud obtained from sensor scanning, corresponding to an RGB image. 90% of the defect-free workpiece data was randomly selected as the training set, and the remaining defect-free and defective workpiece data were mixed as the test set.

[0052] For the MVTEC 3D-AD dataset, the RANSAC algorithm is used to estimate the background plane in the point cloud and remove it; simultaneously, the pixel values ​​corresponding to the removed points in the corresponding RGB image are set to 0. The position tensor and RGB image are uniformly scaled to 224×224 to match the input size of the feature extraction network. For the industrial casting dataset, a joint bilateral filtering method is used to smooth the interfering textures in the RGB image. Furthermore, the unordered point cloud is converted to an ordered format and aligned with the pixel positions of the corresponding image; the position information of the point cloud is stored... In the three-channel tensor of orientation coordinates, LabelMe is used to label outlier regions of the samples, generating ground truth to verify the detection accuracy of the method. Since the workpiece occupies only a small portion of the original acquired image, the position tensor, RGB image, and ground truth map are simultaneously cropped to extract a square region covering the workpiece, and uniformly scaled to 224×224. Finally, data augmentation is performed through translation and rotation, resulting in 635 training samples and 217 test samples.

[0053] S2. Extract sample point clouds using Fast Point Feature Histograms (FPFH). Local geometric features ; Sample RGB image Input a pre-trained MobileViT V3 Transformer to extract image features. Furthermore, the Edge-DetailEnhancer Block (EDE-Block) is used to enhance edge and small region features in the two-dimensional modality. The input point cloud is extracted using Fast Point Feature Histogram (FPFH). Three-dimensional local geometric features And construct a two-dimensional modal feature extraction and enhancement process applicable to the entire invention; and extract sample RGB images. Input MobileViTv3-XS and obtain the feature maps of the second and third layers. Then input the data into the EDE module to process the second layer of features. Perform average pooling, third layer features through After convolution, the features are added pixel-by-pixel to the pooling features to achieve multi-scale fusion; then the fused features are upsampled to the second-layer resolution and combined with the second-layer features. Pixel-by-pixel addition enhances the representation of edges and small regions, yielding the final RGB features. This enhancement process can be represented as:

[0054] in, Represents a two-dimensional modal feature enhancement function. This indicates a 2x upsampling. express convolution, Indicates average pooling. Indicates the first Layer feature map .

[0055] S3. For each category in the dataset, randomly select 10% of the training samples and generate the sample point cloud. Rotation and projection were performed to obtain 27 different views. The 2D feature extraction and enhancement methods in S2 were then used to obtain the multi-view features. Then, the optimal view combination for this category is calculated based on the Adaptive View Combination Pre-selection Block (AVCP-Block). ; Randomly select 10% of the training samples, and rotate the point cloud according to the initial rotation matrix set. The image is rotated, then projected and rendered to generate a multi-view image. The rotation matrix is ​​used for this process. By wrapping Rotation angle of the shaft Composition, initial settings Taken from respectively There are 27 possible combinations. For the... The multi-view images generated from the sample point clouds are used to obtain multi-view features through the two-dimensional modal feature extraction and enhancement process in S2. Then, the optimal view combination for this category is calculated using the AVCP module. Reduce the number of multi-views to Zhang minimizes information redundancy while ensuring the integrity and quality of sample features.

[0056] A multi-objective programming model is constructed in the Adaptive View Combination Preselection Module (AVCP). The objective function consists of three parts: view information content, information complementarity, and global uniformity.

[0057] S301, Calculate the current viewpoint combination joint features The mean of the entropy is used to measure the amount of information contained in the current viewpoint combination. The specific formula is:

[0058] in, It is the entropy function. This represents the number of randomly selected samples.

[0059] S302, Calculate the currently selected... Average ensemble mutual information from each perspective Used to measure the current view combination The degree of information redundancy is calculated using the following formula:

[0060] in, Representation of features and characteristics The joint entropy.

[0061] S303, Calculate the variance of the feature entropy of the viewpoint combination. The formula used to evaluate the global consistency of viewpoint combinations is as follows:

[0062] in, Indicates the first The set consisting of the feature entropy of each viewpoint in each sample.

[0063] Finally, the multi-objective optimization model is obtained:

[0064]

[0065] in, Indicates the number of iterations. The weight coefficients of the objective function ( i =1,2,3).

[0066] In addition, a gradual optimization strategy is adopted during the optimization process to dynamically balance the influence between various indicators.

[0067]

[0068] in, This is a hyperparameter for controlling the decay of the coefficient. For the initial value ( i =1,2,3). Subsequent training and testing steps only generate the values ​​corresponding to the optimal viewpoint combination. Multiple views (default in this invention) (Using 18), thus significantly reducing memory usage and improving detection efficiency.

[0069] S4. Transfer the sample point cloud Based on the calculation obtained in step S3 The system performs rotation and projection rendering at various rotation angles to obtain multiple views, and then uses the two-dimensional feature extraction and enhancement methods in step S2 to obtain multi-view features. The multi-view features are input into the Global-Local AwareFusion Block (GLAF-Block) to obtain fused multi-view features. Then, combined with RGB image features Weighted fusion yields the final two-dimensional feature embedding. ; The GLAF module consists of two weighting mechanisms: global consistency weights and local detail weights, which dynamically evaluate each sample. The contribution of multiple perspective features effectively integrates global information and local detailed features.

[0070] S401. Constructing the cosine similarity matrix Elements in the matrix For the first The first perspective and the first The cosine similarity of two viewpoints represents the difference between them, and the specific formula is as follows:

[0071] S402, Through Mapped to interval; S403, Calculate the first Global consistency weights for multi-perspective features The specific formula is as follows:

[0072] S404, No. The standard deviation of the multi-view features is , No. Local detail weights of multi-view features The specific formula is as follows:

[0073] S405, Combining Global Consistency Weights and local detail weights The formula for aggregating multi-view features is as follows:

[0074] Finally, the fused multi-view features With RGB features Weighted fusion yields the final two-dimensional feature embedding. The specific formula is as follows:

[0076] S5. During the training phase, construct two-dimensional and three-dimensional feature memory databases respectively. Store the normal sample features corresponding to the modality; during the testing phase, input the test features. Defect detection was performed in both two-dimensional and three-dimensional modes, and anomaly scores were obtained for the corresponding modes. and anomaly segmentation map ; The scoring function is as follows:

[0077]

[0078] in, , These are local patch features of normal samples stored in the memory. } represents the features of the test samples. To test the local block features of the sample.

[0079] S6. The segmentation image obtained in step S5 The final anomaly segmentation map is obtained by inputting the Local Gradient-guided Decision Fusion Block (LGDF-Block). and abnormal scores ; Calculate the local gradients of the anomaly heatmaps for the two modes obtained in step S5. Then calculate the relative rate of change of the local gradient. The specific formula is as follows:

[0080] in, It is an extremely small constant.

[0081] Finally, based on the relative rate of change The anomaly heatmaps of the two modes are fused to obtain the final anomaly heatmap. Furthermore, the maximum anomaly score between the two modalities is taken as the final anomaly score. The specific formula is as follows:

[0082]

[0083]

[0084] in, and It is a hyperparameter used to control the proportion of 2D modal decisions. This is the sigmoid function.

[0085] S7. Apply the model to a self-built industrial casting dataset and a public dataset for defect detection, and analyze the defect detection results through quantitative indicators: I-AUROC, P-AUROC, AUPRO and visualization results, thereby demonstrating the network's defect detection performance.

[0086] To evaluate the defect detection performance of this invention, I-AUROC (Image Level Area Under the Receiver Operating Characteristic Curve), P-AUROC (Pixel Level Area Under the Receiver Operating Characteristic Curve), and AUPRO (Area Under the Per-Region-Overlap Curve) are selected as quantitative indicators to measure the defect detection performance of the model. The specific calculation steps for the indicators are as follows: S701. Based on the anomaly score calculated in step S6 Calculate the image-level metric I-AUROC. The specific formula is as follows:

[0087] in The True Positive Rate (TPR) is the percentage of positive positivity. This is the false positive rate (FPR).

[0088] S702. Obtain the abnormal heat map based on step S6. Calculate the pixel-level metric P-AUROC using the same steps as above.

[0089] S703. Obtain the abnormal heat map based on step S6. Calculate the AUPRO metric.

[0090] The abnormal heatmap in the prediction results is converted into a binary image according to a set threshold, and connected real abnormal pixel regions are labeled as connected components. Then, the proportion of the intersection between the prediction results and each real label in the ground truth is calculated, and the average is taken to obtain the PRO score at that threshold. The PRO calculation formula is:

[0091] in, This represents the number of abnormal regions in the sample. For the predicted anomaly region, For the first n A real abnormal connected domain.

[0092] S704. To avoid the influence of the threshold on the evaluation results, the PRO score is calculated under different FPRs, and the area under the PRO curve (AUPRO) is calculated by selecting a threshold between 0 and 30% of the FPR.

[0093] The visualization results are presented by comparing the defect detection results of the model of this invention on the MVTec 3D-AD dataset and the industrial casting dataset.

[0094] In another embodiment of the present invention, a multimodal industrial defect detection system based on adaptive view fusion is provided. This system can be used to implement the above-mentioned multimodal industrial defect detection method based on adaptive view fusion. Specifically, the multimodal industrial defect detection system based on adaptive view fusion includes a data module, a feature module, a filtering module, a fusion module, a detection module, and a decision module.

[0095] The data module is used to collect industrial casting datasets containing defect-free and defective samples. The datasets include point cloud data and corresponding RGB images. The feature module is used to extract the three-dimensional local geometric features of the point cloud data using fast point feature histograms. The two-dimensional features of the RGB image are extracted using a lightweight MobileViTv3 backbone network. The two-dimensional features of the RGB image are enhanced through the edge detail enhancement module. Enhance; The filtering module is used to perform multi-view projection on point cloud data, obtain multi-view features, and filter out the optimal view combination through the adaptive view combination pre-selection module. The fusion module rotates and projects the sample point cloud according to the rotation angle corresponding to the optimal viewpoint combination to obtain a target multi-view image. It then uses feature extraction and enhancement methods to obtain target multi-view features. A global-local perception fusion module fuses these target multi-view features to obtain fused multi-view features. Finally, it combines these fused multi-view features with the two-dimensional features of the RGB image. Weighted fusion yields the final two-dimensional feature embedding. ; The detection module is used to construct a two-dimensional feature memory and a three-dimensional feature memory, respectively storing the two-dimensional feature embeddings. and three-dimensional local geometric features The corresponding normal sample features are used to perform defect detection in two-dimensional and three-dimensional modes during the testing phase, and the abnormal scores and abnormal segmentation maps of the corresponding modes are obtained. The decision module is used to input the anomaly segmentation map into the local gradient-guided decision fusion module to obtain the final anomaly segmentation map. and abnormal scores Complete industrial defect detection.

[0096] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used in the operation of a multimodal industrial defect detection method based on adaptive view fusion, including: A dataset of industrial castings containing both defect-free and defective samples was collected. This dataset includes point cloud data and corresponding RGB images. Three-dimensional local geometric features of the point cloud data were extracted using a fast point feature histogram. The two-dimensional features of the RGB image are extracted using a lightweight MobileViTv3 backbone network. The two-dimensional features of the RGB image are enhanced through the edge detail enhancement module. Enhancement is performed; multi-view projection is performed on the point cloud data to obtain multi-view features, and the optimal view combination is selected through an adaptive view combination pre-selection module; the sample point cloud is rotated and projected according to the rotation angle corresponding to the optimal view combination to obtain the target multi-view image, and the target multi-view features are obtained by feature extraction and enhancement methods; the target multi-view features are fused through a global-local perception fusion module to obtain fused multi-view features; the fused multi-view features are then combined with the two-dimensional features of the RGB image. Weighted fusion yields the final two-dimensional feature embedding. Construct a two-dimensional feature memory and a three-dimensional feature memory to store the two-dimensional feature embeddings, respectively. and three-dimensional local geometric features The corresponding normal sample features are used for defect detection in two-dimensional and three-dimensional modalities during the testing phase to obtain the anomaly score and anomaly segmentation map for the corresponding modality. The anomaly segmentation map is then input into the local gradient-guided decision fusion module to obtain the final anomaly segmentation map. and abnormal scores Complete industrial defect detection.

[0097] Please see Figure 6 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the multimodal industrial defect detection method based on adaptive view fusion described in this embodiment. To avoid repetition, details are omitted here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the multimodal industrial defect detection system based on adaptive view fusion described in this embodiment. To avoid repetition, details are omitted here.

[0098] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 6 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0099] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0100] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device 60.

[0101] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0102] Please see Figure 7 The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0103] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0104] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0105] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0106] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0107] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0108] Example 4 This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0109] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0110] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0111] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the multimodal industrial defect detection method based on adaptive view fusion in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps: A dataset of industrial castings containing both defect-free and defective samples was collected. This dataset includes point cloud data and corresponding RGB images. Three-dimensional local geometric features of the point cloud data were extracted using a fast point feature histogram. The two-dimensional features of the RGB image are extracted using a lightweight MobileViTv3 backbone network. The two-dimensional features of the RGB image are enhanced through the edge detail enhancement module. Enhancement is performed; multi-view projection is performed on the point cloud data to obtain multi-view features, and the optimal view combination is selected through an adaptive view combination pre-selection module; the sample point cloud is rotated and projected according to the rotation angle corresponding to the optimal view combination to obtain the target multi-view image, and the target multi-view features are obtained by feature extraction and enhancement methods; the target multi-view features are fused through a global-local perception fusion module to obtain fused multi-view features; the fused multi-view features are then combined with the two-dimensional features of the RGB image. Weighted fusion yields the final two-dimensional feature embedding. Construct a two-dimensional feature memory and a three-dimensional feature memory to store the two-dimensional feature embeddings, respectively. and three-dimensional local geometric features The corresponding normal sample features are used for defect detection in two-dimensional and three-dimensional modalities during the testing phase to obtain the anomaly score and anomaly segmentation map for the corresponding modality. The anomaly segmentation map is then input into the local gradient-guided decision fusion module to obtain the final anomaly segmentation map. and abnormal scores Complete industrial defect detection.

[0112] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0113] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0114] Please see Figure 3 and Figure 4 , Figure 3The results show the detection performance of three different multimodal defect detection algorithms on an industrial casting dataset. Columns 3 and 4, and columns 5 and 6 respectively show the visualization results of the M3DM and CPMF algorithms on the industrial casting dataset. Columns 7 and 8 show the visualization results of the detection using the model of this invention. The results show that the detection performance of the model of this invention on the industrial parts dataset is better than that of the M3DM and CPMF methods. Figure 4 The results show the detection of cutting residue on the outer side of the casting lifting lug. Columns 2 and 3 show the detection results of the M3DM and CPMF algorithms, respectively. Both methods identify the lifting lug and cutting residue as defects. Column 4 shows the detection results of the model of this invention, which can effectively identify the real defects, indicating that the model of this invention has significant advantages in the identification and location of small defects.

[0115] Please see Figure 5 , Figure 5 The results show the detection performance of three different multimodal defect detection algorithms on the MVTec 3D-AD public dataset. Columns 3 and 4, and columns 5 and 6 respectively show the visualization results of the M3DM and CPMF algorithms on MVTec 3D-AD. Columns 7 and 8 show the visualization results of the detection using the model of this invention. The results show that the detection performance of the model of this invention on the MVTec 3D-AD dataset is better than the other two methods, M3DM and CPMF.

[0116] Table 1 shows the results of the ablation experiment of this invention.

[0117] Please refer to Table 1, which shows the results of the ablation experiment of this invention. Specifically, Table 1 shows the number of multiple views. The performance of three models on an industrial casting dataset was compared: baseline models with 27 and 18 views, models with the GLAF module added, and models with both AVCP and GLAF modules added. The ablation results show that the GLAF module effectively improves defect detection accuracy on the industrial casting dataset; the AVCP module effectively improves the quality of multi-view features, resulting in better defect detection performance with 18 views than with 27 views, demonstrating the effectiveness of the AVCP module.

[0118] Table 2 illustrates the computational complexity of this invention.

[0119] Please refer to Table 2, which illustrates the computational complexity of this invention. Specifically, Table 2 shows the inference speed, memory usage, and defect detection accuracy of M3DM, CPMF, and the model of this invention on three devices with different performance levels: RTX 3090, RTX 3060, and GTX 1080Ti. The results show that this invention can achieve excellent detection performance and maintain its detection speed advantage on devices with different computing power.

[0120] In summary, this invention presents a multimodal industrial defect detection method and system based on adaptive view fusion. By fusing 3D point cloud structural information with 2D RGB texture features and combining viewpoint optimization and decision fusion mechanisms, it effectively addresses the shortcomings of traditional methods in detecting minute defects, adapting to complex working conditions, and inference speed. Experimental results show that this method exhibits superior performance on both a self-built industrial casting dataset and the publicly available MVTec 3D-AD dataset, particularly demonstrating significant advantages in locating edge defects and small-area anomalies. Furthermore, the lightweight network structure and adaptive computation strategy employed significantly improve detection efficiency while maintaining high accuracy, meeting the real-time requirements of industrial environments. This invention possesses strong generalization ability and engineering applicability, providing a new technical path for industrial quality inspection.

[0121] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0124] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0128] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A multimodal industrial defect detection method based on adaptive view fusion, characterized in that, Includes the following steps: S1. Collect an industrial casting dataset containing defect-free and defective samples, the dataset including point cloud data and corresponding RGB images; S2. Extract the three-dimensional local geometric features of the point cloud data using fast point feature histogram. The two-dimensional features of the RGB image are extracted using a lightweight MobileViTv3 backbone network. The two-dimensional features of the RGB image are enhanced through the edge detail enhancement module. Enhance; S3. Perform multi-view projection on the point cloud data to obtain multi-view features, and filter out the optimal view combination through the adaptive view combination pre-selection module. S4. Rotate and project the sample point cloud according to the rotation angle corresponding to the optimal viewpoint combination to obtain a target multi-view image. Use the feature extraction and enhancement method in step S2 to obtain target multi-view features. Fuse the target multi-view features through a global-local perception fusion module to obtain fused multi-view features. Combine the fused multi-view features with the two-dimensional features of the RGB image. Weighted fusion yields the final two-dimensional feature embedding. ; S5. Construct a two-dimensional feature memory and a three-dimensional feature memory to store the two-dimensional feature embeddings, respectively. and three-dimensional local geometric features The corresponding normal sample features are used to perform defect detection in two-dimensional and three-dimensional modes during the testing phase, and the abnormal scores and abnormal segmentation maps of the corresponding modes are obtained. S6. Input the anomaly segmentation map into the local gradient-guided decision fusion module to obtain the final anomaly segmentation map. and abnormal scores Complete industrial defect detection.

2. The multimodal industrial defect detection method based on adaptive view fusion according to claim 1, characterized in that, Step S1 also includes preprocessing operations on the industrial casting dataset: The RGB image is smoothed by a joint bilateral filtering method to remove interfering textures; the disordered point cloud data is converted into an ordered format and aligned with the pixel positions of the corresponding RGB image; the position information of the point cloud data is stored as a three-channel tensor with x, y, and z coordinates; the three-channel tensor, the RGB image, and the reference truth map are simultaneously cropped to extract a square area covering the industrial casting and uniformly scaled to 224×224. Data augmentation is performed through translation and rotation operations.

3. The multimodal industrial defect detection method based on adaptive view fusion according to claim 1, characterized in that, The two-dimensional features of the RGB image described in step S2 are enhanced by the edge detail enhancement module. The specific process of enhancement is as follows: Obtain the second-layer feature map output from the MobileViTv3 backbone network. and the third layer feature map For the second layer feature map Perform average pooling on the third layer feature map. After performing 1×1 convolution, it is added pixel by pixel to the second layer feature map after pooling. The fused features obtained by addition are upsampled to the resolution of the second-layer feature map, and then combined with the second-layer feature map. By adding pixels one by one, the two-dimensional features of the enhanced RGB image are obtained. The expression for this process is: in, Represents a two-dimensional modal feature enhancement function. This indicates a 2x upsampling. express convolution, Indicates average pooling. Indicates the first Layer feature map.

4. The multimodal industrial defect detection method based on adaptive view fusion according to claim 1, characterized in that, Step S3, obtaining the multi-view features specifically includes: 10% of the sample point clouds were randomly selected from the training set of the industrial casting dataset. The sample point clouds were then rotated according to an initial rotation matrix set, wherein the initial rotation matrix was derived from the surrounding... Rotation angle of the shaft The rotation angles are respectively taken from... A total of 27 rotation combinations were formed; the rotated point cloud was projected and rendered to obtain 27 multi-view images from different angles; the feature extraction and enhancement methods in step S2 were used to process each multi-view image to obtain the corresponding multi-view features.

5. The multimodal industrial defect detection method based on adaptive view fusion according to claim 1, characterized in that, In step S3, the process of selecting the optimal view combination through the adaptive view combination pre-selection module includes: Calculate the joint features of the current view combination The mean of the entropy is used to measure the amount of information contained in the current combination of perspectives; Calculate the currently selected Average ensemble mutual information from each perspective Used to measure the current view combination The degree of information redundancy; Calculate the variance of the feature entropy of the combined viewpoints This is used to evaluate the global consistency of the viewpoint combination; A multi-objective optimization model is constructed, and the optimal viewpoint combination is solved through an asymptotic optimization strategy. The expression of the multi-objective optimization model is as follows: in, Indicates the number of iterations. These are the weight coefficients of the objective function. This is the initial set of viewpoints. The number of viewpoints for the optimal viewpoint combination.

6. The multimodal industrial defect detection method based on adaptive view fusion according to claim 1, characterized in that, In step S4, the process of fusing the multi-view features of the target through the global-local perception fusion module to obtain fused multi-view features includes: Constructing a cosine similarity matrix Elements in the matrix For the first The first perspective and the first The cosine similarity of two perspectives represents the difference between them. Elements in the cosine similarity matrix Mapped to the [0,1] interval; Calculate the first Global consistency weights for multi-perspective features ; Calculate the first The standard deviation of the multi-view features is , No. Local detail weights of multi-view features ; Combined with global consistency weights and local detail weights Aggregating multi-view features ; Finally, the fused multi-view features Two-dimensional features of RGB images Weighted fusion yields the final two-dimensional feature embedding. .

7. The multimodal industrial defect detection method based on adaptive view fusion according to claim 6, characterized in that, Two-dimensional feature embedding The specific formula is as follows: in, The number of viewpoints for the optimal viewpoint combination.

8. The multimodal industrial defect detection method based on adaptive view fusion according to claim 1, characterized in that, In step S5, the calculation processes for obtaining the anomaly score and anomaly segmentation map for the corresponding modality are as follows: Anomaly scores are calculated using the following formula: The anomaly segmentation map is calculated using the following formula: in, , It is a two-dimensional feature memory. For three-dimensional feature memory, These are local block features of normal samples stored in the memory. To test sample features, To test the local block features of the sample, This is the function for calculating outlier scores. This is a function for calculating anomaly segmentation maps.

9. The multimodal industrial defect detection method based on adaptive view fusion according to claim 1, characterized in that, Step S6 is as follows: Calculate the local gradients of the two-dimensional and three-dimensional modal anomaly segmentation maps, respectively, and denote them as follows: and ; Calculate the relative rate of change of the local gradient. Based on relative rate of change Calculate the weights of the 2D modal anomaly segmentation map The final anomaly segmentation map is obtained by fusing the anomaly segmentation maps from the two-dimensional and three-dimensional modes. According to the abnormal heat map Calculate pixel-level metrics P-AUROC and AUPRO; The maximum value between the two-dimensional modal anomaly score and the three-dimensional modal anomaly score is taken as the final anomaly score. According to abnormal scores Calculate the image-level index I-AUROC; analyze the defect detection results using quantitative indices I-AUROC, P-AUROC, AUPRO, and visualization results. Anomaly Segmentation Map and abnormal scores They are respectively: in, and It is a hyperparameter used to control the proportion of 2D modal decisions. For the sigmoid function, This is a 3D modal anomaly segmentation diagram. This is a two-dimensional modal anomaly segmentation diagram. For three-dimensional modal anomaly scoring, The score is for two-dimensional modal anomalies.

10. A multimodal industrial defect detection system based on adaptive view fusion, characterized in that, include: The data module is used to collect industrial casting datasets containing defect-free and defective samples, the datasets including point cloud data and corresponding RGB images; The feature module is used to extract the three-dimensional local geometric features of the point cloud data using fast point feature histograms. The two-dimensional features of the RGB image are extracted using a lightweight MobileViTv3 backbone network. The two-dimensional features of the RGB image are enhanced through the edge detail enhancement module. Enhance; The filtering module is used to perform multi-view projection on point cloud data, obtain multi-view features, and filter out the optimal view combination through the adaptive view combination pre-selection module. The fusion module rotates and projects the sample point cloud according to the rotation angle corresponding to the optimal viewpoint combination to obtain a target multi-view image. It then uses feature extraction and enhancement methods to obtain target multi-view features. A global-local perception fusion module fuses these target multi-view features to obtain fused multi-view features. Finally, it combines these fused multi-view features with the two-dimensional features of the RGB image. Weighted fusion yields the final two-dimensional feature embedding. ; The detection module is used to construct a two-dimensional feature memory and a three-dimensional feature memory, respectively storing the two-dimensional feature embeddings. and three-dimensional local geometric features The corresponding normal sample features are used to perform defect detection in two-dimensional and three-dimensional modes during the testing phase, and the abnormal scores and abnormal segmentation maps of the corresponding modes are obtained. The decision module is used to input the anomaly segmentation map into the local gradient-guided decision fusion module to obtain the final anomaly segmentation map. and abnormal scores Complete industrial defect detection.

Images