Three-dimensional industrial anomaly detection method and device, storage medium and electronic equipment

Abstract

The application belongs to the technical field of anomaly detection, and relates to a three-dimensional industrial anomaly detection method, device, storage medium and electronic equipment, comprising the following steps: acquiring an RGB image and a point cloud image corresponding to a to-be-detected object, preprocessing the point cloud image to obtain a plurality of point cloud groups; inputting the point cloud groups into a first feature extractor to obtain a point cloud feature group, generating a point cloud feature map according to the point cloud feature group and the point cloud image, and projecting the point cloud feature map to form a point cloud projection map with the same size as the RGB image; inputting the RGB image into a second feature extractor to obtain an RGB feature; inputting the RGB feature and the point cloud feature in the point cloud projection map into a multi-layer perceptron respectively, fusing feature information output by the multi-layer perceptron to obtain fused features; and performing anomaly prediction on the to-be-detected object according to the RGB feature, the point cloud feature and the fused features to obtain an anomaly score and an anomaly representation map. The application can perform preliminary screening on industrial anomalies and reduce the labor cost of industrial quality inspection.

Description

Technical Field

[0001] This application belongs to the field of anomaly detection technology, specifically relating to a three-dimensional industrial anomaly detection method, a three-dimensional industrial anomaly detection device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Industrial anomaly detection aims to identify abnormal areas in products and plays an important role in industrial quality inspection.

[0003] In the quality inspection of industrial products, inspectors use both three-dimensional shape and color features to determine whether a product is defective. Three-dimensional shape information is very important and necessary for accurate inspection. However, most existing industrial anomaly detection methods are based on two-dimensional images. Multimodal industrial anomaly detection based on three-dimensional point clouds and RGB images still has many untouched areas, which leads to poor detection performance and poor accuracy of detection results. Summary of the Invention

[0004] The purpose of this application is to provide a three-dimensional industrial anomaly detection method, a three-dimensional industrial anomaly detection device, a computer-readable storage medium, and an electronic device, which can overcome the problems of poor detection performance and poor accuracy of detection results in multimodal industrial anomaly detection in related technologies.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of the embodiments of this application, a three-dimensional industrial anomaly detection method is provided. The method includes: acquiring an RGB image and a point cloud map corresponding to an object to be detected; preprocessing the point cloud map to obtain multiple point cloud groups; inputting the point cloud groups into a first feature extractor for feature extraction to obtain point cloud feature groups; generating a point cloud feature map based on the point cloud feature groups and the point cloud map; and projecting the point cloud feature map to form a point cloud projection map of the same size as the RGB image; inputting the RGB image into a second feature extractor for feature extraction to obtain RGB features; inputting the RGB features and the point cloud features in the point cloud projection map into a multilayer perceptron for feature extraction, and fusing the feature information output by the multilayer perceptron to obtain fused features; and performing anomaly prediction on the object to be detected based on the RGB features, the point cloud features, and the fused features to obtain an anomaly score and an anomaly representation map corresponding to the object to be detected.

[0007] According to one aspect of the embodiments of this application, a three-dimensional industrial anomaly detection device is provided. The device includes: a preprocessing module, configured to acquire an RGB image and a point cloud image corresponding to an object to be detected, and preprocess the point cloud image to obtain multiple point cloud groups; a registration module, configured to input the point cloud groups to a first feature extractor for feature extraction to obtain point cloud feature groups, generate a point cloud feature map based on the point cloud feature groups and the point cloud image, and project the point cloud feature map to form a point cloud projection map of the same size as the RGB image; input the RGB image to a second feature extractor for feature extraction to obtain RGB features; a fusion module, configured to input the RGB features and the point cloud features in the point cloud projection map to a multilayer perceptron for feature extraction, and fuse the feature information output by the multilayer perceptron to obtain fused features; and a decision module, configured to predict anomalies in the object to be detected based on the RGB features, the point cloud features, and the fused features to obtain an anomaly score and an anomaly representation map corresponding to the object to be detected.

[0008] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional industrial anomaly detection method as described above.

[0009] According to one aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform a three-dimensional industrial anomaly detection method as described above by executing the executable instructions.

[0010] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions, which, when executed on a computer, cause the computer to perform the three-dimensional industrial anomaly detection method as described in the above technical solutions.

[0011] The three-dimensional industrial anomaly detection method provided in this application involves acquiring an RGB image and a point cloud map corresponding to the object to be detected. First, the point cloud map is preprocessed to obtain multiple point cloud groups. Then, the point cloud groups are input to a first feature extractor for feature extraction to obtain point cloud feature groups. A point cloud feature map is generated based on the point cloud feature groups and the point cloud map, and the point cloud feature map is projected to form a point cloud projection map of the same size as the RGB image. Simultaneously, the RGB image is input to a second feature extractor for feature extraction to obtain RGB features. Then, the RGB features and the point cloud features in the point cloud projection map are respectively input to a multilayer perceptron for feature extraction, and the feature information output by the multilayer perceptron is fused to obtain fused features. Finally, based on the RGB features, point cloud features, and fused features, anomaly prediction is performed on the object to be detected to obtain an anomaly score and anomaly representation map corresponding to the object to be detected. This application enables multimodal industrial anomaly detection based on 3D point clouds and RGB images. By fusing RGB and point cloud features, it improves the interactivity between features and reduces mutual interference between high-dimensional features. Furthermore, it predicts anomalies in the target object based on various data features stored in the memory bank, RGB features, point cloud features, and fused features, obtaining anomaly scores and anomaly representation maps, thus improving the accuracy of anomaly detection results. Moreover, this application adopts an unsupervised approach, eliminating the need for manual annotation before anomaly detection, reducing manual annotation costs. Using this technical solution for initial screening of industrial anomalies can also reduce manual quality inspection costs.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0014] Figure 1 A schematic diagram of the system architecture for applying the three-dimensional industrial anomaly detection method in the embodiments of this application is shown.

[0015] Figure 2 The schematic diagram illustrates the steps of the three-dimensional industrial anomaly detection method in the embodiments of this application.

[0016] Figure 3 The schematic diagram illustrates the process of generating a point cloud projection map in an embodiment of this application.

[0017] Figure 4 A schematic diagram of the structure of the three-dimensional industrial anomaly detection system in an embodiment of this application is shown.

[0018] Figure 5 The schematic diagram illustrates the process of using a three-dimensional industrial anomaly detection system to detect anomalies in cable sealing sleeves in an embodiment of this application.

[0019] Figure 6 The illustration shows the anomaly characterization map and anomaly location map determined after anomaly detection of various objects to be detected in the embodiments of this application.

[0020] Figure 7 The schematic diagram illustrates the process flow of the unsupervised training feature fusion module in an embodiment of this application.

[0021] Figure 8 A schematic diagram of the structure of the three-dimensional industrial anomaly detection device in an embodiment of this application is shown.

[0022] Figure 9 A schematic diagram of a computer system architecture suitable for implementing the embodiments of this application is shown. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0024] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0027] In the related technologies of this application, it is easy to obtain a large number of normal instances in industrial scenarios, but very few defective instances can be obtained. Most current industrial anomaly detection methods are unsupervised methods, that is, they are only trained on normal instances and tested only on detected instances during the inference process.

[0028] Most existing industrial anomaly detection methods are based on two-dimensional images, and multimodal industrial anomaly detection based on three-dimensional point clouds and RGB images still has many unexplored areas. However, in the quality inspection of industrial products, inspectors simultaneously utilize three-dimensional shape and color features to determine whether a product is defective, with three-dimensional shape information being crucial and necessary for accurate detection. Even though some multimodal industrial anomaly detection methods exist, these methods directly connect multimodal features. When the feature dimensionality is high, this can lead to strong interference between features, impairing detection performance.

[0029] The core idea of ​​unsupervised anomaly detection is to identify the differences between anomaly and normal representations. Currently, industrial anomaly detection methods can be divided into two categories: reconstruction-based methods and pre-trained feature extractor-based methods. When using reconstruction-based methods for industrial anomaly detection, they are easy to implement for single-modal inputs (2D images or 3D point clouds), but for multimodal inputs, it is difficult to find a reconstructed target. When using pre-trained feature extractor-based methods for industrial anomaly detection, the extracted features are mapped onto a normal distribution, and the feature that deviates from the distribution is identified as an anomaly. Compared to reconstruction-based methods, pre-trained feature extractors do not require retraining, thus avoiding the design of multimodal reconstruction targets and making them a better choice for multimodal tasks.

[0030] In view of the relevant technologies in this field, this application proposes a three-dimensional industrial anomaly detection method, which belongs to the multimodal industrial anomaly detection method. In this application, a pre-trained feature extractor is used for industrial anomaly detection to improve the accuracy of industrial anomaly detection and anomaly area location.

[0031] Before providing a detailed explanation of the three-dimensional industrial anomaly detection method in this application, the technical terms that may be involved in this application will be explained first.

[0032] 1. Anomaly detection: In data mining, anomaly detection refers to the identification of items, events, or observations that do not match expected patterns or other items in the dataset.

[0033] 2. Point cloud: A collection of point data on the surface of a product obtained through measuring instruments in reverse engineering.

[0034] 3. Farthest Point Sampling: This is a commonly used sampling algorithm that uniformly samples M points from a point cloud of N points, so that these points can better represent the overall contour of the point cloud.

[0035] 4. Inverse Distance Weighted Interpolation: A commonly used and simple spatial interpolation method that uses the distance between the interpolation point and the sample point as the weight for a weighted average. The closer the sample point is to the interpolation point, the greater the weight it is given.

[0036] 5. One-Class Support Vector Machine: Also known as a single-class support vector machine, its training samples have only one class, which is either positive or negative. It calculates a hypersphere with the smallest radius from the training samples and includes all test samples inside this hypersphere. When the test set is classified using this hypersphere, the samples falling inside the hypersphere are classified as Class 1, and the samples falling outside the hypersphere are classified as Class 2.

[0037] Next, an exemplary system architecture applying the technical solution of this application will be described.

[0038] Figure 1 An exemplary system architecture block diagram illustrating the application of the technical solution of this application is shown schematically.

[0039] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a server 102, and a network 103. The terminal device 101 may be various electronic devices with both a display screen and a camera unit, such as smartphones, tablets, laptops, smart TVs, and smart vehicle terminals. In addition, the terminal device 101 may also include an electronic device with a display screen and a camera connected to that electronic device. The server 102 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The network 103 may be a communication medium of various connection types capable of providing a communication link between the terminal device 101 and the server 102, such as a wired communication link or a wireless communication link.

[0040] Depending on the implementation requirements, the system architecture in this application embodiment can have any number of terminal devices, networks, and servers. For example, the server can be a server group composed of multiple server devices. Furthermore, the technical solutions provided in this application embodiment can be applied to terminal device 101.

[0041] In one embodiment of this application, an imaging device or imaging unit captures an image of the object to be detected to obtain an RGB image and an initial point cloud map corresponding to the object. The captured RGB image and initial point cloud map are then sent to an electronic device with a display screen in terminal device 101. The electronic device can then send the RGB image and initial point cloud map to server 102 via network 103. For the initial point cloud map, server 102 can retain only the point coordinate information to form a point cloud map. The point cloud map can then be preprocessed by sampling the farthest point and the nearest neighbor point to obtain multiple point cloud groups corresponding to the point cloud map. A first feature extractor encodes all point cloud groups to obtain point cloud feature groups. Finally, the point cloud feature groups are interpolated back to the point cloud map to generate a point cloud feature map, and the point cloud feature map is projected to form a point cloud projection map of the same size as the RGB image. For the RGB image, server 102 can input it to a second feature extractor for encoding to obtain RGB features. Next, in server 102, the RGB features and the point cloud features in the point cloud projection map can be input into the multilayer perceptron for feature extraction, and the feature information output by the multilayer perceptron can be fused to obtain fused features. Finally, based on the RGB features, point cloud features, fused features and preset memory, anomaly prediction is performed on the object to be detected to obtain the anomaly score and anomaly representation map corresponding to the object to be detected.

[0042] In one embodiment of this application, the first feature extractor and the second feature extractor are pre-trained feature extractors belonging to a machine learning model, which relates to artificial intelligence.

[0043] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0044] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0045] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0046] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0047] In one embodiment of this application, the server 102 in this application may be a cloud server that provides cloud computing services, that is, this application relates to cloud storage and cloud computing technologies.

[0048] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0049] Cloud computing is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" appear infinitely scalable, readily available, on-demand, and expandable, with payment based on usage.

[0050] The following detailed description of the technical solutions provided in this application, including the three-dimensional industrial anomaly detection method, the three-dimensional industrial anomaly detection device, the computer-readable medium, and the electronic equipment, is based on specific embodiments.

[0051] Figure 2 This illustration schematically shows a flowchart of a three-dimensional industrial anomaly detection method according to one embodiment of this application. The three-dimensional industrial anomaly detection method is executed by a server, which may specifically be... Figure 1 Server 102 in the middle. For example... Figure 2 As shown, the three-dimensional industrial anomaly detection method in this application embodiment can mainly include the following steps S210 to S240.

[0052] Step S210: Obtain the RGB image and point cloud map corresponding to the object to be detected, and preprocess the point cloud map to obtain multiple point cloud groups;

[0053] Step S220: Input the point cloud group into the first feature extractor for feature extraction to obtain the point cloud feature group; generate a point cloud feature map based on the point cloud feature group and the point cloud image; and project the point cloud feature map to form a point cloud projection map with the same size as the RGB image; input the RGB image into the second feature extractor for feature extraction to obtain RGB features;

[0054] Step S230: Input the RGB features and the point cloud features in the point cloud projection map into the multilayer perceptron for feature extraction, and perform fusion processing on the feature information output by the multilayer perceptron to obtain fused features;

[0055] Step S240: Perform anomaly prediction on the object to be detected based on the RGB features, the point cloud features, and the fusion features to obtain anomaly scores and anomaly representation maps corresponding to the object to be detected.

[0056] In the three-dimensional industrial anomaly detection method provided in this application embodiment, after acquiring the RGB image and point cloud map corresponding to the object to be detected, the point cloud map is first preprocessed to obtain multiple point cloud groups; then, the point cloud groups are input to a first feature extractor for feature extraction to obtain point cloud feature groups; a point cloud feature map is generated based on the point cloud feature groups and the point cloud map, and the point cloud feature map is projected to form a point cloud projection map of the same size as the RGB image; simultaneously, the RGB image is input to a second feature extractor for feature extraction to obtain RGB features; then, the RGB features and the point cloud features in the point cloud projection map are respectively input to a multilayer perceptron for feature extraction, and the feature information output by the multilayer perceptron is fused to obtain fused features; finally, anomaly prediction is performed on the object to be detected based on the RGB features, point cloud features, and fused features to obtain anomaly scores and anomaly representation maps corresponding to the object to be detected. This application enables multimodal industrial anomaly detection based on 3D point clouds and RGB images. By fusing RGB and point cloud features, it improves the interactivity between features and reduces mutual interference between high-dimensional features. Furthermore, it predicts anomalies in the target object based on various data features stored in the memory bank, RGB features, point cloud features, and fused features, obtaining anomaly scores and anomaly representation maps, thus improving the accuracy of anomaly detection results. Moreover, this application adopts an unsupervised approach, eliminating the need for manual annotation before anomaly detection, reducing manual annotation costs. Using this technical solution for initial screening of industrial anomalies can also reduce manual quality inspection costs.

[0057] The specific implementation of each method step of the three-dimensional industrial anomaly detection method in the embodiments of this application will be described in detail below.

[0058] In step S210, an RGB image and a point cloud map corresponding to the object to be detected are acquired, and the point cloud map is preprocessed to obtain multiple point cloud groups.

[0059] In one embodiment of this application, the object to be inspected can be any object that needs to be inspected in an industrial quality inspection scenario, such as machine parts, fruits and vegetables, processed foods, handicrafts, ropes, etc. After acquiring the object to be inspected, different cameras can be used to photograph it to obtain an initial point cloud map and RGB image corresponding to the object to be inspected. Specifically, a two-dimensional camera, such as an area scan camera, can be used to photograph the object to be inspected to obtain an RGB image, and a three-dimensional camera, such as a laser line scan camera, can be used to photograph the object to be inspected to obtain an initial point cloud map.

[0060] In one embodiment of this application, after obtaining an initial point cloud image, coordinate preservation processing can be performed to form a new point cloud image, and the point cloud image can be preprocessed to obtain multiple point cloud groups corresponding to the point cloud image. The coordinate preservation processing involves retaining the point coordinates in the initial point cloud image while deleting other information, such as color information or color information and laser reflection information. Specifically, the preprocessing can involve secondary sampling of the point cloud image. First, the farthest point is sampled to obtain a first number of target points. Then, nearest neighbor sampling is performed with each target point as the center point to obtain a second number of feature points associated with each target point. Finally, point cloud groups are formed based on each target point and the feature points associated with each target point. The first and second numbers are integers greater than one, and both are less than the number of points contained in the point cloud image. In this embodiment, nearest neighbor sampling uses the point closest to the center point as the sampling point.

[0061] In step S220, the point cloud group is input to the first feature extractor for feature extraction to obtain the point cloud feature group. A point cloud feature map is generated based on the point cloud feature group and the point cloud map. The point cloud feature map is then projected to form a point cloud projection map with the same size as the RGB image. The RGB image is input to the second feature extractor for feature extraction to obtain the RGB features.

[0062] In one embodiment of this application, after obtaining multiple preprocessed point cloud groups, the point cloud group sequence formed by the point cloud groups can be input into a first feature extractor to extract point cloud features, thereby obtaining a three-dimensional representation corresponding to each point cloud group. Based on the three-dimensional representation corresponding to each point cloud group, a point cloud feature group corresponding to the point cloud map can be formed. In an embodiment of this application, when multiple point cloud groups form a point cloud group sequence, it can be generated by sorting the center points of each point cloud group in the point position sequence formed by all points in the point cloud map. The first feature extractor is a pre-trained feature extractor, specifically, a self-attention network Transformer model can be used as the first feature extractor. After the point cloud group sequence is input into the Transformer model, the Transformer model can encode the point cloud group based on the information of other point cloud groups related to a certain point cloud group, so as to output the feature vector corresponding to the point cloud group. Since the Transformer model can encode each point cloud group, it can correspondingly output the same number of point cloud features. Specifically, when M target points are obtained by sampling from the farthest point, and M point cloud groups are obtained based on the M target points through sampling from the farthest point, the Transformer model can encode the M point cloud groups and output M point cloud features. Furthermore, these M point cloud features form a point cloud feature group. In addition, since each point cloud group is obtained by sampling from the nearest neighbor point with any one of the M target points as the center point, each point cloud feature in the point cloud feature group can be regarded as the feature of the corresponding center point. It is worth noting that the first feature extractor can also be other pre-trained feature extractors, such as ResNet, Swin-Transformer, etc., and this application embodiment does not specifically limit it in this way.

[0063] Because the distribution of target points in space is not uniform after sampling at the farthest point, the density of point features is unbalanced, and there is spatial inconsistency between the point cloud features and the grid pattern of the RGB image. In order to align the spatial positions of the point cloud features and the image features of the RGB image, and to facilitate the subsequent fusion of the color features of the RGB image and the point cloud features of the point cloud map, it is first necessary to register the point cloud features. Specifically, the point cloud feature group can be interpolated back into the point cloud map to generate a point cloud feature map. Then, the point cloud feature map is projected to form a point cloud projection map of the same size as the RGB image. The point cloud features in the point cloud projection map can then be aligned with the spatial positions of the image features of the RGB image.

[0064] Next, the methods for generating point cloud feature maps through interpolation and point cloud projection maps through projection will be explained in detail.

[0065] In one embodiment of this application, inverse distance weighted interpolation is used to interpolate the point cloud feature set back to the point cloud map to generate a point cloud feature map. Assume that the point cloud feature set G output by the Transformer module contains M point cloud features g. i (i = 1, 2, ..., M), where the M point cloud features correspond to M target points (center points) c respectively. i (i = 1, 2, ..., M), the point cloud contains N points p. j (j = 1, 2, ..., N), by interpolating M point cloud features back into the point cloud map, a rasterized point cloud feature map can be generated, where the point cloud feature corresponding to each point in the feature map is p′. j (j = 1, 2, ...

[0066] …, N).

[0067] In one embodiment of this application, point cloud feature p′ j and weight α i Target point c i The corresponding point cloud feature g i And the first quantity M satisfies the first relation: Furthermore, the weight α i With target point c i The point cloud feature map and the target point c i The corresponding point p j Minimal constant ε, all target points c k and all points p in the point cloud diagram t Satisfying the second relation:

[0068]

[0069] Analysis of the first and second relations reveals that, with respect to the target point c i The corresponding weight α i It is based on the target point c i The coordinates and point cloud diagram of the target point c i The corresponding point p j The L2 norm determined by the coordinates of all target points c k The coordinates and all points p in the point cloud diagram t The L2 norm, determined by the coordinates, is used for calculation. The minimum constant ε is set to avoid a denominator of 0. It is worth noting that... During the calculation, for each target point c k Both require calculating the target point c. k coordinates and p t The second norm between the coordinates of (t = 1, 2, ..., N) is then used to iterate through each c when k = 1, 2, ..., M based on this logic. kThis leads to the final calculation result. Because c k and p t It remains unchanged, therefore, in each calculation of the weight α i When the denominator of the second expression is the same, it is used to determine the target point c. i The corresponding weight α i Then, based on this weight α i For target point c i The corresponding point cloud feature g i Perform interpolation to obtain the result at point p. j The corresponding interpolated point cloud feature p′ j .

[0070] In one embodiment of this application, after interpolation, the point cloud feature map still corresponds to a feature map in three-dimensional space. To facilitate spatial alignment with the image features of the RGB image, the point cloud feature map needs to be projected onto a two-dimensional plane to construct a point cloud projection map of the same size as the RGB image. During projection, the point cloud features in the point cloud feature map can be projected onto the two-dimensional plane according to the point coordinates and camera parameters.

[0071] Figure 3 This schematically illustrates the process of generating a point cloud projection map, such as... Figure 3 As shown, in step S301, the first camera parameters corresponding to the RGB image and the second camera parameters corresponding to the point cloud map are obtained; in step S302, the transfer matrix is ​​determined according to the first camera parameters and the second camera parameters; in step S303, the point coordinates in the point cloud feature map are projected based on the transfer matrix to obtain the two-dimensional coordinates corresponding to the point coordinates, and the point cloud projection map is determined according to the two-dimensional coordinates and the point cloud features corresponding to the point coordinates.

[0072] Since the imaging devices used to capture RGB images and point cloud maps are different, the corresponding camera parameters are also different. These camera parameters may include, for example, focal length, aperture, camera coordinate system, etc. For ease of description, the camera parameters corresponding to the RGB image are taken as the first camera parameters, and the camera parameters corresponding to the point cloud map are taken as the second camera parameters. After obtaining the first and second camera parameters, a transition matrix can be determined based on the first and second camera parameters. This transition matrix can convert the point coordinates in three-dimensional space into point coordinates in a two-dimensional plane. Then, based on this transition matrix, the point coordinates of each point in the point cloud feature map can be projected onto the two-dimensional plane, and the point cloud feature p′ corresponding to the point coordinates can be... j It is also projected onto the corresponding point coordinates in the two-dimensional plane. The point cloud projection map can be determined based on the two-dimensional coordinates corresponding to the point coordinates and the point cloud features. The size of the point cloud projection map is the same as the size of the RGB image.

[0073] In one embodiment of this application, since the point cloud may be sparse, there are points in the two-dimensional plane that match the point cloud feature map and points that do not match. Therefore, during projection, the pixels of the points in the point cloud projection map that match the point cloud feature map can be set to non-zero values, and the pixels of the points in the point cloud projection map that do not match the point cloud feature map can be set to 0. For example, the position where there is no object to be detected in the two-dimensional plane is a mismatched point, and it can be marked as 0.

[0074] In one embodiment of this application, while preprocessing the point cloud image to obtain multiple point cloud groups, extracting features from the point cloud groups using a first feature extractor to obtain point cloud feature groups, and performing inverse distance weighted interpolation on the point cloud image based on the point cloud feature groups to generate a point cloud feature map, and projecting the point cloud feature map to generate a point cloud projection map, a second feature extractor can also extract features from the RGB image to obtain RGB features. This second feature extractor is also a pre-trained feature extractor, such as a pre-trained Transformer self-attention network model, or other pre-trained self-attention network models; this application does not specifically limit its application to this.

[0075] In one embodiment of this application, since the purpose of this application is to detect industrial anomalies, specifically to obtain an anomaly score for the object to be detected, locate the abnormal parts, and output an anomaly characterization map, the detection process requires dividing the point cloud projection map and RGB image into blocks. The point cloud projection map is divided into multiple blocks, and the RGB image is divided into multiple patches. Then, industrial anomaly detection is performed based on the point cloud features corresponding to the blocks and the RGB features corresponding to the patches. In this embodiment of the application, the standard for dividing the point cloud projection map and RGB image into blocks is the same, which ensures that the blocks in the point cloud projection map and the patches in the RGB image are in one-to-one correspondence, and the point cloud features and RGB features can be aligned.

[0076] In one embodiment of this application, a point cloud projection map can be divided into multiple patches according to a preset division standard. Then, average pooling is performed on the point cloud projection maps corresponding to each patch to obtain the patch point cloud features corresponding to each patch. Accordingly, the point cloud features corresponding to the point cloud projection map can be obtained based on the patch point cloud features corresponding to all patches. Similarly, an RGB image can be divided according to the preset division standard to obtain patches corresponding to the RGB image. Then, the RGB image information corresponding to each patch is simultaneously input into a second feature extractor to obtain the RGB features corresponding to each patch. The RGB features corresponding to the RGB image can be obtained based on the RGB features corresponding to all patches.

[0077] In one embodiment of this application, when acquiring the point cloud map and RGB image corresponding to the object to be detected, an index i can be set for the point cloud map and RGB image of the object to be detected. When processing the point cloud map to generate the point cloud projection map, the index i remains unchanged. After dividing the point cloud feature map into patches and dividing the RGB image into spots, an index j can be set for each patch and spot according to a certain order rule. Patches and spots corresponding to the same position in the point cloud projection map and the RGB image correspond to the same index. Based on the index i and the index j, a patch can be uniquely determined in the point cloud projection map corresponding to index i, and a spot can be uniquely determined in the RGB image corresponding to index i. Then, the point cloud features corresponding to the patch and the RGB features corresponding to the spot can be fused to extract the interaction information between the two modalities.

[0078] In step S230, the RGB features and the point cloud features in the point cloud projection map are respectively input into a multilayer perceptron for feature extraction, and the feature information output by the multilayer perceptron is fused to obtain fused features.

[0079] In one embodiment of this application, after completing the division and indexing of the point cloud projection map and the RGB image, the RGB features and the point cloud features in the point cloud projection map can be input into a multilayer perceptron for feature extraction. Then, the feature information output by the multilayer perceptron is fused to obtain a fused feature that incorporates color and position information. Since the patches in the point cloud projection map and the patches in the RGB image are in one-to-one correspondence, and patches and patches at the same location have the same index, the multilayer perceptron can extract features from the RGB features of the target patch and the patch point cloud features of the target patch at the same location to obtain first feature information and second feature information corresponding to the target patch. Furthermore, the first feature information and the second feature information can be fused to obtain a fused feature. This fused feature includes the color features of the target patch and the point cloud features of the target patch; that is, the fused feature contains the interaction information between the two modalities.

[0080] For ease of description, in the embodiments of this application, the point cloud features corresponding to the map tiles are labeled as follows: The RGB feature markers corresponding to the image patches are as follows: Fusion features are labeled as Where i is the index of the object to be detected, and j is the index of the patch or tile; the multilayer perceptron includes a first multilayer perceptron χ for feature extraction of point cloud features. pt Second multilayer perceptron χ rgb Accordingly,

[0081]

[0082] By extracting and fusing the RGB features corresponding to all image patches and the point cloud features corresponding to all image blocks, fused features corresponding to the RGB image and the point cloud projection map can be obtained.

[0083] In step S240, anomaly prediction is performed on the object to be detected based on the RGB features, the point cloud features, and the fusion features to obtain an anomaly score and an anomaly representation map corresponding to the object to be detected.

[0084] In one embodiment of this application, after obtaining the RGB features corresponding to the RGB image, the point cloud features corresponding to the point cloud projection map, and the fusion features corresponding to the RGB image and the point cloud projection map, these features can be input into a decision layer. This decision layer has three memory banks: a first memory bank, a second memory bank, and a third memory bank. The first memory bank stores the original color features of the detected objects, the second memory bank stores the point cloud features of the detected objects, and the third memory bank stores the fusion features of the detected objects. When the RGB features, point cloud features, and fusion features corresponding to the object to be detected are input into the decision layer, anomaly prediction can be performed on the object to be detected based on the memory banks in the decision layer, the RGB features, point cloud features, and fusion features, thereby obtaining an anomaly score and an anomaly representation map corresponding to the object to be detected.

[0085] In one embodiment of this application, since the three memory banks correspond to the original color features, point cloud features, and fusion features respectively, anomaly prediction of the object to be detected can be performed from the color dimension based on the first memory bank and RGB features, from the position dimension based on the second memory bank and point cloud features, and from the feature fusion dimension based on the third memory bank and fusion features. Then, the prediction information from the three dimensions is combined to obtain the anomaly score and anomaly representation map corresponding to the object to be detected.

[0086] Since the prediction of anomaly scores and anomaly representation maps are two separate tasks, in the embodiments of this application, two learnable single-class support vector machines, Da and Ds, can be used to make decisions on anomaly scores and anomaly representation maps. Specifically, when predicting anomaly scores, a first anomaly score corresponding to the object to be detected can be determined based on a first scoring function, a first memory, and RGB features; a second anomaly score can be determined based on a first scoring function, a second memory, and point cloud features; and a third anomaly score can be determined based on a first scoring function, a third memory, and fusion features. Then, the first, second, and third anomaly scores are input into a first single-class support vector machine Da to obtain the anomaly score corresponding to the object to be detected. When predicting anomaly representation maps, a first anomaly segmentation map score corresponding to the object to be detected can be determined based on a second scoring function, a first memory, and RGB features; a second anomaly segmentation map score corresponding to the object to be detected can be determined based on a second scoring function, a second memory, and point cloud features; and a third anomaly segmentation map score corresponding to the object to be detected can be determined based on a second scoring function, a third memory, and fusion features. Then, the first, second, and third anomaly segmentation map scores are input into a second single-class support vector machine Ds to obtain the segmentation map with anomalies corresponding to the object to be detected. Finally, the segmentation maps with anomalies are marked with a color different from the normal segmentation maps to obtain the anomaly representation map.

[0087] The expression for the first scoring function is: in, M is the first memory bank, the second memory bank, or the third memory bank; f is the RGB feature, the point cloud feature, or the fusion feature; η is the weight parameter; i is the index corresponding to the RGB image or the point cloud projection map; and j is the index corresponding to the patch in the RGB image or the block in the point cloud projection map.

[0088] The expression for the second scoring function is: Where M is the first memory bank, the second memory bank, or the third memory bank; f is the RGB feature, the point cloud feature, or the fusion feature; i is the index of the RGB image; and j is the index corresponding to the patch in the RGB image.

[0089] According to the expression of the first scoring function, when f is an RGB feature... At that time, the first memory bank M can be acquired. rgb ,according to and M rgb The determination of each original color feature in the data makes Take the minimum value m rgb As m* and taking the maximum value As f (i,j),*Then, based on the weight parameters η, m* and f (i,j),* The first abnormality score Φ(M) can be determined. rgb ,f rgb When f is a point cloud feature At that time, the second memory bank M can be acquired. pt ,according to and M pt Determining the features of each point cloud in the data enables Take the minimum value m pt As m* and taking the maximum value As f (i,j),* Then, based on the weight parameters η, m* and f (i,j),* It can be determined that the second abnormal score Φ(M) is obtained. pt ,f pt Correspondingly, when f is a fusion feature At that time, the third memory bank M can be obtained. fs ,according to and M fs The determination of each fusion feature in the process enables Take the minimum value m fs As m* and taking the maximum value As f (i,j),* Then, based on the weight parameters η, m* and f (i,j),* It can be determined that the third abnormality score Φ(M) is obtained. fs ,f fs ).

[0090] According to the expression of the second scoring function, when f is an RGB feature... At that time, the first memory bank M can be acquired. rgb ,according to and M rgb The determination of each original color feature in the data makes Take the minimum value m rgb As the first anomaly segmentation map score When f is a point cloud feature At that time, the second memory bank M can be acquired. pt ,according to and M pt Determining the features of each point cloud in the data enables Take the minimum value m pt As the second anomaly segmentation map score Accordingly, when f is a fusion feature At that time, the third memory bank M can be obtained. fs ,according to and M fs The determination of each fusion feature in the process enables Take the minimum value m fsAs the third anomaly segmentation map score

[0091] Furthermore, in obtaining the first abnormal score Φ(M) rgb ,f rgb ), second abnormal score Φ(M) pt ,f pt ) and the third abnormality score Φ(M) fs ,f fs After that, it can be input into the first single-class support vector machine to obtain the anomaly score, the expression being: a = Da(Φ(M) rgb ,f rgb ), Φ(M pt ,f pt ), Φ(M fs ,f fs Meanwhile, in obtaining the score of the first anomaly segmentation map... Second anomaly segmentation map score and the third anomaly segmentation map score Then, it can be input into the second single-class support vector machine to obtain the segmentation map of the anomaly, with the expression:

[0092] The 3D industrial anomaly detection method in this application embodiment is only used for initial screening of industrial anomalies. After obtaining the anomaly score and anomaly characterization map corresponding to the object to be detected, manual quality inspection is still required to determine whether the object to be detected truly has anomalies. This is because the 3D industrial anomaly detection in this application embodiment only identifies possible anomalies in the object to be detected. However, if the object to be detected has gaps, cracks, or other defects, it will also be identified as anomaly, which may lead to misjudgment. Although manual quality inspection is still required after obtaining the anomaly score and anomaly characterization map using the technical solution of this application, the accuracy of the anomaly detection results is improved because this application can perform industrial anomaly detection based on 3D point cloud and color information. Furthermore, manual quality inspection only needs to be performed on the objects to be detected that are determined to have anomalies, rather than on all objects to be detected, thus greatly saving the cost of manual quality inspection.

[0093] In one embodiment of this application, the memory is generated through random sampling. Specifically, the first memory, the second memory, and the third memory are generated by randomly sampling RGB features, point cloud features, or fused features from a large number of samples. By selecting a portion of data from a large amount of data to form the memory through random sampling, the amount of data processing can be reduced and the data processing efficiency can be improved.

[0094] The three-dimensional industrial anomaly detection method in this application can be implemented using a three-dimensional industrial anomaly detection system. Figure 4A schematic diagram of the structure of a three-dimensional industrial anomaly detection system is shown, such as... Figure 4 As shown, the 3D industrial anomaly detection system 400 includes a point cloud feature space registration module 401, a feature fusion module 402, and a decision layer fusion module 403. The point cloud feature space registration module 401 is used to process the point cloud image to generate a point cloud projection map of the same size as the RGB image. The feature fusion module 402 is used to fuse the RGB features in the RGB image and the point cloud features in the point cloud projection map to generate fused features. The decision layer fusion module 403 is used to predict anomalies of the object to be detected based on the RGB features, point cloud features, fused features, and a preset memory library to obtain an anomaly score and an anomaly representation map corresponding to the object to be detected.

[0095] Next, taking the three-dimensional industrial anomaly detection of cable sealing sleeves as an example, the detection process of the three-dimensional industrial anomaly detection system will be explained in detail.

[0096] Figure 5 This schematic diagram illustrates the process of using a three-dimensional industrial anomaly detection system to detect anomalies in cable sealing sleeves. Figure 5As shown, in step S501, after acquiring the RGB image and initial point cloud image corresponding to the cable sealing sleeve, the initial point cloud image is processed to obtain a point cloud image containing only point coordinate information; in step S502, the point cloud image is input to the point cloud feature space registration module, and steps S503-S508 are performed on the point cloud image through the point cloud feature space registration module; in step S503, the farthest point sampling and nearest neighbor point sampling are performed on the point cloud image to generate multiple point cloud groups; in step S504, the multiple point cloud groups are arranged according to a preset sorting rule to form a point cloud group sequence; in step S505, the point cloud group sequence is input to the first feature extractor for feature extraction to obtain point cloud feature groups; in step S506, the point cloud feature groups are interpolated... The point cloud image is retrieved and a point cloud feature map is generated. In step S507, the point cloud feature map is projected to generate a point cloud projection map of the same size as the RGB image. In step S508, the point cloud projection map is divided into multiple patches according to a preset division standard, and the point cloud features corresponding to each patch are averaged to obtain the patch point cloud features corresponding to each patch. In step S509, the RGB image is input to the feature fusion module, and steps S510-S513 are performed on the RGB image through the feature fusion module. In step S510, the RGB image is divided into multiple patches according to a preset division standard, and the patch sequence formed by the multiple patches is input to the second feature extractor. In step S511, the second feature extractor performs feature extraction. In step S512, the point cloud features corresponding to each patch are input to the first multilayer perceptron for feature extraction to obtain first feature information, and the RGB features corresponding to each patch are input to the second multilayer perceptron for feature extraction to obtain second feature information. In step S513, the first feature information and second feature information corresponding to the same position are fused to obtain fused features. In step S514, the RGB features corresponding to each patch, the point cloud features corresponding to each patch, and the fused features corresponding to each patch / patch are input to the decision layer fusion module, and the decision layer fusion module performs step S515- on the RGB features, point cloud features, and fused features. S522; In step S515, a first anomaly score corresponding to the cable sealing sleeve is determined based on the first scoring function, the first memory bank, and RGB features; In step S516, a second anomaly score corresponding to the cable sealing sleeve is determined based on the first scoring function, the second memory bank, and point cloud features; In step S517, a third anomaly score corresponding to the cable sealing sleeve is determined based on the first scoring function, the third memory bank, and fusion features; In step S518, a first anomaly segmentation map score corresponding to the cable sealing sleeve is determined based on the second scoring function, the first memory bank, and RGB features; In step S519, a second anomaly segmentation map score corresponding to the cable sealing sleeve is determined based on the second scoring function, the second memory bank, and point cloud features.In step S520, a third anomaly segmentation map score corresponding to the cable sealing sleeve is determined based on the second scoring function, the third memory, and the fusion features. In step S521, the first anomaly score, the second anomaly score, and the third anomaly score are input into the first single-class support vector machine to obtain the anomaly score corresponding to the cable sealing sleeve. In step S522, the first anomaly segmentation map score, the second anomaly segmentation map score, and the third anomaly segmentation map score are input into the second single-class support vector machine to obtain the segmentation map with anomalies corresponding to the cable sealing sleeve, and the segmentation maps with anomalies are marked with a color different from the normal segmentation maps to obtain an anomaly characterization map.

[0097] At this point, the anomaly score and anomaly representation map corresponding to the object to be detected can be obtained. Further, Figure 5 The process may also include step S523: setting the pixels at locations with anomalies in the anomaly representation map to 255 and setting the pixels at locations without anomalies to 0, in order to obtain an anomaly location map.

[0098] Although the above embodiments describe three-dimensional industrial anomaly detection of cable sealing sleeves, it is understood that the three-dimensional industrial anomaly detection method in this application can also be applied to other types of industrial anomaly detection scenarios. Figure 6 The illustration schematically shows the anomaly characterization map and anomaly location map determined after anomaly detection of various objects using the three-dimensional industrial anomaly detection method of this application, such as... Figure 6 As shown, the objects to be inspected are bagels, cable sealing sleeves, carrots, biscuits, hidden tenons, foam, peaches, potatoes, ropes, and tires. The first row is the RGB image corresponding to each object to be inspected, the second row is the point cloud image corresponding to each object to be inspected, the third row is the anomaly representation map corresponding to each object to be inspected generated by 3D industrial anomaly detection, and the fourth row is the anomaly location map corresponding to each object to be inspected generated by 3D industrial anomaly detection.

[0099] In one embodiment of this application, before using the 3D industrial anomaly detection system to perform anomaly detection on the object to be detected, it is necessary to train each module in the 3D industrial anomaly detection system to obtain a stable 3D industrial anomaly detection system. Since the point cloud feature space registration module only contains a first feature extractor, which is a pre-trained feature extractor, there is no need to train the point cloud feature space registration module. The feature fusion module contains a second feature extractor, a first multilayer perceptron, and a second multilayer perceptron. Although the second feature extractor is a pre-trained feature extractor, the parameters in the first and second multilayer perceptrons still need to be optimized; that is, the feature fusion module needs to be trained. The decision layer fusion module contains three memory banks, a learnable first single-class support vector machine, and a second single-class support vector machine; therefore, the decision layer fusion module also needs to be trained. In this embodiment of the application, the training of the feature fusion module and the decision layer fusion module is performed asynchronously and both adopt an unsupervised training mode. Since both adopt an unsupervised training mode, no manual annotation is required before training, thereby reducing the cost of manual annotation.

[0100] In one embodiment of this application, the feature fusion module is trained using a patch-based contrast loss, so that the feature fusion module can fully learn the intrinsic relationship between color and 3D point cloud modes present in the training data. For RGB feature f rgb and point cloud features f pt The training objective is to encourage features from different modalities at the same location to have more corresponding information, while features from different locations have less corresponding information.

[0101] When acquiring training samples, an index can be set for the training samples. Since multimodal information needs to be extracted from the RGB image and point cloud map of each training sample, the RGB image and point cloud map corresponding to the same training sample, as well as the point cloud projection map generated by processing the point cloud map, all correspond to the same index. Furthermore, the feature fusion module is trained using a patch-based contrastive loss, so the RGB image and point cloud projection map also need to be divided into blocks, and the same block division standard is used to ensure that the patches formed by the RGB image blocks and the patches formed by the point cloud projection map blocks are in one-to-one correspondence, thereby achieving spatial alignment between RGB features and point cloud features.

[0102] Similar to the method for three-dimensional industrial anomaly detection of the object to be detected in the above embodiments, the embodiments of this application employ... To represent the RGB features of the image patches, using Let m be the index of the training sample and n be the index of the corresponding patch / plot. After obtaining the RGB features and point cloud features corresponding to the training samples, a multilayer perceptron (MLP) layer {χ rgb ,χ pt} to extract interaction information between two modalities, and use a fully connected layer {σ rgb ,σ pt This maps the processed features to a query or key vector, and the mapped features are represented as follows:

[0103] Figure 7 This schematically illustrates the flowchart of the unsupervised training feature fusion module, such as... Figure 7 As shown: In step S701, the point cloud map corresponding to the batch training samples is input to the point cloud feature space registration module. The point cloud feature space registration module performs farthest point sampling, nearest neighbor point sampling, encoding, interpolation, projection, block division, and average pooling processing on the point cloud map corresponding to the batch training samples. In step S702, the point cloud feature space registration module outputs multiple patches constituting the point cloud projection map and the patch point cloud features corresponding to each patch. In step S703, the patch point cloud features corresponding to each patch are input to the first multilayer perceptron. In step S704, the first multilayer perceptron outputs the first feature information corresponding to the patch and inputs the first feature information to the first fully connected layer. In step S705, the first fully connected layer outputs the first mapping feature. In step S706, the RGB images corresponding to the batch training samples are divided into multiple patches, and the patch sequences corresponding to each training sample are input to the second feature extractor; in step S707, the second feature extractor outputs the RGB features corresponding to the patch sequences; in step S708, the RGB features corresponding to each patch are input to the second multilayer perceptron; in step S709, the second multilayer perceptron outputs the second feature information corresponding to the patches, and inputs the second feature information to the second fully connected layer; in step S710, the second fully connected layer outputs the second mapping feature to determine the contrast loss function based on the first mapping feature and the second mapping feature, and tunes the feature fusion module according to the loss function to obtain a feature fusion module with stable performance.

[0104] In step S710, the contrast loss function is the InfoNCE loss function, and its specific expression is as follows: Where N b N represents the batch size. pThe number of non-zero patches is T, where T is the transpose. It's worth noting that the batch size is the number of training samples contained in the batch of training samples, and each training sample has a different index. The number of non-zero patches is the number of patches formed by dividing the RGB image into blocks or the number of patches formed by dividing the point cloud projection map into blocks.

[0105] After tuning the feature fusion module using the contrastive loss function, a stable feature fusion module can be obtained. This module can extract a large amount of corresponding information for features of different modalities at the same location, but for features of different modalities at different locations, it will not extract corresponding information or will extract very little corresponding information. Figure 7 The comparison matrix shown has tiles on the diagonal corresponding to the information extracted from features of different modalities located at the same position, and tiles off the diagonal corresponding to the information extracted from features of different modalities located at different positions. The darker color of the diagonal tiles indicates richer corresponding information, while the lighter color of the off-diagonal tiles indicates less corresponding information.

[0106] Furthermore, after training the feature fusion module is complete, the first feature information and the second feature information can be fused to obtain the fused feature f. fs ,like Figure 7 As shown by the dashed line in the image.

[0107] In one embodiment of this application, after training the feature fusion module is completed, the decision layer fusion module can be trained. In this embodiment, firstly, through each memory bank M... rgb M fp M fs The abnormal score and abnormal segmentation map score results corresponding to all training samples are given. Then, the abnormal score and abnormal segmentation map score results are used as the training inputs of the first single-class support vector machine and the second single-class support vector machine, respectively, to achieve unsupervised training with only positive examples.

[0108] The three-dimensional industrial anomaly detection method in this application first acquires an RGB image and a point cloud map corresponding to the object to be detected. Then, the point cloud map is preprocessed to obtain multiple point cloud groups. Next, the point cloud groups are input to a first feature extractor for feature extraction to obtain point cloud feature groups. A point cloud feature map is generated based on the point cloud feature groups and the point cloud map, and the point cloud feature map is projected to form a point cloud projection map of the same size as the RGB image. Simultaneously, the RGB image is input to a second feature extractor for feature extraction to obtain RGB features. Then, the RGB features and the point cloud features from the point cloud projection map are respectively input to a multilayer perceptron for feature extraction, and the feature information output by the multilayer perceptron is fused to obtain fused features. Finally, based on the RGB features, point cloud features, and fused features, anomaly prediction is performed on the object to be detected to obtain an anomaly score and anomaly representation map corresponding to the object to be detected. This application enables multimodal industrial anomaly detection based on 3D point clouds and RGB images. By fusing RGB and point cloud features, it improves the interactivity between features and reduces mutual interference between high-dimensional features. Furthermore, it predicts anomalies in the target object based on various data features stored in the memory bank, RGB features, point cloud features, and fused features, obtaining anomaly scores and anomaly representation maps, thus improving the accuracy of anomaly detection results. Moreover, this application adopts an unsupervised approach, eliminating the need for manual annotation before anomaly detection, reducing manual annotation costs. Using this technical solution for initial screening of industrial anomalies can also reduce manual quality inspection costs.

[0109] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0110] The following describes an embodiment of the apparatus of this application, which can be used to execute the three-dimensional industrial anomaly detection method in the above embodiments of this application. Figure 8 The schematic diagram illustrates the structural block diagram of the three-dimensional industrial anomaly detection device provided in an embodiment of this application. For example... Figure 8 As shown, the 3D industrial anomaly detection device 800 includes: a preprocessing module 810, a registration module 820, a fusion module 830, and a decision module 840, specifically:

[0111] A preprocessing module is used to acquire an RGB image and a point cloud map corresponding to the object to be detected, and preprocess the point cloud map to obtain multiple point cloud groups; a registration module is used to input the point cloud groups to a first feature extractor for feature extraction to obtain point cloud feature groups, generate a point cloud feature map based on the point cloud feature groups and the point cloud map, and project the point cloud feature map to form a point cloud projection map of the same size as the RGB image; input the RGB image to a second feature extractor for feature extraction to obtain RGB features; a fusion module is used to input the RGB features and the point cloud features in the point cloud projection map to a multilayer perceptron for feature extraction, and fuse the feature information output by the multilayer perceptron to obtain fused features; a decision module is used to perform anomaly prediction on the object to be detected based on the RGB features, the point cloud features, and the fused features to obtain anomaly scores and anomaly representation maps corresponding to the object to be detected.

[0112] In some embodiments of this application, based on the above technical solutions, the preprocessing module 810 is configured to: sample the farthest point of the point cloud map to obtain a first number of target points; sample the nearest neighbor points with the target points as the center points to obtain a second number of feature points related to each target point; and form the point cloud group according to each target point and the feature points related to each target point; wherein the first number and the second number are integers greater than one.

[0113] In some embodiments of this application, based on the above technical solutions, the registration module 820 includes: an interpolation unit, used to interpolate the point cloud feature group back to the point cloud map based on inverse distance weights, so as to obtain the point cloud feature map.

[0114] In some embodiments of this application, based on the above technical solutions, the interpolation unit is configured as: point cloud features p in the point cloud feature map j ′ and weight α i The point cloud feature g corresponding to the target point i And the first quantity M satisfies the first relation: The weight α i With target point c i The point cloud feature map and the target point c i The corresponding point p j Minimal constant ε, all the target points c k and all points p in the point cloud diagram t Satisfying the second relation:

[0115]

[0116] In some embodiments of this application, based on the above technical solutions, the registration module 820 is configured to: obtain first camera parameters corresponding to the RGB image and second camera parameters corresponding to the point cloud map; determine a transfer matrix based on the first camera parameters and the second camera parameters; project the point coordinates in the point cloud feature map based on the transfer matrix to obtain two-dimensional coordinates corresponding to the point coordinates, and determine the point cloud projection map based on the two-dimensional coordinates and the point cloud features corresponding to the point coordinates.

[0117] In some embodiments of this application, based on the above technical solutions, the registration module 820 is further configured to: when there are points in the point cloud projection map that do not match the point cloud feature map, set the pixel at the position corresponding to the point to 0.

[0118] In some embodiments of this application, based on the above technical solutions, the three-dimensional industrial anomaly detection device 800 further includes: a first segmentation module, used to divide the point cloud projection map into multiple blocks according to a preset segmentation standard, and to perform average pooling on the point cloud projection map corresponding to each block to obtain the block point cloud features corresponding to each block; and a point cloud feature acquisition module, used to acquire the point cloud features corresponding to the point cloud projection map based on the block point cloud features corresponding to all the blocks.

[0119] In some embodiments of this application, based on the above technical solutions, the registration module 820 is configured to: divide the RGB image according to the preset division standard to obtain the patches corresponding to the RGB image; input the RGB image information corresponding to each patch to the second feature extractor to obtain the RGB features corresponding to each patch; and determine the RGB features based on the RGB features corresponding to all the patches.

[0120] In some embodiments of this application, the map patch and the map feature are in one-to-one correspondence, and the corresponding map patch and map feature have the same index; based on the above technical solution, the fusion module 830 is configured to: extract features from the RGB features of the target map feature and the map feature point cloud features of the target map patch corresponding to the same position through the multilayer perceptron, so as to obtain first feature information corresponding to the target map feature and second feature information corresponding to the target map patch; and perform fusion processing on the first feature information and the second feature information to obtain the fused feature.

[0121] In some embodiments of this application, based on the above technical solutions, the decision module 840 is configured to: determine a first anomaly score corresponding to the object to be detected based on a first scoring function, a first memory, and the RGB features; determine a second anomaly score corresponding to the object to be detected based on the first scoring function, a second memory, and the point cloud features; determine a third anomaly score corresponding to the object to be detected based on the first scoring function, the third memory, and the fusion features; input the first anomaly score, the second anomaly score, and the third anomaly score into a first single-class support vector machine to obtain an anomaly score corresponding to the object to be detected, wherein the first memory is used to store the original color features of the detected object, the second memory is used to store the point cloud features of the detected object, and the third memory is used to store the fusion features of the detected object.

[0122] In some embodiments of this application, based on the above technical solutions, the decision module 840 is configured to: determine a first abnormal segmentation map score corresponding to the object to be detected based on a second scoring function, a first memory, and the RGB features; determine a second abnormal segmentation map score corresponding to the object to be detected based on a second scoring function, a second memory, and the point cloud features; determine a third abnormal segmentation map score corresponding to the object to be detected based on a second scoring function, a third memory, and the fusion features; input the first abnormal segmentation map score, the second abnormal segmentation map score, and the third abnormal segmentation map score into a second single-class support vector machine to obtain an abnormal segmentation map corresponding to the object to be detected; and mark the abnormal segmentation map with a color different from the normal segmentation map to obtain the abnormal characterization map.

[0123] In some embodiments of this application, based on the above technical solutions, the expression for the first scoring function is: Φ(M,f)=η||f (i,j),* -m * ||2, where, M is the first memory bank, the second memory bank, or the third memory bank; f is the RGB feature, the point cloud feature, or the fusion feature; η is the weight parameter; i is the index corresponding to the RGB image or the point cloud projection map; and j is the index corresponding to the patch in the RGB image or the block in the point cloud projection map.

[0124] In some embodiments of this application, based on the above technical solutions, the expression of the second scoring function is as follows: Where M is the first memory bank, the second memory bank, or the third memory bank; f is the RGB feature, the point cloud feature, or the fusion feature; i is the index corresponding to the RGB image or the point cloud projection map; and j is the index corresponding to the patch in the RGB image or the block in the point cloud projection map.

[0125] The specific details of the three-dimensional industrial anomaly detection device provided in the various embodiments of this application have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0126] Figure 9 This schematically illustrates a computer system architecture block diagram for implementing an electronic device according to embodiments of the present application. The electronic device may be, for example: Figure 1 The terminal device 101 and server 102 shown are illustrated.

[0127] It should be noted that, Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0128] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM). The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output interface 905 (I / O interface) is also connected to the bus 904.

[0129] In some embodiments, the following components are connected to the input / output interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a local area network card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0130] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit 901, it performs various functions defined in the system of this application.

[0131] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable medium, or any combination of the above. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0134] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause an electronic device to execute the method according to the embodiments of this application.

[0135] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0136] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A three-dimensional industrial anomaly detection method, characterized in that, include: Obtain the RGB image and point cloud map corresponding to the object to be detected, and preprocess the point cloud map to obtain multiple point cloud groups; The point cloud group is input to the first feature extractor for feature extraction to obtain the point cloud feature group. A point cloud feature map is generated based on the point cloud feature group and the point cloud map. The point cloud feature map is then projected to form a point cloud projection map with the same size as the RGB image. The RGB image is input to the second feature extractor for feature extraction to obtain the RGB features. The RGB features and the point cloud features in the point cloud projection map are respectively input into a multilayer perceptron for feature extraction, and the feature information output by the multilayer perceptron is fused to obtain fused features. A first anomaly score corresponding to the object to be detected is determined based on a first scoring function, a first memory bank, and the RGB features; a second anomaly score corresponding to the object to be detected is determined based on a first scoring function, a second memory bank, and the point cloud features; and a third anomaly score corresponding to the object to be detected is determined based on a first scoring function, a third memory bank, and the fusion features. The first abnormal score, the second abnormal score, and the third abnormal score are input into the first single-class support vector machine to obtain the abnormal score corresponding to the object to be detected. The first memory is used to store the original color features of the detected objects, the second memory is used to store the point cloud features of the detected objects, and the third memory is used to store the fused features of the detected objects.

2. The method according to claim 1, characterized in that, The preprocessing of the point cloud image to obtain multiple point cloud groups includes: The point cloud map is sampled for the farthest point to obtain a first number of target points; Nearest neighbor sampling is performed with the target point as the center point to obtain a second number of feature points related to each target point; The point cloud group is formed based on each target point and the feature points associated with each target point; Wherein, the first quantity and the second quantity are integers greater than one.

3. The method according to claim 1, characterized in that, Generating a point cloud feature map based on the point cloud feature group and the point cloud map includes: The point cloud feature group is interpolated back to the point cloud map based on the inverse distance weight to obtain the point cloud feature map.

4. The method according to claim 1 or 3, characterized in that, The step of projecting the point cloud feature map to form a point cloud projection map of the same size as the RGB image includes: Obtain the first camera parameters corresponding to the RGB image and the second camera parameters corresponding to the point cloud image; Determine the transition matrix based on the first camera parameters and the second camera parameters; The point coordinates in the point cloud feature map are projected based on the transition matrix to obtain the two-dimensional coordinates corresponding to the point coordinates, and the point cloud projection map is determined based on the two-dimensional coordinates and the point cloud features corresponding to the point coordinates.

5. The method according to claim 4, characterized in that, The method further includes: If there are points in the point cloud projection map that do not match the point cloud feature map, the pixels at the position corresponding to the point are set to 0.

6. The method according to claim 1, characterized in that, The method further includes: The point cloud projection map is divided into multiple blocks according to a preset division standard, and the point cloud projection map corresponding to each block is averaged and pooled to obtain the block point cloud features corresponding to each block. Obtain the point cloud features corresponding to the point cloud projection map based on the point cloud features of all the aforementioned tiles.

7. The method according to claim 6, characterized in that, The step of inputting the RGB image into a second feature extractor for feature extraction to obtain RGB features includes: The RGB image is divided according to the preset division criteria to obtain the image patches corresponding to the RGB image; The RGB image information corresponding to each of the aforementioned patches is input into the second feature extractor to obtain the RGB features corresponding to each of the aforementioned patches; The RGB features are determined based on the RGB features corresponding to all the said patches.

8. The method according to claim 7, characterized in that, The map blocks and map patches correspond one-to-one, and the corresponding map blocks and map patches have the same index; The step of inputting the RGB features and the point cloud features in the point cloud projection map into a multilayer perceptron for feature extraction, and then fusing the feature information output by the multilayer perceptron to obtain fused features, includes: The multilayer perceptron extracts the RGB features of the target patch and the point cloud features of the target block corresponding to the same location to obtain the first feature information corresponding to the target patch and the second feature information corresponding to the target block. The first feature information and the second feature information are fused to obtain the fused feature.

9. The method according to claim 1, characterized in that, The step of predicting anomalies in the object to be detected based on the RGB features, the point cloud features, and the fusion features to obtain an anomaly representation map corresponding to the object to be detected includes: A first anomaly segmentation map score corresponding to the object to be detected is determined based on the second scoring function, the first memory bank, and the RGB features; a second anomaly segmentation map score corresponding to the object to be detected is determined based on the second scoring function, the second memory bank, and the point cloud features; and a third anomaly segmentation map score corresponding to the object to be detected is determined based on the second scoring function, the third memory bank, and the fusion features. The first abnormal segmentation map score, the second abnormal segmentation map score, and the third abnormal segmentation map score are input into the second single-class support vector machine to obtain the segmentation map with anomalies corresponding to the object to be detected. The segmentation maps with anomalies are marked with a color that distinguishes them from normal segmentation maps to obtain the anomaly characterization map.

10. A three-dimensional industrial anomaly detection device, characterized in that, include: The preprocessing module acquires the RGB image and point cloud map corresponding to the object to be detected, and preprocesses the point cloud map to obtain multiple point cloud groups; The registration module is used to input the point cloud group into a first feature extractor for feature extraction to obtain a point cloud feature group, generate a point cloud feature map based on the point cloud feature group and the point cloud map, and project the point cloud feature map to form a point cloud projection map with the same size as the RGB image; and input the RGB image into a second feature extractor for feature extraction to obtain RGB features. The fusion module is used to input the RGB features and the point cloud features in the point cloud projection map into the multilayer perceptron for feature extraction, and to perform fusion processing on the feature information output by the multilayer perceptron to obtain fused features; The decision module is used to determine a first anomaly score corresponding to the object to be detected based on a first scoring function, a first memory, and the RGB features; to determine a second anomaly score corresponding to the object to be detected based on the first scoring function, a second memory, and the point cloud features; and to determine a third anomaly score corresponding to the object to be detected based on the first scoring function, a third memory, and the fusion features. The first anomaly score, the second anomaly score, and the third anomaly score are input into a first single-class support vector machine to obtain anomaly scores corresponding to the object to be detected. The first memory stores the original color features of the detected object, the second memory stores the point cloud features of the detected object, and the third memory stores the fusion features of the detected object.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional industrial anomaly detection method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, include: processor; as well as Memory, used to store instructions; The processor executes the instructions stored in the memory to implement the three-dimensional industrial anomaly detection method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed on a computer, cause the computer to perform the three-dimensional industrial anomaly detection method according to any one of claims 1 to 9.

Images