Multi-modal preprocessing method and device for industrial anomaly detection and storage medium

Abstract

The embodiment of the invention discloses a multi-modal preprocessing method and device for industrial anomaly detection and a storage medium, relates to the technical field of intelligent manufacturing, and can reduce the consumption of computing resources such as a CPU (Central Processing Unit) and a memory in the process of preprocessing 3D point cloud. The method comprises the steps that a black area in an original RGB image is recognized through a certain threshold value, corresponding background points in point cloud of the original RGB image are eliminated, and the point cloud with only a main body reserved is obtained and used for subsequent training and reasoning. Compared with a traditional preprocessing method, the method has the advantages that the preprocessing time is reduced by 151 times, the CPU use is reduced by 5 times, and the memory consumption is reduced by 18 times. When the preprocessed data is used for training the most advanced method, the I-AUROC is improved by 1.2%, and the difference of other indexes is only smaller than 0.005. According to the scheme of the embodiment, consumption of computing resources such as a CPU and a memory can be reduced in the process of preprocessing the 3D point cloud.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing technology, and in particular to a multimodal preprocessing method, device and storage medium for industrial anomaly detection. Background Art

[0002] Industrial anomaly detection is critical to ensuring product quality by identifying abnormal features or defects. Existing industrial anomaly detection methods mainly analyze RGB images, and there are inherent limitations in relying solely on color information. Changes in lighting conditions may cause errors, and subtle color differences that indicate anomalies may not be detectable by the human eye or standard cameras. To address these issues, researchers have turned to multimodal industrial anomaly detection, which utilizes 3D point clouds and RGB images. By combining the geometric information of point clouds, multimodal industrial anomaly detection methods can capture complex surface details and detect anomalies that may be missed by color-based methods alone. In the current 3D anomaly detection research field, there is a large amount of 3D point cloud data extracted from different scenes, and these collected point cloud data may be subject to various interferences, such as sensor errors, environmental noise, and motion vibrations of the acquisition equipment, resulting in various types of noise in the data. These noises will affect the subsequent anomaly detection algorithm. Therefore, in order to accurately process 3D point cloud data and ensure the effectiveness of anomaly detection, it is crucial to eliminate all background noise. This is the preprocessing process of 3D point clouds, which is the basis for efficient and accurate anomaly detection.

[0003] The traditional data preprocessing method is to use a combination of RANSAC and DB-Scan algorithms for preprocessing without interfering with the RGB data. This method first makes a simplifying assumption: the objects in the dataset are placed in the center of the image and the boundaries of all images are on the same plane. Then, a 10-pixel-wide strip around the image boundary is extracted from the ordered point cloud, and RANSAC is used to approximate the plane that best describes the boundary. The distance from each point in the point cloud to the plane is then calculated, and the point cloud is formed into multiple connected components. After that, by running DB-Scan as a connected component method, each cluster is regarded as a connected component, only the largest connected component is retained, and all points included in other connected components are deleted. Although this preprocessing method is effective, it is very time-consuming, and it takes an average of about 4 seconds to process a single sample pair. Therefore, the current mainstream multimodal industrial anomaly detection method chooses to preprocess all samples before performing anomaly detection, which is not feasible in real industrial production lines that need to perform real-time inference from field data. In addition, this method consumes a lot of CPU and memory resources, which may not be feasible in actual industrial production environments, making traditional methods difficult to deploy on production lines.

[0004] Therefore, how to reduce the consumption of computing resources such as CPU and memory during the preprocessing of 3D point clouds has become a topic that needs to be studied. Summary of the invention

[0005] Embodiments of the present invention provide a multimodal preprocessing method, device and storage medium for industrial anomaly detection, which can reduce the consumption of computing resources such as CPU and memory during the preprocessing of 3D point clouds.

[0006] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method comprising:

[0008] Step S1: acquiring point cloud data of an image to be processed from a data set corresponding to an industrial scene, wherein the image data of the image to be processed is in RGB format;

[0009] Step S2: preprocessing the point cloud data, the preprocessing includes: grayscale conversion, Mask generation and point cloud processing; S2 also includes: identifying the black area in the image data in RGB format according to the preset recognition threshold, filtering the color area in the image data, and eliminating the background points in the point cloud data corresponding to the black area; the remaining image data and point cloud data are combined into an image point cloud pair {T i ,P i}, where T i represents the image data with sequence number i, P i Indicates the corresponding T i Point cloud data, i∈{1,...,N}, i represents the digital sequence, and N is a positive integer.

[0010] Step S3: transmitting the preprocessed point cloud data and the image to be processed to a downstream processing module, wherein the downstream processing module includes an anomaly detection unit, and a program for detecting image anomalies is run in the anomaly detection unit.

[0011] In a second aspect, an embodiment of the present invention provides a device, comprising:

[0012] A data extraction module, used to obtain point cloud data of an image to be processed from a data set corresponding to an industrial scene, wherein the image data of the image to be processed is in RGB format;

[0013] A preprocessing module, used for preprocessing the point cloud data, wherein the preprocessing includes: a grayscale conversion link, a Mask generation link and a point cloud processing link;

[0014] The downstream processing module is used to receive the pre-processed point cloud data and the image to be processed. The downstream processing module includes an anomaly detection unit, and a program for detecting image anomalies is run in the anomaly detection unit.

[0015] In a third aspect, an embodiment of the present invention provides a storage medium storing a computer program or instructions, which implements the method in the embodiment when the computer program or instructions are executed.

[0016] The multimodal preprocessing method, device and storage medium for industrial anomaly detection provided by the embodiment of the present invention include: using a certain threshold to identify the "black area" in the original RGB image, and eliminating the corresponding background points in its point cloud, so as to obtain a point cloud that only retains the main body for subsequent training and reasoning. The method is combined with the existing anomaly detection method to form an end-to-end multimodal anomaly detection framework. Compared with the traditional preprocessing method. This method reduces the preprocessing time by 151 times, the CPU usage by 5 times, and the memory consumption by 18 times. When the preprocessed data is used to train the most advanced method, the I-AUROC is improved by 1.2%, while the difference in other indicators is only less than 0.005. Therefore, the scheme of this embodiment can reduce the consumption of computing resources such as CPU and memory in the process of preprocessing 3D point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A workflow diagram for performing anomaly detection on an MVTec 3D-AD dataset provided by an embodiment of the present invention;

[0019] Figure 2 A flowchart of the preprocessing method provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a process for generating a mask matrix according to an embodiment of the present invention;

[0021] Figure 4 A comparison diagram of the effects of the embodiment of the present invention and the traditional pretreatment method;

[0022] Figure 5 A schematic diagram of the main process provided by an embodiment of the present invention;

[0023] Figure 6 A schematic diagram of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention will be described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. It can be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "one", "said" and "the" used herein may also include plural forms. It should be further understood that the term "including" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used here may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0025] The traditional data preprocessing method is to use a combination of RANSAC and DB-Scan algorithms for preprocessing without interfering with the RGB data. This method first makes a simplifying assumption: the objects in the dataset are placed in the center of the image and the boundaries of all images are on the same plane. Then, a 10-pixel-wide strip around the image boundary is extracted from the ordered point cloud, and RANSAC is used to approximate the plane that best describes the boundary. The distance from each point in the point cloud to the plane is then calculated, and the point cloud is formed into multiple connected components. After that, by running DB-Scan as a connected component method, each cluster is regarded as a connected component, only the largest connected component is retained, and all points included in other connected components are deleted. Although this preprocessing method is effective, it is very time-consuming, and it takes an average of about 4 seconds to process a single sample pair. Therefore, the current mainstream multimodal industrial anomaly detection method chooses to preprocess all samples before performing anomaly detection, which is not feasible in real industrial production lines that need to perform real-time inference from field data. In addition, this method consumes a lot of CPU and memory resources, which may not be feasible in actual industrial production environments, making traditional methods difficult to deploy on production lines. In addition, when performing DB-Scan in traditional methods, only the largest connected component is retained. However, after applying RANSAC, the inspected object may be incorrectly divided into multiple connected components due to possible defects such as splitting at the edges, and some smaller connected components are directly discarded, which causes the inspected object to lose part of the original foreground subject, which will affect the accuracy of subsequent anomaly detection. Therefore, how to reduce the consumption of time, CPU and memory while reducing the loss of original data when preprocessing 3D point clouds has become a topic that needs to be studied.

[0026] In response to the above-mentioned topics that need to be studied, this embodiment designs a simple and effective multimodal preprocessing method for industrial anomaly detection, which can improve problems such as high latency and high power consumption when using traditional preprocessing methods, and can also realize real-time end-to-end anomaly detection in industrial scenarios while maintaining high accuracy and high efficiency.

[0027] The embodiment of the present invention provides a multimodal preprocessing method for industrial anomaly detection, such as Figure 5 As shown, including:

[0028] Step S1: acquiring point cloud data of an image to be processed from a data set corresponding to an industrial scene, wherein the image data of the image to be processed is in RGB format;

[0029] Step S2: preprocessing the point cloud data, the preprocessing includes: grayscale conversion, Mask generation and point cloud processing;

[0030] Step S3: The preprocessed point cloud data and the image to be processed are transmitted to a downstream processing module, wherein the downstream processing module includes an anomaly detection unit, in which a program for detecting image anomalies is run, so as to obtain a detection result by training and reasoning through EasyNet.

[0031] In practical applications, there may be most unnecessary background parts (for example, irrelevant to the current business) in 3D point cloud data, and the main part that really needs attention needs to be screened and extracted from the 3D point cloud data. Therefore, preprocessing may be performed before using the 3D point cloud data for other downstream tasks. This embodiment is an improvement on the preprocessing method of 3D point cloud, and the subsequent downstream task is anomaly detection. The downstream processing module described in this embodiment actually refers to the processing module used to implement downstream tasks. Such downstream tasks and corresponding processing models, business procedures, etc. can be implemented using currently available technologies. For example, in addition to the downstream task of anomaly detection performed in this embodiment, specific downstream tasks may also include: object detection and positioning, object classification, segmentation, semantic segmentation, 3D reconstruction and modeling, etc., and the specific implementation methods of these downstream tasks actually adopt existing technical solutions, so they are not repeated in this embodiment.

[0032] The general design idea of ​​preprocessing the point cloud in step S2 is: specifically, the original RGB image is identified by using a certain threshold value in the "black area", and the corresponding background points in its point cloud are eliminated, so that only the point cloud of the subject is retained for subsequent training and reasoning. Suppose we have a dataset of image point cloud pairs, where the image and point cloud are represented as T and P respectively. Given any image point cloud pair i∈{1,...,N}{T i ,P i}. The specific execution process includes the following steps: grayscale conversion, Mask generation and point cloud processing.

[0033] Specifically, the grayscale conversion process includes: The image data T i Convert to grayscale image toGray() is a function for converting a color image into a grayscale image. In this embodiment, the cvtColor() function in the OpenCV2 library can be used as a grayscale conversion function.

[0034] The Mask generation process includes: Generate a mask indicating the boundary between background and foreground, where represents a grayscale image, σ represents the threshold and 0<σ<255, and the mask mask is A binary matrix with the same dimension. If the pixel value in the grayscale image is greater than or equal to the threshold σ, the value of the corresponding position in the mask matrix is ​​1, otherwise it is 0. .

[0035] The point cloud processing step includes: processing the point cloud data Pi according to the mask mask to obtain the part of the point cloud corresponding to the 1 in the mask matrix, wherein: represents the processed point cloud, and × represents element-by-element multiplication.

[0036] In the preferred embodiment, the data set corresponding to the industrial scene adopts the MVTec 3D-AD data set; the anomaly detection method in EasyNet is run in the anomaly detection unit. Among them, EasyNet (An Easy Network for 3D Industrial Anomaly Detection, a simple network for 3D industrial anomaly detection) is a complete framework applied to the field of 3D anomaly detection, which can be designed as an anomaly detection unit in the process of implementation through code programming. After preprocessing, the processed data is handed over to the EasyNet framework for detection, and the subsequent comparative experiment is to send the two preprocessed data into the framework respectively to compare the experimental results. Therefore, this embodiment is called "the anomaly detection method in EasyNet", or it can also be called "the anomaly detection method in the recent 3D industrial anomaly detection framework EasyNet".

[0037] This embodiment of the invention provides a multi-modal preprocessing device for industrial anomaly detection. Figure 6 As shown, including:

[0038] A data extraction module, used to obtain point cloud data of an image to be processed from a data set corresponding to an industrial scene, wherein the image data of the image to be processed is in RGB format;

[0039] A preprocessing module, used for preprocessing the point cloud data, wherein the preprocessing includes: a grayscale conversion link, a Mask generation link and a point cloud processing link;

[0040] The downstream processing module is used to receive the pre-processed point cloud data and the image to be processed. The downstream processing module includes an anomaly detection unit, and a program for detecting image anomalies is run in the anomaly detection unit.

[0041] The preprocessing module is further used to identify the black area in the image data in RGB format according to a preset recognition threshold, filter the color area in the image data, and eliminate the background points in the point cloud data corresponding to the black area; and combine the remaining image data and point cloud data into an image point cloud pair. i ,Pi}, where T i represents the image data with sequence number i, P o Indicates the corresponding T o Point cloud data, i∈{1,...,N}, i represents the digital sequence, and N is a positive integer.

[0042] The preprocessing module includes: a grayscale conversion unit, a Mask generation unit and a point cloud processing unit;

[0043] The grayscale conversion unit is used to convert the grayscale model The image data T o Convert to grayscale image toGray() is a function that converts a color image to a grayscale image;

[0044] The Mask generation unit is used to generate a mask model Generate a mask, where represents a grayscale image, σ represents the threshold and 0<σ<255, and the mask mask is A binary matrix with the same dimension, where the element with value 1 in the matrix represents the corresponding grayscale, the value is greater than or equal to the threshold σ, otherwise it is 0;

[0045] The point cloud processing unit is used to process the point cloud data Pi according to the mask mask to obtain the part of the point cloud corresponding to 1 in the mask matrix, wherein: represents the processed point cloud, and × represents element-by-element multiplication.

[0046] The preprocessing method of this embodiment can be combined with the existing anomaly detection method to form an end-to-end multimodal anomaly detection framework. The general design idea is: analyze the data set acquisition method of various detection methods, find the relevant parts in the code, and modify the preprocessing code appropriately according to the data set dimensions required for various anomaly detection. Finally, the preprocessing method is integrated into the data set acquisition process, and the whole process forms an end-to-end process. In specific applications, the operating environment of the examples of the present invention is carried out on a single Nvidia RTX 4090 GPU and Intel (R) Xeon (R) Platinum 8352V CPU. At the same time, in order to evaluate the anomaly localization and image-level anomaly detection capabilities of the inventive method combined with various anomaly detection methods, we selected three common indicators: for image-level detection, the standard image-level AUROC is used; for pixel-level detection, the average pixel-level AUROC is used for positioning evaluation; for anomaly localization, the PRO metric is used.

[0047] The following examples 1 and 2 are listed to demonstrate the superiority of this example solution compared with the prior art:

[0048] Example 1

[0049] The MVTec 3D-AD dataset used in this example is sampled from real industrial scenes. It is more suitable for industrial environments and is widely used by researchers in the field of 3D anomaly detection. However, due to the presence of many distractors in the point cloud data, the MVTec 3D-AD dataset cannot be used directly for training. In order to accurately process 3D data and ensure the effectiveness of anomaly detection, it is crucial to eliminate all background noise.

[0050] Reference Figure 1 As shown in the flowchart, the present invention proposes a simple and effective multimodal preprocessing method for industrial anomaly detection. This example uses it in the MVTec 3D-AD dataset and combines it with the anomaly detection method in EasyNet to be applied in the field of industrial anomaly detection. Specifically, it includes the following steps:

[0051] Step 1: Get the point cloud of the preprocessed image and the RGB format image in the MVTec 3D-AD dataset and use the zip function to iterate both at the same time.

[0052] Step 2: Reference Figure 2 As shown in the flowchart, the point cloud is preprocessed. The design idea is to use a certain threshold to identify the "black area" in the original RGB image and eliminate the corresponding background points in its point cloud, so as to obtain a point cloud that only retains the subject for subsequent training and reasoning. Suppose we have a dataset of N image point cloud pairs, where the image and point cloud are represented as I and P respectively. Given any image point cloud pair {T o ,P o}. Specifically include the following steps:

[0053] Step 2.1: Grayscale conversion

[0054] By using the method provided by the OpenCV framework to convert the image into RGB format, the subsequent processing steps are simplified and the computational complexity is reduced.

[0055] Step 2.2: Mask generation

[0056] A mask is generated by setting a threshold σ between 0 and 255 to indicate the boundary between the background and the foreground. The mask is a binary matrix of the same dimension as the grayscale image obtained previously. If the grayscale value is greater than or equal to the threshold, the corresponding mask matrix part takes 1, otherwise it takes 0. For detailed process, refer to Figure 3 As shown in the flow chart.

[0057] Step 2.3: Point cloud processing

[0058] The mask matrix obtained above is multiplied element by element with the original point cloud. If it is multiplied by 1, it is retained, and if it is multiplied by 0, it is removed. The points with a value of 1 in the corresponding mask in the point cloud are retained. In the end, only the foreground main part of the point cloud is left, which is the preprocessing result. The comparison between the effect after the preprocessing method and the traditional preprocessing method is shown in the figure below. Figure 4 shown.

[0059] When preprocessing the MVTec 3D-AD dataset in this example, we used the psutil package provided by Python to monitor the CPU and memory usage, and used the time package to record the time. The results are shown in Table 1. Our invented method reduces the preprocessing time by 151 times, the CPU usage by 5 times, and the memory consumption by 18 times.

[0060] Table 1. Time, CPU usage and memory usage records of traditional preprocessing methods and methods

[0061]

[0062] Step S3: The obtained preprocessed point cloud data and the original RGB image are sent to the anomaly detection method for training and reasoning. In this example, we only need to replace the data set used by the EasyNet method with our preprocessed data set, and then we can seamlessly carry out subsequent training and reasoning to obtain the detection results. The final detection results are statistically analyzed, as shown in Table 2. Compared with the traditional preprocessing method,

[0063] The I-AUROC improves by 1.2%, while the differences in other metrics are only less than 0.005.

[0064] Table 2 Detection results of the EasyNet anomaly detection method using traditional preprocessing methods and methods

[0065]

[0066] Example 2

[0067] Example 2 is an optimization and supplement based on Example 1.

[0068] In this example, we organically combine the data preprocessing process with the algorithm reasoning process, simulate the entire workflow of industrial data collection and anomaly detection in real industrial scenarios, and build an end-to-end anomaly detection framework that meets the needs of industrial manufacturing. The specific implementation ideas are:

[0069] We analyzed the corresponding data acquisition codes in the training and test datasets in the EasyNet anomaly detection method, made appropriate modifications to the transform_image method call part, modified the required dataset dimensions and directly called the preprocessing method, integrated the preprocessing method into the dataset acquisition process, and the entire process became an end-to-end process instead of the traditional two-stage process (preprocessing the dataset first, saving the preprocessed dataset, and then training and reasoning the preprocessed dataset). The FPS (number of samples processed per second) in the statistical reasoning process is shown in Table 3. Our final FPS increased by 36 times.

[0070] Table 3 FPS of the traditional preprocessing method and the method on the EasyNet anomaly detection method

[0071]

[0072] Through end-to-end integration, combined with the advantages of low latency and low power consumption of preprocessing methods, a real-time end-to-end multimodal industrial anomaly detection framework can be formed, which is convenient for deployment in real industrial environments.

[0073] It can be seen from this that the solution of this example solves the problems of high latency and high power consumption when using traditional preprocessing methods, and can realize real-time end-to-end anomaly detection in industrial scenarios while maintaining high accuracy and efficiency. Among them, the original RGB image is used to identify the "black area" in it by using a certain threshold, and the corresponding background points in its point cloud are eliminated, so that only the point cloud retaining the subject is obtained for subsequent training and reasoning. The method is combined with the existing anomaly detection method to form an end-to-end multimodal anomaly detection framework. Compared with the traditional preprocessing method. This method reduces the preprocessing time by 151 times, the CPU usage by 5 times, and the memory consumption by 18 times. After the method of the present invention is seamlessly integrated with the existing anomaly detection method, a real-time end-to-end multimodal industrial anomaly detection framework is formed, which can increase the FPS by up to 36 times compared with the traditional anomaly detection method. When the preprocessed data is used to train the most advanced method, the I-AUROC is improved by 1.2%, while the differences in other indicators are only less than 0.005.

[0074] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal preprocessing method for industrial anomaly detection, characterized in that: include: Step S1: The computer obtains point cloud data of the image to be processed from a data set corresponding to the industrial scene, wherein the image data of the image to be processed is in RGB format; Step S2: preprocessing the point cloud data, the preprocessing includes: grayscale conversion, Mask generation and point cloud processing; Step S3: transmitting the preprocessed point cloud data and the image to be processed to a downstream processing module, wherein the downstream processing module includes an anomaly detection unit, and a program for detecting image anomalies is run in the anomaly detection unit.

2. The method according to claim 1, characterized in that Also included in S2: According to a preset recognition threshold, a black area in the image data in RGB format is identified, the black area in the image data is filtered, and background points in the point cloud data corresponding to the black area are eliminated, wherein the value of the pixel point in the black area is 0; The remaining image data and point cloud data are combined into an image point cloud pair {T i ,P i }, where T i represents the image data with sequence number i, P i Indicates the corresponding T i Point cloud data, i∈{1,...,N}, i represents the digital sequence, and N is a positive integer.

3. The method according to claim 1 or 2, characterized in that: The grayscale conversion process includes: using a grayscale conversion model T i gray = toGray(T i ), the image data T i Convert to grayscale image T i gray , toGray() is a function that converts a color image to a grayscale image.

4. The method according to claim 3, characterized in that The Mask generation process includes: Through the mask model mask = T i gray ≥σ, generate mask mask, where T i gray represents a grayscale image, σ represents the threshold and 0<σ<255, mask mask and T i gray A binary matrix with the same dimension. If the pixel value in the grayscale image is greater than or equal to the threshold σ, the value of the corresponding position in the mask matrix is ​​1, otherwise it is 0.

5. The method according to claim 4, characterized in that The point cloud processing step includes: Process the point cloud data Pi according to the mask mask to obtain the part of the point cloud corresponding to the 1 in the mask matrix, where P i proc =mask×P i , P i proc represents the processed point cloud, and × represents element-by-element multiplication.

6. The method according to claim 1, characterized in that The data set corresponding to the industrial scene adopts the MVTec3D-AD data set; The anomaly detection unit runs the anomaly detection method in EasyNet.

7. A multimodal preprocessing device for industrial anomaly detection, characterized in that: include: A data extraction module, used to obtain point cloud data of an image to be processed from a data set corresponding to an industrial scene, wherein the image data of the image to be processed is in RGB format; A preprocessing module, used for preprocessing the point cloud data, wherein the preprocessing includes: a grayscale conversion link, a Mask generation link and a point cloud processing link; The downstream processing module is used to receive the pre-processed point cloud data and the image to be processed. The downstream processing module includes an anomaly detection unit, and a program for detecting image anomalies is run in the anomaly detection unit.

8. The device according to claim 1, characterized in that The preprocessing module is further used to identify the black area in the image data in RGB format according to a preset recognition threshold, filter the color area in the image data, and eliminate the background points in the point cloud data corresponding to the black area; The remaining image data and point cloud data are combined into an image point cloud pair {T i ,P i }, where T i represents the image data with sequence number i, P i Indicates the corresponding T i Point cloud data, i∈{1,...,N}, i represents the digital sequence, and N is a positive integer.

9. The device according to claim 1 or 2, characterized in that: The preprocessing module includes: a grayscale conversion unit, a Mask generation unit and a point cloud processing unit; The grayscale conversion unit is used to convert the grayscale model T i gray = toGray(T i ), the image data T i Convert to grayscale image T i gray , toGray() is a function that converts a color image into a grayscale image; The Mask generating unit is used to generate a mask model mask=T i gray ≥σ, generate mask mask, where T i gray represents a grayscale image, σ represents the threshold and 0<σ<255, mask mask and T i gray A binary matrix with the same dimension, where the element with value 1 in the matrix represents the corresponding grayscale, the value is greater than or equal to the threshold σ, otherwise it is 0; The point cloud processing unit is used to process the point cloud data Pi according to the mask mask to obtain the part of the point cloud corresponding to 1 in the mask matrix, where P i proc =mask×P i , P i proc represents the processed point cloud, and × represents element-by-element multiplication.

10. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is executed, the method according to any one of claims 1 to 6 is implemented.

Images