Industrial defect detection method, system, device and storage medium

Abstract

The present invention discloses an industrial defect detection method, system, device and storage medium, which relate to the field of automation technology, including collecting multimodal data of workpieces through a high-resolution visible light camera, an infrared camera and an ultrasonic sensor, and performing preprocessing; using a multimodal data fusion algorithm to extract and fuse features of the preprocessed data to generate a comprehensive feature map; analyzing the lighting conditions of the comprehensive feature map through computer vision technology, dynamically adjusting the lighting conditions, and generating a corrected comprehensive feature map; constructing a defect detection model, performing instance segmentation and target detection on the corrected comprehensive feature map, generating a pre-selected box and marking possible defect areas; introducing an online learning mechanism, and updating and optimizing the defect detection model in real time according to newly collected data; through multimodal data fusion, dynamic lighting condition correction and online learning mechanism, defects can be accurately identified and located, and the accuracy of industrial defect detection is significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of automation technology, and in particular to an industrial defect detection method, system, device and storage medium. Background Art

[0002] In recent years, with the rapid development of manufacturing industry and the continuous advancement of automation technology, industrial defect detection has become a key link in improving product quality and production efficiency. Traditional defect detection methods mainly rely on manual visual inspection, which is not only time-consuming and labor-intensive, but also easily affected by human factors, resulting in inconsistency and low reliability of detection results. With the rapid development of computer vision and machine learning technology, image-based automatic defect detection methods have gradually become a research hotspot.

[0003] However, existing defect detection methods still have some shortcomings, especially in terms of multimodal data fusion, illumination condition correction, and real-time model updating. First, although multimodal data fusion technology can improve the accuracy of detection, existing fusion methods often rely on fixed feature extraction methods and lack dynamic adaptability to different modal data, resulting in poor detection results under complex working conditions. Secondly, changes in lighting conditions have a significant impact on image quality. Existing illumination correction methods mostly use static correction strategies, which cannot adapt to environmental changes in real time, resulting in unstable detection results. In addition, existing defect detection models often need to be retrained when faced with new defect types and changes in working conditions, and lack the ability to update and optimize in real time. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an industrial defect detection method, system, device and storage medium to solve the problems of insufficient multimodal data fusion, influence of lighting conditions and poor model adaptability in existing industrial defect detection methods.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides an industrial defect detection method, which includes:

[0008] Collect multimodal data of visible light images, infrared images and ultrasonic images of the workpiece to be inspected, and perform preprocessing;

[0009] Using a multimodal data fusion algorithm, the preprocessed multimodal data of visible light images, infrared images and ultrasonic images are fused to generate a comprehensive feature map;

[0010] Use computer vision technology to analyze the lighting conditions of the image, dynamically adjust the lighting conditions, and obtain a corrected comprehensive feature map;

[0011] Build a defect detection model, perform instance segmentation and target detection on the corrected comprehensive feature map, generate pre-selected boxes and mark possible defect areas;

[0012] According to the inspection results, the defect information of the workpiece is output and visualized to generate annotated images;

[0013] An online learning mechanism is introduced to update and optimize the defect detection model in real time based on the newly acquired multimodal data of visible light images, infrared images and ultrasonic images.

[0014] As a preferred solution of the industrial defect detection method of the present invention, wherein: collecting multimodal data of visible light image, infrared image and ultrasonic image of the workpiece to be detected and preprocessing includes the following steps:

[0015] Using a high-resolution visible light camera to capture a visible light image of the workpiece;

[0016] Use an infrared camera to take infrared images of the workpiece and capture temperature distribution information;

[0017] Use ultrasonic sensors to scan workpieces and obtain ultrasonic images of internal structural information;

[0018] Grayscale, denoise and contrast enhance visible light images, perform temperature calibration and generate thermal maps for infrared images, perform signal processing on ultrasonic images and generate two-dimensional images;

[0019] Unify all preprocessed images to the same size and format.

[0020] As a preferred solution of the industrial defect detection method of the present invention, the multimodal data fusion algorithm is used to fuse the preprocessed multimodal data of the visible light image, infrared image and ultrasonic image to generate a comprehensive feature map, which includes the following steps:

[0021] Convert visible light images, infrared images, and ultrasound images into tensor form;

[0022] Use convolutional neural networks to extract feature maps for each modality;

[0023] Use a linear projection layer to adjust the number of channels of each feature map to a uniform dimension and generate a new feature map;

[0024] Each new feature map after mapping is concatenated to form a multimodal feature map;

[0025] A multi-head self-attention mechanism is used to fuse multimodal feature maps to generate a comprehensive feature map.

[0026] As a preferred solution of the industrial defect detection method of the present invention, wherein: using computer vision technology to analyze the lighting conditions of the image, dynamically adjusting the lighting conditions, and obtaining a corrected comprehensive feature map includes the following steps:

[0027] The global characteristics of the image are quantified by calculating the average brightness and average contrast of the image to obtain the global illumination information of the image;

[0028] A local binary pattern algorithm is used to capture the relative brightness changes between pixels and their neighboring pixels in the global illumination information image, identify the locations of highlight areas and dark areas, and generate a binary image;

[0029] Determine the current light intensity and direction based on the average brightness, average contrast, and the positions of the identified highlight and dark areas;

[0030] The image is processed by using illumination correction technology to generate a corrected image;

[0031] The corrected image is subjected to histogram equalization to generate a corrected comprehensive feature map.

[0032] As a preferred solution of the industrial defect detection method of the present invention, wherein: constructing a defect detection model, performing instance segmentation and target detection on the corrected comprehensive feature map, generating a pre-selected box and marking possible defect areas include the following steps:

[0033] Input the corrected comprehensive feature map into the backbone network;

[0034] Using the top-down path and lateral connections of the feature pyramid, high-level low-resolution feature maps are combined with low-level high-resolution feature maps to generate multi-scale feature maps;

[0035] Slide a window on each feature map to generate a fixed number of anchor boxes;

[0036] Classify and regress each anchor box to determine whether it contains the target and adjust its position and size;

[0037] Select anchor boxes with higher scores as candidate regions;

[0038] Use the region of interest pooling layer to resize candidate regions of different sizes to the same size;

[0039] Classify each candidate region through the fully connected layer to determine which category it belongs to;

[0040] The position of the candidate area is further adjusted through bounding box regression to generate the final pre-selected box;

[0041] Generate a segmentation mask for each candidate region through the segmentation branch;

[0042] Generate a pre-selected box for each candidate region classified as foreground;

[0043] Keep the pre-selected boxes with higher confidence and remove the background and pre-selected boxes with low confidence;

[0044] For each retained pre-selected box, obtain its corresponding confidence score from the classification branch and sort them from high to low to obtain a sorted pre-selected box list;

[0045] Initialize an empty list and set the confidence score threshold;

[0046] When the confidence score is greater than the threshold, the pre-selected box is added to the empty list;

[0047] Calculate the intersection and union ratio of the pre-selected box and other unprocessed pre-selected boxes. The expression is:

[0048] ;

[0049] in, is the i-th pre-selected box, is the jth pre-selected box, is the intersection-over-union ratio of the i-th pre-selected box and the j-th pre-selected box, For two pre-selected boxes and The intersection area of For two pre-selected boxes and The area of ​​the union of ;

[0050] Remove all pre-selected boxes whose intersection-over-union ratio with the pre-selected box is greater than the set threshold;

[0051] When all pre-selected boxes have been processed, the final list of retained pre-selected boxes is returned;

[0052] The pre-selected boxes remaining in the pre-selected box list are marked as possible defect areas.

[0053] As a preferred solution of the industrial defect detection method of the present invention, wherein: according to the detection results, the defect information of the workpiece is output and visualized, and the generation of the annotated image includes the following steps:

[0054] Read the original image and obtain the pre-selected box, classification label and confidence score of each defect area from the processed pre-selected box list;

[0055] Draw the bounding box and classification label of each defect area on the original image, and save the drawn image as an annotated image;

[0056] The severity of the defect is calculated based on the confidence score, and the expression is:

[0057] ;

[0058] in, is the severity of the defect, is the weight coefficient of the confidence score, is the confidence score of the i-th defect, is the area of ​​the i-th pre-selected box, is the original image, is the area of ​​the original image, is the weight coefficient of the defect area ratio;

[0059] Classify workpieces into acceptable and unacceptable based on the severity of defects;

[0060] When the workpiece is classified as qualified, the workpiece is sent to the qualified material box;

[0061] When the workpiece is classified as unqualified, the workpiece is sent to the unqualified material box;

[0062] The information of the defect location, type, confidence score and severity is saved to the database, and the annotated image is saved to the specified path.

[0063] As a preferred solution of the industrial defect detection method of the present invention, wherein: introducing an online learning mechanism, updating and optimizing the defect detection model in real time according to the newly collected multimodal data of visible light images, infrared images and ultrasonic images includes the following steps:

[0064] Continuously collect new multimodal data of visible light images, infrared images, and ultrasound images, use the online stochastic gradient descent algorithm to gradually update the model weights and regularly evaluate the model performance;

[0065] Adjust the learning rate and batch size based on the evaluation results;

[0066] Use transfer learning technology to migrate existing large-scale labeled data to new detection tasks to adapt to new defect types and changes.

[0067] In a second aspect, the present invention provides an industrial defect detection system, comprising:

[0068] The data acquisition and preprocessing module acquires multimodal data of visible light images, infrared images and ultrasonic images of the workpiece to be inspected, and performs preprocessing;

[0069] The multimodal data fusion module uses a multimodal data fusion algorithm to fuse the preprocessed multimodal data of visible light images, infrared images, and ultrasonic images to generate a comprehensive feature map;

[0070] The lighting condition analysis and correction module uses computer vision technology to analyze the lighting conditions of the image, dynamically adjusts the lighting conditions, and obtains the corrected comprehensive feature map;

[0071] The defect detection model building module builds the defect detection model, performs instance segmentation and target detection on the corrected comprehensive feature map, generates pre-selected boxes and marks possible defect areas;

[0072] The inspection result visualization module outputs the defect information of the workpiece and visualizes it according to the inspection results to generate annotated images;

[0073] The online learning mechanism module introduces an online learning mechanism to update and optimize the defect detection model in real time based on the newly collected multimodal data of visible light images, infrared images and ultrasonic images.

[0074] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the industrial defect detection method as described in the first aspect of the present invention is implemented.

[0075] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the industrial defect detection method as described in the first aspect of the present invention is implemented.

[0076] The beneficial effects of the present invention are as follows: multimodal data of a workpiece is collected and preprocessed by a high-resolution visible light camera, an infrared camera and an ultrasonic sensor; a multimodal data fusion algorithm is used to extract and fuse features of the preprocessed data to generate a comprehensive feature map; the lighting conditions of the comprehensive feature map are analyzed by computer vision technology, the lighting conditions are dynamically adjusted, and a corrected comprehensive feature map is generated; a defect detection model is constructed, instance segmentation and target detection are performed on the corrected comprehensive feature map, a preselection box is generated and possible defect areas are marked; an online learning mechanism is introduced to update and optimize the defect detection model in real time according to the newly collected data; through multimodal data fusion, dynamic lighting condition correction and online learning mechanism, defects can be accurately identified and located, and the accuracy of industrial defect detection is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0078] Figure 1 This is a flow chart of the industrial defect detection method in Example 1.

[0079] Figure 2 This is a flow chart of the industrial defect detection system in Example 1. DETAILED DESCRIPTION

[0080] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0081] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0082] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0083] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides an industrial defect detection method, comprising the following steps:

[0084] S1, collecting multimodal data of visible light image, infrared image and ultrasonic image of the workpiece to be detected, and preprocessing includes the following steps:

[0085] Use a high-resolution visible light camera to capture visible light images of the workpiece; use an infrared camera to capture infrared images of the workpiece and capture temperature distribution information; use an ultrasonic sensor to scan the workpiece and obtain ultrasonic images of internal structure information; grayscale, denoise and contrast enhance the visible light image, perform temperature calibration and generate thermal maps for the infrared image, perform signal processing on the ultrasonic image and generate a two-dimensional image; unify all pre-processed images to the same size and format.

[0086] Furthermore, grayscale refers to converting an RGB image into a grayscale image, expressed as:

[0087] ;

[0088] Among them, R, G, and B are the pixel values ​​of the red, green, and blue channels respectively. is the pixel value of the grayscale image;

[0089] Denoising refers to using a median filter to remove noise from an image. A 3x3 window size is selected, the neighborhood of each pixel is sorted, and the median is taken as the new value of the pixel.

[0090] Contrast enhancement refers to the use of histogram equalization technology to enhance the contrast of the image, the expression is:

[0091] ;

[0092] in, is the pixel value of the enhanced image, is the horizontal coordinate of the image, is the vertical coordinate of the image, is the cumulative histogram, is the total number of pixels in the image, is the number of gray levels (usually 256);

[0093] Temperature calibration refers to the use of radiation response curves to calibrate the temperature of infrared images; thermal map generation refers to the use of pseudo-color mapping technology to map the calibrated temperature data to the color space to generate an intuitive thermal map; signal processing refers to the use of wavelet transform to decompose ultrasonic signals into multiple scales, remove high-frequency noise, and use Fourier transform to convert the denoised signal into a two-dimensional image; unified size and format refers to the use of bilinear interpolation to adjust all images to the same size and convert all images into a unified format.

[0094] S2. Using a multimodal data fusion algorithm, the pre-processed multimodal data of the visible light image, infrared image and ultrasonic image are fused to generate a comprehensive feature map, including the following steps:

[0095] Convert visible light images, infrared images, and ultrasonic images into tensor form; use convolutional neural networks to extract feature maps of each modality; use linear projection layers to adjust the number of channels of each feature map to a uniform dimension to generate new feature maps; concatenate each new feature map after mapping to form a multimodal feature map; use a multi-head self-attention mechanism to fuse the multimodal feature maps to generate a comprehensive feature map.

[0096] Furthermore, by converting data of different modalities into tensor form, all images are cropped or padded to a uniform resolution (such as 256x256), and grayscale images are converted to three-channel RGB images, and the converted images are converted into tensor form, usually a four-dimensional array with a shape of [B, C, H, W], where B represents the batch size, C represents the number of channels, and H and W represent the height and width respectively; the introduction of a multi-head self-attention mechanism allows the model to focus on the important associations between different modalities, while also capturing the dependencies between different regions. Specifically, the multimodal feature map can be first mapped to the query, key, and value vector spaces through a linear transformation, and then the attention weights are calculated, and finally the weighted sum is obtained to obtain a new feature representation. This process can be repeated many times to form multiple "heads", each of which focuses on different information.

[0097] S3. Using computer vision technology to analyze the lighting conditions of the image, dynamically adjusting the lighting conditions, and obtaining a corrected comprehensive feature map includes the following steps:

[0098] The global characteristics of the image are quantified by calculating the average brightness and average contrast of the image to obtain the global illumination information of the image; the relative brightness changes between pixels and their neighboring pixels in the global illumination information image are captured using the local binary pattern algorithm, the positions of highlight areas and dark areas are identified, and a binary image is generated; the current illumination intensity and illumination direction are determined based on the average brightness, average contrast, and the positions of the identified highlight and dark areas; the image is processed using illumination correction technology to generate a corrected image; the corrected image is histogram equalized to generate a corrected comprehensive feature map.

[0099] Furthermore, by quantifying the global illumination information of an image, it means first converting the image from the RGB color space to the grayscale space, because the grayscale image is more suitable for the calculation of brightness and contrast, and then calculating the average value of all pixel values ​​of the image as the average brightness. For the calculation of the average contrast, the brightness difference between each pixel and its surrounding pixels can be calculated first, and then the average value of these differences can be taken, so that the overall brightness level of the image and the degree of brightness change can be quantified; applying the local binary algorithm means comparing each pixel with the pixels around it, if the brightness of the central pixel is greater than or equal to the neighboring pixel, it is marked as 1, otherwise it is marked as 0. This generates an 8-bit binary number (for a 3x3 neighborhood). The binary number is converted into a decimal number as the new value of the pixel to form a binary image. In this way, the highlight and dark areas in the image can be effectively distinguished, the texture information in the image can be highlighted, especially the details such as edges and corners, which helps to accurately identify the distribution of illumination in the image; illumination correction techniques include gamma correction, linear stretching and Gaussian filtering. For example, for images that are too bright or too dark, you can use gamma correction to adjust the brightness distribution of the image; for uneven lighting, you can try using Gaussian filtering to smooth out the lighting differences.

[0100] S4, build a defect detection model, perform instance segmentation and target detection on the corrected comprehensive feature map, generate pre-selected boxes and mark possible defect areas, including the following steps,

[0101] Input the corrected comprehensive feature map into the backbone network; Use the top-down path and lateral connection of the feature pyramid to combine the high-level low-resolution feature map with the low-level high-resolution feature map to generate a multi-scale feature map; Slide a window on each feature map to generate a set of fixed number of anchor boxes; Classify and regress each anchor box to determine whether it contains the target and adjust its position and size; Select the anchor box with a higher score as the candidate region; Use the interest pooling layer to adjust the candidate regions of different sizes to the same size; Classify each candidate region through the fully connected layer to determine which category it belongs to; Bounding box regression further adjusts the position of the candidate region to generate the final pre-selected box; generates a segmentation mask for each candidate region through the segmentation branch; generates a pre-selected box for each candidate region classified as the foreground; retains the pre-selected boxes with higher confidence and removes the background and low-confidence pre-selected boxes; for each retained pre-selected box, obtains its corresponding confidence score from the classification branch and sorts them from high to low to obtain a sorted pre-selected box list; initializes an empty list and sets a confidence score threshold; when the confidence score is greater than the threshold, adds the pre-selected box to the empty list; calculates the intersection and union ratio of the pre-selected box and other unprocessed pre-selected boxes, the expression is:

[0102] ;

[0103] in, is the i-th pre-selected box, is the jth pre-selected box, is the intersection-over-union ratio of the i-th pre-selected box and the j-th pre-selected box, For two pre-selected boxes and The intersection area of For two pre-selected boxes and The area of ​​the union of ;

[0104] Remove all pre-selected boxes whose intersection-over-union ratio with the pre-selected box is greater than the set threshold; when all pre-selected boxes have been processed, return the final list of retained pre-selected boxes; mark the pre-selected boxes retained in the pre-selected box list as possible defect areas.

[0105] S5. Output the defect information of the workpiece and visualize it according to the detection results. Generating the annotated image includes the following steps:

[0106] Read the original image and obtain the pre-selected box, classification label and confidence score of each defect area from the processed pre-selected box list; draw the bounding box and classification label of each defect area on the original image, and save the drawn image as an annotated image; calculate the severity of the defect based on the confidence score, the expression is:

[0107] ;

[0108] in, is the severity of the defect, is the weight coefficient of the confidence score, is the confidence score of the i-th defect, is the area of ​​the i-th pre-selected box, is the original image, is the area of ​​the original image, is the weight coefficient of the defect area ratio;

[0109] The workpieces are classified into qualified and unqualified according to the severity of the defects; when the workpiece is classified as qualified, the workpiece is sent to the qualified material box; when the workpiece is classified as unqualified, the workpiece is sent to the unqualified material box; the information of the location, type, confidence score and severity of the defect is saved in the database, and the annotated image is saved to the specified path.

[0110] S6. Introducing an online learning mechanism to update and optimize the defect detection model in real time based on newly acquired multimodal data of visible light images, infrared images, and ultrasonic images includes the following steps:

[0111] Continuously collect new multimodal data of visible light images, infrared images, and ultrasonic images, use the online stochastic gradient descent algorithm to gradually update the model weights and regularly evaluate the model performance; adjust the learning rate and batch size based on the evaluation results; use transfer learning technology to migrate existing large-scale annotated data to new inspection tasks to adapt to new defect types and changes.

[0112] The present embodiment also provides an industrial defect detection system, including: a data acquisition and preprocessing module, which acquires multimodal data of visible light images, infrared images and ultrasonic images of a workpiece to be detected and performs preprocessing; a multimodal data fusion module, which uses a multimodal data fusion algorithm to fuse the preprocessed multimodal data of visible light images, infrared images and ultrasonic images to generate a comprehensive feature map; a lighting condition analysis and correction module, which uses computer vision technology to analyze the lighting conditions of the image, dynamically adjusts the lighting conditions, and obtains a corrected comprehensive feature map; a defect detection model construction module, which constructs a defect detection model, performs instance segmentation and target detection on the corrected comprehensive feature map, generates a pre-selected box and marks possible defect areas; a detection result visualization module, which outputs the defect information of the workpiece and visualizes it according to the detection results, and generates a labeled image; an online learning mechanism module, which introduces an online learning mechanism, and updates and optimizes the defect detection model in real time according to the newly collected visible light images, infrared images and ultrasonic images.

[0113] This embodiment also provides a computer device suitable for the case of an industrial defect detection method, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the industrial defect detection method proposed in the above embodiment.

[0114] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0115] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the industrial defect detection method proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0116] In summary, the present invention collects multimodal data of workpieces and performs preprocessing through: a high-resolution visible light camera, an infrared camera and an ultrasonic sensor; uses a multimodal data fusion algorithm to extract and fuse features of the preprocessed data to generate a comprehensive feature map; uses computer vision technology to analyze the lighting conditions of the comprehensive feature map, dynamically adjusts the lighting conditions, and generates a corrected comprehensive feature map; constructs a defect detection model, performs instance segmentation and target detection on the corrected comprehensive feature map, generates a pre-selected box and marks possible defect areas; introduces an online learning mechanism to update and optimize the defect detection model in real time according to the newly collected data; through multimodal data fusion, dynamic lighting condition correction and online learning mechanism, defects can be accurately identified and located, which significantly improves the accuracy of industrial defect detection.

[0117] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of an industrial defect detection method are provided.

[0118] A metal product manufacturing company was selected as the research object. The company mainly produces high-strength metal parts for the automotive industry, and the quality of these parts directly affects the safety and reliability of the final product. Therefore, extremely high requirements are placed on defect detection in the production process. This embodiment selected 100 metal parts samples on the production line, of which 50 were standard defect-free samples and the other 50 were samples with various types of defects known to exist, including cracks, dents, pores, etc. The size, shape and material of these samples are the same to ensure the consistency of the experimental conditions.

[0119] Before the experiment began, a detection platform integrating multiple sensors was built, including a high-resolution visible light camera, an infrared camera, and a set of ultrasonic sensors. In addition, a dedicated computer vision system and deep learning framework were deployed to perform multimodal data fusion, feature extraction, defect detection model training, and the implementation of online learning mechanisms. In order to ensure the repeatability and accuracy of the experiment, all equipment was strictly calibrated to ensure the accuracy and reliability of the measurement data.

[0120] During the experiment, each sample was first photographed by a high-resolution visible light camera to obtain a clear surface image; then, an infrared camera was used to capture the temperature distribution information of the sample to assist in identifying potential internal defects; finally, an ultrasonic sensor scanned the sample to obtain detailed information on its internal structure. The collected multimodal data was preprocessed, including grayscale, denoising and contrast enhancement of visible light images, temperature calibration and thermal map generation of infrared images, and signal processing and two-dimensional image generation of ultrasonic data. All preprocessed images were then unified to the same size and format for subsequent fusion processing.

[0121] Next, a convolutional neural network is used to extract feature maps from the preprocessed images of each modality, and the number of channels of the feature maps is adjusted to a uniform dimension through a linear projection layer to generate new feature maps. These feature maps are then spliced ​​together to form a multimodal feature map. In order to further improve the richness and robustness of the feature representation, a multi-head self-attention mechanism is used to fuse the multimodal feature maps, and finally a comprehensive feature map is generated.

[0122] Next, a deep learning-based defect detection model was built, which can perform instance segmentation and target detection on the corrected comprehensive feature map, generate pre-selected boxes and mark possible defect areas. This model can not only accurately locate the location of defects, but also identify the type of defects, providing an important basis for subsequent processing.

[0123] Based on the detection results of the model, the defect information of each workpiece is output, and annotated images are generated in a visual way. These annotated images not only intuitively show the specific situation of the defect, but also provide an assessment of the severity of the defect, helping the factory to make decisions quickly and remove unqualified products from the production line.

[0124] The details are shown in Table 1 below:

[0125] Table 1 Experimental data record table

[0126]

[0127] Through in-depth analysis of the above experimental data, it can be clearly seen that in terms of defect detection accuracy, the samples reached 98.5%, 97.8% and 98.2% under standard, complex and variable lighting conditions, respectively, which is significantly higher than the 85.0%, 80.0% and 83.0% of the prior art. This shows that the present invention can more effectively identify various types of defects and maintain a high detection accuracy even under poor lighting conditions.

[0128] Secondly, regarding the recognition rate of defect types, the recognition rates of the method of the present invention under standard, complex and variable lighting conditions are 97.3%, 96.2% and 96.8% respectively, while the prior art is only 80.0%, 75.0% and 78.0%, which means that the method of the present invention can not only detect defects, but also accurately distinguish different defect types.

[0129] Furthermore, the processing time of the method of the present invention is shorter, with an average processing time of between 1.2 seconds and 1.4 seconds, while the prior art requires 2.0 seconds to 2.2 seconds, making the method of the present invention more suitable for application in high-speed production lines and can greatly improve production efficiency.

[0130] In addition, the false alarm rate of the method of the present invention is very low, only 0.01% to 0.02%, which is much lower than the 0.05% to 0.06% of the prior art. The low false alarm rate means that fewer normal products are mistakenly marked as defective, thereby reducing unnecessary waste and costs.

[0131] Finally, the recognition rate of the method of the present invention in identifying severe defects under standard, complex and variable lighting conditions is close to 100%, while the recognition rate of the prior art does not exceed 88%. The method of the present invention can effectively prevent severe defective products from entering the market, thereby ensuring product quality and user safety.

[0132] In summary, the method proposed in the present invention has demonstrated significant technological progress in the field of industrial defect detection, which not only improves the accuracy and efficiency of detection, but also reduces the false alarm rate and enhances the ability to identify severe defects.

[0133] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An industrial defect detection method, characterized in that: include, Collect multimodal data of visible light images, infrared images and ultrasonic images of the workpiece to be inspected, and perform preprocessing; Using a multimodal data fusion algorithm, the preprocessed multimodal data of visible light images, infrared images, and ultrasonic images are fused to generate a comprehensive feature map; Use computer vision technology to analyze the lighting conditions of the image, dynamically adjust the lighting conditions, and obtain a corrected comprehensive feature map; Build a defect detection model, perform instance segmentation and target detection on the corrected comprehensive feature map, generate pre-selected boxes and mark possible defect areas; According to the inspection results, the defect information of the workpiece is output and visualized to generate annotated images; Introducing an online learning mechanism to update and optimize the defect detection model in real time based on newly acquired multimodal data of visible light images, infrared images, and ultrasonic images; The method of using computer vision technology to analyze the lighting conditions of an image, dynamically adjusting the lighting conditions, and obtaining a corrected comprehensive feature map includes the following steps: The global characteristics of the image are quantified by calculating the average brightness and average contrast of the image to obtain the global illumination information of the image; A local binary pattern algorithm is used to capture the relative brightness changes between pixels and their neighboring pixels in the global illumination information image, identify the locations of highlight areas and dark areas, and generate a binary image; Determine the current light intensity and direction based on the average brightness, average contrast, and the positions of the identified highlight and dark areas; The image is processed by using illumination correction technology to generate a corrected image; Performing histogram equalization on the corrected image to generate a corrected comprehensive feature map; Building a defect detection model, performing instance segmentation and target detection on the corrected comprehensive feature map, generating pre-selected boxes and marking possible defect areas include the following steps: Input the corrected comprehensive feature map into the backbone network; Using the top-down path and lateral connections of the feature pyramid, high-level low-resolution feature maps are combined with low-level high-resolution feature maps to generate multi-scale feature maps; Slide a window on each feature map to generate a fixed number of anchor boxes; Classify and regress each anchor box to determine whether it contains the target and adjust its position and size; Select anchor boxes with higher scores as candidate regions; Use the region of interest pooling layer to resize candidate regions of different sizes to the same size; Classify each candidate region through the fully connected layer to determine which category it belongs to; The position of the candidate area is further adjusted through bounding box regression to generate the final pre-selected box; Generate a segmentation mask for each candidate region through the segmentation branch; Generate a pre-selected box for each candidate region classified as foreground; Keep the pre-selected boxes with higher confidence and remove the background and pre-selected boxes with low confidence; For each retained pre-selected box, obtain its corresponding confidence score from the classification branch and sort them from high to low to obtain a sorted pre-selected box list; Initialize an empty list and set the confidence score threshold; When the confidence score is greater than the threshold, the pre-selected box is added to the empty list; Calculate the intersection and union ratio of the pre-selected box and other unprocessed pre-selected boxes. The expression is: Among them, B i is the i-th pre-selected box, B j is the jth pre-selected box, IoU(B i ,B j ) is the intersection and union ratio of the i-th pre-selected box and the j-th pre-selected box, Area(B i ∩B j ) are two pre-selected boxes B i and B j The intersection area, Area(B i ∪B j ) are two pre-selected boxes B i and B j The area of ​​the union of Remove all pre-selected boxes whose intersection-over-union ratio with the pre-selected box is greater than the set threshold; When all pre-selected boxes have been processed, the final list of retained pre-selected boxes is returned; Mark the pre-selected boxes retained in the pre-selected box list as possible defect areas; According to the inspection results, the defect information of the workpiece is output and visualized. The generation of the annotated image includes the following steps: Read the original image and obtain the pre-selected box, classification label and confidence score of each defect area from the processed pre-selected box list; Draw the bounding box and classification label of each defect area on the original image, and save the drawn image as an annotated image; The severity of the defect is calculated based on the confidence score, and the expression is: Among them, C i is the severity of the defect, α is the weight coefficient of the confidence score, S i is the confidence score of the i-th defect, Area(B i ) is the area of ​​the i-th pre-selected box, I o is the original image, Area(I o ) is the area of ​​the original image, β is the weight coefficient of the defect area ratio; Classify workpieces into acceptable and unacceptable based on the severity of defects; When the workpiece is classified as qualified, the workpiece is sent to the qualified material box; When the workpiece is classified as unqualified, the workpiece is sent to the unqualified material box; The information of the defect location, type, confidence score and severity is saved to the database, and the annotated image is saved to the specified path.

2. The industrial defect detection method according to claim 1, characterized in that: Collecting multimodal data of visible light image, infrared image and ultrasonic image of the workpiece to be inspected and preprocessing the data includes the following steps: Using a high-resolution visible light camera to capture a visible light image of the workpiece; Use an infrared camera to take infrared images of the workpiece and capture temperature distribution information; Use ultrasonic sensors to scan workpieces and obtain ultrasonic images of internal structural information; Grayscale, denoise and contrast enhance visible light images, perform temperature calibration and generate thermal maps for infrared images, perform signal processing on ultrasonic images and generate two-dimensional images; Unify all preprocessed images to the same size and format.

3. The industrial defect detection method according to claim 2, characterized in that: Using a multimodal data fusion algorithm, the preprocessed multimodal data of visible light images, infrared images and ultrasonic images are fused to generate a comprehensive feature map, including the following steps: Convert visible light images, infrared images, and ultrasound images into tensor form; Use convolutional neural networks to extract feature maps for each modality; Use a linear projection layer to adjust the number of channels of each feature map to a uniform dimension and generate a new feature map; Each new feature map after mapping is concatenated to form a multimodal feature map; A multi-head self-attention mechanism is used to fuse multimodal feature maps to generate a comprehensive feature map.

4. The industrial defect detection method according to claim 3, characterized in that: Introducing an online learning mechanism to update and optimize the defect detection model in real time based on newly acquired multimodal data of visible light images, infrared images, and ultrasonic images includes the following steps: Continuously collect new multimodal data of visible light images, infrared images, and ultrasound images, use the online stochastic gradient descent algorithm to gradually update the model weights and regularly evaluate the model performance; Adjust the learning rate and batch size based on the evaluation results; Use transfer learning technology to migrate existing large-scale labeled data to new detection tasks to adapt to new defect types and changes.

5. An industrial defect detection system, based on the industrial defect detection method according to any one of claims 1 to 4, characterized in that: include, The data acquisition and preprocessing module acquires multimodal data of visible light images, infrared images and ultrasonic images of the workpiece to be inspected, and performs preprocessing; The multimodal data fusion module uses a multimodal data fusion algorithm to fuse the preprocessed multimodal data of visible light images, infrared images, and ultrasonic images to generate a comprehensive feature map; The lighting condition analysis and correction module uses computer vision technology to analyze the lighting conditions of the image, dynamically adjusts the lighting conditions, and obtains the corrected comprehensive feature map; The defect detection model building module builds the defect detection model, performs instance segmentation and target detection on the corrected comprehensive feature map, generates pre-selected boxes and marks possible defect areas; The inspection result visualization module outputs the defect information of the workpiece and visualizes it according to the inspection results to generate annotated images; The online learning mechanism module introduces an online learning mechanism to update and optimize the defect detection model in real time based on the newly collected multimodal data of visible light images, infrared images and ultrasonic images.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the industrial defect detection method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the industrial defect detection method according to any one of claims 1 to 4 are implemented.

Images