Color-coated sheet defect detection method, device and system based on multi-modal information

Abstract

The invention provides a color-coated sheet defect detection method, device and system based on multi-modal information, and relates to the technical field of sheet surface detection. The method comprises the following steps: acquiring multi-modal information of the color-coated sheet; wherein the multi-modal information comprises laser scanning data, infrared thermal imaging data and hyperspectral imaging data; performing alignment, data fusion and preprocessing on the multi-modal information in sequence; inputting the preprocessed data into a trained improved convolutional neural network model to obtain a defect detection result; wherein the improved convolutional neural network model is obtained based on color-coated sheet surface defect sample training; the defect detection result comprises a defect type and a quantitative parameter of the defect. According to the color-coated sheet defect detection device, the color-coated sheet defect detection efficiency is improved, and the detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of plate surface detection, and in particular to a method, device and system for detecting defects of color-coated plates based on multimodal information. Background Art

[0002] At present, the surface defect detection methods of color-coated plates mainly include visual inspection, optical inspection (such as optical microscope, machine vision inspection system), physical inspection (such as touch inspection, coating thickness inspection) and chemical inspection (such as solvent wiping inspection, observation after salt spray test), etc. Visual inspection relies on manual experience and has low efficiency and limited accuracy; optical microscope inspection requires sample preparation and has a small inspection range; machine vision inspection is insufficient in identifying complex defects; touch inspection in physical inspection is highly subjective, and coating thickness inspection cannot fully detect other defects; chemical inspection is time-consuming and some are destructive inspections. Therefore, a highly efficient surface defect detection method for color-coated plates is needed. Summary of the invention

[0003] The embodiments of the present invention provide a method, device and system for color-coated plate defect detection based on multimodal information to solve the problem of efficient detection of surface defects of color-coated plates, taking into account both detection efficiency and detection accuracy.

[0004] In a first aspect, an embodiment of the present invention provides a method for detecting defects of a color-coated plate based on multimodal information, comprising: Acquire multimodal information of the color-coated plate; wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data; Aligning, fusing and preprocessing the multimodal information in sequence; The preprocessed data is input into a trained improved convolutional neural network model to obtain defect detection results; wherein the improved convolutional neural network model is trained based on surface defect samples of color-coated plates; the defect detection results include defect types and quantitative parameters of defects.

[0005] In a possible implementation, the improved convolutional neural network model includes: a convolution layer, a pooling layer, a feature fusion module, a fully connected layer, a classifier and a deconvolution layer; The convolution layer is used to capture feature information of different scales; the convolution layer includes multiple convolution kernels, each of which has a different size and step size; The pooling layer includes a downsampling module for downsampling the feature map; The feature fusion module is used to fuse the features corresponding to the data collected from different modalities; The classifier is used to output the probability value of whether there is a defect on the surface of the color-coated plate and the defect type; The deconvolution layer is used to map the position information of the defect back to the coordinate system of the original color-coated plate image, and output the defect detection result according to the specific position of the defect on the surface of the color-coated plate.

[0006] In a possible implementation manner, the quantitative parameter of the defect includes one or more of the length, width, depth and area of ​​the defect.

[0007] In a possible implementation manner, aligning the multimodal information includes: According to the timestamp information, the laser scanning data, infrared thermal imaging data and hyperspectral imaging data of the same position of the color-coated plate collected at the same time are matched; According to the size information of the color-coated plate and the installation position of the equipment, the spatial coordinates of the laser scanning data, infrared thermal imaging data and hyperspectral imaging data are calibrated.

[0008] In a possible implementation manner, performing data fusion on the multimodal information includes: Extracting surface geometric features from the laser scanning data, extracting temperature gradient features and thermal anomaly region features from the infrared thermal imaging data, and extracting reflectance values ​​of characteristic spectral bands and slopes of spectral curves from the hyperspectral imaging data; The feature extraction results of the multimodal information are fused to obtain a fused feature vector.

[0009] In a possible implementation, preprocessing the multimodal information includes: denoising and normalization; The denoising process includes: selecting a corresponding filtering algorithm according to the data type of the multimodal information to remove outliers and random noise; Normalization processing includes: for numerical features, the maximum-minimum normalization method is used to map the feature values ​​to the [0,1] interval; for ratio features and dimensionless features, the standardization method is used to make the mean 0 and the variance 1.

[0010] In a possible implementation, the method further includes: performing accuracy calibration, temperature calibration, and spectrum calibration on the laser scanning device, the infrared thermal imager, and the hyperspectral imager, respectively, according to the multimodal information.

[0011] In a second aspect, an embodiment of the present invention provides a color-coated plate defect detection device based on multimodal information, comprising: An acquisition module is used to acquire multimodal information of the color-coated plate; wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data; A data processing module, used for sequentially aligning, data fusion and preprocessing the multimodal information; The recognition module is used to input the preprocessed data into a trained improved convolutional neural network model to obtain defect detection results; wherein the improved convolutional neural network model is trained based on surface defect samples of color-coated plates; and the detection results include defect types and quantitative parameters of defects.

[0012] In the third aspect, an embodiment of the present invention provides a color-coated plate defect detection system based on multimodal information, characterized in that it includes: a laser scanning device, an infrared thermal imager, a hyperspectral imager and a color-coated plate defect detection device based on multimodal information as described in the second aspect above; the device executes the method in the first aspect above or any possible implementation of the first aspect.

[0013] In a fourth aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method in the first aspect or any possible implementation of the first aspect is implemented.

[0014] In the embodiment of the present invention, multimodal information of color-coated plates is obtained, wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data. The advantages of multiple data complement each other, make up for the limitations of a single detection method, and greatly improve the accuracy of defect type identification. The multimodal information is aligned, fused and preprocessed in sequence. The alignment process can avoid the analysis deviation caused by data dislocation. The data fusion integrates the advantages of different modal data to obtain a comprehensive feature vector to enrich the data connotation. The preprocessing can improve the data quality to improve the detection accuracy. The preprocessed data is input into the trained improved convolutional neural network model to obtain the defect detection result; wherein the improved convolutional neural network model is obtained based on the surface defect sample of the color-coated plate training; the defect detection result includes the defect type and the quantitative parameter of the defect. The improved convolutional neural network model, through the training of a large number of color-coated plate defect samples, has a strong learning ability, can quickly extract and analyze features, can not only quickly identify the defect type, but also accurately calculate the quantitative parameter of the defect. The embodiment of the present application greatly reduces the detection time through model automation processing, which not only improves the efficiency of color-coated plate defect detection, but also improves the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an application scenario diagram of a color-coated plate defect detection method based on multimodal information provided by an embodiment of the present invention; Figure 2 It is a flow chart of the implementation of a color-coated plate defect detection method based on multimodal information provided by an embodiment of the present invention; Figure 3 is a flowchart of a method for detecting color-coated plate defects based on multimodal information provided by another embodiment of the present invention; Figure 4 It is a schematic structural diagram of a color-coated plate defect detection device based on multimodal information provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0017] Figure 1 This is an application scenario diagram of the color-coated plate defect detection method based on multimodal information provided by an embodiment of the present invention. Figure 1 As shown, the color-coated plate defect detection system based on multimodal information includes a laser scanning device, an infrared thermal imager, a hyperspectral imager and a data analysis device.

[0018] In the actual implementation process, different inspection equipment is integrated on the production line, taking into account their spacing, angle and time synchronization to form a tightly matched inspection unit. The high-precision synchronous controller ensures multi-modal data collection at the same position of the color-coated plate at the same time to ensure data consistency and reliability.

[0019] Through multimodal data collection, we can ensure that the surface status of the color-coated plate is captured in all directions and at multiple angles, thus avoiding the limitation of traditional single detection methods that can only detect a single type of defect.

[0020] Figure 2 The following is a flow chart of a method for detecting color-coated plate defects based on multimodal information provided by an embodiment of the present invention. Figure 2 As shown, the following steps are included: S201, obtaining multimodal information of the color-coated plate; wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data.

[0021] The execution subject of each embodiment of the present application can be a server, processor, microprocessor or other device with data processing function. In the actual implementation process, the specific implementation method of the execution subject can be selected according to actual needs. This embodiment does not impose any special restrictions on this. As long as it is a device with data processing function, it can be used. To facilitate the understanding of the solution, a data analysis device is used as an example for explanation.

[0022] like Figure 1 In the scene diagram shown, laser scanning data, infrared thermal imaging data and hyperspectral imaging data are collected by laser scanning equipment, infrared thermal imager and hyperspectral imager respectively. After each device collects the detection information of the color-coated plate, it sends the collected information to the data analysis device through wired or wireless communication. The data analysis device integrates the multi-modal information of the color-coated plate for centralized processing and defect detection to improve data processing efficiency, thereby improving detection accuracy and efficiency.

[0023] In the specific implementation process, when the color-coated plate passes through the detection area at a constant speed on the production line, the laser scanning equipment quickly scans the surface of the color-coated plate to obtain the three-dimensional profile data of the surface; the infrared thermal imager simultaneously collects the temperature distribution data of the surface of the color-coated plate to detect the internal defects under the coating; the hyperspectral imager collects the spectral data of the surface of the color-coated plate to analyze the chemical properties of the coating. The collected data is transmitted to the background data analysis device in real time through high-speed data transmission lines or remote wireless communication.

[0024] The specific process of collecting laser scanning data, infrared thermal imaging data and hyperspectral imaging data is as follows: The laser scanning equipment scans the surface of the color-coated plate line by line at the set scanning speed and resolution. During the scanning process, the laser emission module continuously emits laser beams, the detector receives the reflected light in real time, and converts the optical signal into an electrical signal. Through the built-in signal processing unit, the electrical signal is converted into the corresponding three-dimensional coordinate data to form the three-dimensional contour point cloud data of the color-coated plate surface. These data are transmitted to the background data analysis device in real time in the form of data packets through high-speed data transmission lines.

[0025] Before the color-coated board enters the detection area, the thermal excitation source (such as infrared lamp) of the infrared thermal imager heats the surface of the color-coated board evenly for a short time. Then, the infrared thermal imager continuously images the surface of the color-coated board at a set frame rate. The temperature value of each pixel is measured by an infrared detector and converted into a digital signal. These temperature data form a thermal image sequence, which is also transmitted to the data analysis device in real time through a high-speed data transmission line.

[0026] The hyperspectral imager uses push-broom or gaze imaging to scan the surface of the color-coated plate. During the scanning process, the spectroscopic system decomposes the reflected light into spectral components of different wavelengths, and the detector array detects the light of each wavelength at the same time. By collecting and processing the detector output signal, the spectral reflectance data of each pixel at different wavelengths is obtained to form a hyperspectral image data cube. After the data cube is packaged in a certain format, it is transmitted to the background data analysis device in real time through a high-speed data transmission line.

[0027] S202, aligning, data fusion and preprocessing the multimodal information in sequence.

[0028] In the data analysis device, the three-dimensional profile data of laser scanning, the temperature data of infrared thermal imaging, and the spectral data of hyperspectral imaging are aligned and integrated according to specific coordinates and time information. Then the data is fused to obtain the fused feature vector. Finally, the fused data is preprocessed, including denoising, normalization and other operations, to improve the data quality and prepare for the subsequent deep learning algorithm analysis.

[0029] During the specific implementation process, in order to improve the efficiency of multimodal information alignment, laser scanning equipment, infrared thermal imagers and hyperspectral imagers are synchronized through high-precision synchronization controllers to ensure that the error does not exceed 1 millisecond.

[0030] S203, inputting the preprocessed data into a trained improved convolutional neural network model to obtain defect detection results; wherein the improved convolutional neural network model is trained based on surface defect samples of color-coated plates; the defect detection results include defect types and quantitative parameters of defects.

[0031] During the model training process, a large number of samples of surface defects of color-coated plates of different types and degrees are collected, covering common defects such as scratches, pinholes, coating aging, and internal delamination. At the same time, each sample is accurately labeled to clarify information such as defect type, location, and size. For example, professional image annotation tools are used to draw the outline of the defect on the image and record its corresponding category and size parameters.

[0032] The collected samples are divided into training set, validation set and test set, optionally in the ratio of 70%, 15%, and 15%. The training set is used to learn the model parameters, the validation set is used to adjust the model hyperparameters and prevent overfitting, and the test set is used to evaluate the final performance of the model.

[0033] Input the training set data into the built improved convolutional neural network model, and set the appropriate loss function (such as the cross entropy loss function) and optimizer (such as the Adam optimizer). During the training process, the model continuously adjusts parameters through the back propagation algorithm to minimize the loss function. At the same time, regularly evaluate the model performance on the validation set, and adjust hyperparameters such as the learning rate and regularization parameters based on the validation results to prevent the model from overfitting.

[0034] Among them, by learning from a large number of samples of surface defects of color-coated plates, it is possible to automatically extract features from the data and judge whether there are defects on the surface of the color-coated plates based on these features. If defects are detected, the model will further identify the type of defects (such as scratches, pinholes, coating aging, internal delamination, etc.), location and size.

[0035] In this embodiment, multimodal information of color-coated plates is obtained, wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data. The advantages of multiple data complement each other, make up for the limitations of a single detection method, and greatly improve the accuracy of defect type identification. The multimodal information is aligned, fused and preprocessed in turn. The alignment process avoids the analysis deviation caused by data dislocation. The data fusion integrates the advantages of different modal data to obtain a comprehensive feature vector to enrich the data connotation. The preprocessing improves the data quality to improve the detection accuracy. The preprocessed data is input into the trained improved convolutional neural network model to obtain the defect detection result; wherein, the improved convolutional neural network model is obtained based on the surface defect sample of the color-coated plate; the defect detection result includes the defect type and the quantitative parameters of the defect. The improved convolutional neural network model, through the training of a large number of color-coated plate defect samples, has a strong learning ability, can quickly extract and analyze features, can not only quickly identify the defect type, but also accurately calculate the quantitative parameters of the defect. The embodiment of the present application greatly reduces the detection time through model automation processing, which not only improves the efficiency of color-coated plate defect detection, but also improves the accuracy of detection.

[0036] In a possible implementation, the convolutional neural network model is improved, including: a convolutional layer, a pooling layer, a feature fusion module, a fully connected layer, a classifier, and a deconvolution layer; Among them, the convolution layer is used to capture feature information of different scales; the convolution layer includes multiple convolution kernels, and the size and step length of each convolution kernel are different; The pooling layer includes a downsampling module for downsampling the feature map; Feature fusion module, used to fuse features corresponding to data collected from different modalities; A classifier is used to output the probability value of whether there are defects on the surface of the color-coated sheet and the defect type; The deconvolution layer is used to map the position information of the defect back to the coordinate system of the original color-coated plate image and output the defect detection result according to the specific position of the defect on the surface of the color-coated plate.

[0037] In the convolution layer, convolution kernels of different sizes and steps are set to capture features of different scales. For example, 3×3 and 5×5 convolution kernels are used to extract higher-level features by stacking multiple convolution layers. The pooling layer uses maximum pooling or average pooling operations to downsample the feature map, reducing the amount of data while retaining the main features. In the feature fusion module, the features corresponding to different modal data are fused to further extract comprehensive features. Finally, after the fully connected layer and classifier, the probability value of whether there are defects on the surface of the color-coated plate and the type of defects is output.

[0038] In a possible implementation manner, the quantitative parameters of the defect include one or more of the length, width, depth and area of ​​the defect.

[0039] In a possible implementation, in step S102, aligning the multimodal information includes: According to the timestamp information, the laser scanning data, infrared thermal imaging data and hyperspectral imaging data of the same position of the color-coated plate collected at the same time are matched; According to the size information of the color-coated plate and the installation position of the equipment, the spatial coordinates of the laser scanning data, infrared thermal imaging data and hyperspectral imaging data are calibrated.

[0040] Among them, a time synchronization mechanism is established according to the movement speed of the color-coated plate on the production line and the scanning time interval of each device. By adding timestamp information to the data, the data on the same position of the color-coated plate collected at the same time by laser scanning, infrared thermal imaging and hyperspectral imaging are matched. At the same time, the spatial coordinates of the three types of data are calibrated using the size information of the color-coated plate and the installation position of the equipment to ensure the consistency of the data in space.

[0041] In this embodiment, the time unification of multimodal information is achieved according to the timestamp information. In a possible implementation, in step S102, performing data fusion on multimodal information includes: Extract surface geometric features from laser scanning data, extract temperature gradient features and thermal anomaly area features from infrared thermal imaging data, and extract reflectance values ​​of characteristic spectral bands and slopes of spectral curves from hyperspectral imaging data; The feature extraction results of multimodal information are fused to obtain a fused feature vector.

[0042] Among them, surface geometric features including curvature, roughness, etc. are extracted from laser scanning data.

[0043] In a possible implementation, in step S102, preprocessing the multimodal information includes: denoising and normalization; The denoising process includes: selecting a corresponding filtering algorithm according to the data type of the multimodal information to remove outliers and random noise; Normalization processing includes: for numerical features, the maximum-minimum normalization method is used to map the feature values ​​to the [0,1] interval; for ratio features and dimensionless features, the standardization method is used to make the mean 0 and the variance 1.

[0044] Among them, for the noise in laser scanning data, filtering algorithms (such as Gaussian filtering and median filtering) are used to remove outliers and random noise; for infrared thermal imaging data, due to the existence of thermal noise, wavelet denoising algorithm is used to suppress noise in different frequency domains; for hyperspectral imaging data, spectral smoothing algorithm (such as Savitzky-Golay filtering) is used to remove high-frequency noise in the spectral curve and improve the signal-to-noise ratio of the data.

[0045] Using different normalization methods for different types of data can avoid algorithm training bias caused by large differences in numerical ranges between different features.

[0046] In a possible implementation, the method further includes: performing precision calibration, temperature calibration, and spectrum calibration on the laser scanning device, infrared thermal imager, and hyperspectral imager, respectively, based on the multimodal information.

[0047] Among them, the laser scanning equipment is precisely calibrated through standard samples so that it can accurately measure the three-dimensional contour of the surface; the infrared thermal imager is temperature calibrated to ensure the accuracy of the thermal imaging data; and the hyperspectral imager is spectrally calibrated so that it can accurately obtain spectral information.

[0048] Specifically, laser scanning equipment calibration: use a standard sample with high-precision three-dimensional coordinates, whose surface has known geometric features (such as scratches of a specific depth, protrusions of a specific height). Scan the standard sample with the laser scanning equipment, obtain the scanning data and compare it with the actual three-dimensional coordinate data of the standard sample. By adjusting the emission angle, detector position and scanning parameters (such as scanning speed and resolution) of the laser scanning equipment, the error between the scanning data and the actual data is within the allowable range, ensuring that the laser scanning equipment can accurately measure the three-dimensional profile of the surface of the color-coated plate.

[0049] Infrared thermal imager calibration: Use a standard black body radiation source with a precisely known temperature and emissivity. Aim the infrared thermal imager at the black body radiation source and collect thermal images at different temperature settings. According to the black body radiation law, compare the temperature value in the collected thermal image with the actual temperature value of the black body radiation source, and adjust the temperature calibration parameters (such as gain and offset) of the infrared thermal imager to ensure that the infrared thermal imager can accurately measure the temperature distribution on the surface of the color coated plate.

[0050] Hyperspectral imager calibration: Use a standard spectral reflectance sample whose reflectance at different wavelengths is known. Scan the standard spectral reflectance sample with the hyperspectral imager, obtain the spectral data and compare it with the actual spectral reflectance data of the standard sample. By adjusting the wavelength calibration parameters of the hyperspectral imager (such as wavelength offset and bandwidth correction), the collected spectral data is consistent with the actual data, ensuring that the hyperspectral imager can accurately obtain the spectral information of the surface of the color-coated plate.

[0051] In the actual implementation process, after the defect detection of the color-coated plate is completed, the defect detection result output and feedback process is also included.

[0052] Specifically, when a defect is detected, the system immediately issues an alarm and displays the relevant information of the defect (type, location, size, etc.) in a visual manner on the monitoring screen. At the same time, this information is fed back to the control system of the production line so that the production process can be adjusted or the defective color-coated plates can be marked and processed. On the monitoring screen, the image of the color-coated plate is used as the background, and the detected defects are marked with marker boxes or marker points of different colors. At the same time, the information bar next to or below the image displays the type, location coordinates, size and other information of the defects in detail. For multiple defects, they are numbered in a certain order to facilitate the operator to view and record.

[0053] The defect information is sent to the control system of the production line through the network communication protocol. The control system takes different measures according to the type and severity of the defect. For minor defects, real-time corrections can be made by adjusting the process parameters on the production line (such as coating thickness, drying temperature, etc.); for serious defects, the control system can control the production line to pause and mark the defective color-coated plates for subsequent processing (such as rework, scrapping, etc.).

[0054] See also Figure 3 , which shows a flowchart of a method for detecting color-coated plate defects based on multimodal information provided by another embodiment of the present invention, which is described in detail as follows: Obtain multimodal information of color-coated plates. Multimodal information includes laser scanning data, infrared thermal imaging data, and hyperspectral imaging data, so as to achieve complementary advantages of multiple data, make up for the limitations of a single detection method, and improve the accuracy of defect type identification.

[0055] The multimodal information is aligned, fused, denoised and normalized in turn to obtain processed data, improve data quality and thus enhance detection accuracy.

[0056] The processed data is input into the trained improved convolutional neural network model to obtain defect detection results, realize automated detection, improve the efficiency of color-coated plate defect detection, and improve the accuracy of detection.

[0057] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0058] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0059] Figure 4 The schematic diagram of the structure of the color-coated plate defect detection device based on multimodal information provided by an embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows: like Figure 4 As shown, the color-coated plate defect detection device 4 based on multimodal information includes: The acquisition module 401 is used to acquire multimodal information of the color-coated plate; wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data; The data processing module 402 is used to sequentially align, fuse and preprocess the multimodal information; The identification module 403 is used to input the preprocessed data into the trained improved convolutional neural network model to obtain defect detection results; wherein the improved convolutional neural network model is trained based on surface defect samples of color-coated plates; the detection results include defect types and quantitative parameters of defects.

[0060] In a possible implementation, the data processing module 402 is specifically used to match the laser scanning data, infrared thermal imaging data and hyperspectral imaging data of the same position of the color-coated plate collected at the same time according to the timestamp information; According to the size information of the color-coated plate and the installation position of the equipment, the spatial coordinates of the laser scanning data, infrared thermal imaging data and hyperspectral imaging data are calibrated.

[0061] In a possible implementation, the data processing module 402 is specifically used to extract surface geometric features from the laser scanning data, extract temperature gradient features and thermal anomaly region features from the infrared thermal imaging data, and extract reflectance values ​​of characteristic spectral bands and slopes of spectral curves from the hyperspectral imaging data; The feature extraction results of the multimodal information are fused to obtain a fused feature vector.

[0062] In a possible implementation, the data processing module 402 is specifically configured to select a corresponding filtering algorithm according to the data type of the multimodal information to remove outliers and random noise; For numerical features, the maximum-minimum normalization method is used to map the feature values ​​to the [0,1] interval; for ratio features and dimensionless features, the standardization method is used to make the mean 0 and the variance 1.

[0063] In this embodiment, multimodal information of color-coated plates is obtained, wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data. The advantages of multiple data complement each other, make up for the limitations of a single detection method, and greatly improve the accuracy of defect type identification. The multimodal information is aligned, fused and preprocessed in turn. The alignment process avoids the analysis deviation caused by data dislocation. The data fusion integrates the advantages of different modal data to obtain a comprehensive feature vector to enrich the data connotation. The preprocessing improves the data quality to improve the detection accuracy. The preprocessed data is input into the trained improved convolutional neural network model to obtain the defect detection result; wherein, the improved convolutional neural network model is obtained based on the surface defect sample of the color-coated plate; the defect detection result includes the defect type and the quantitative parameters of the defect. The improved convolutional neural network model, through the training of a large number of color-coated plate defect samples, has a strong learning ability, can quickly extract and analyze features, can not only quickly identify the defect type, but also accurately calculate the quantitative parameters of the defect. The embodiment of the present application greatly reduces the detection time through model automation processing, which not only improves the efficiency of color-coated plate defect detection, but also improves the accuracy of detection.

[0064] The embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the method in the above method embodiment when executing the computer program. For example, the electronic device may be a computer, a notebook, etc., which is not limited here.

[0065] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. If there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form a new embodiment according to their internal logical relationship.

[0066] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A color-coated plate defect detection method based on multimodal information, characterized in that: include: Acquire multimodal information of the color-coated plate; wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data; Aligning, fusing and preprocessing the multimodal information in sequence; The preprocessed data is input into a trained improved convolutional neural network model to obtain defect detection results; wherein the improved convolutional neural network model is trained based on surface defect samples of color-coated plates; the defect detection results include defect types and quantitative parameters of defects.

2. The color-coated plate defect detection method based on multimodal information according to claim 1 is characterized in that: The improved convolutional neural network model includes: a convolution layer, a pooling layer, a feature fusion module, a fully connected layer, a classifier and a deconvolution layer; The convolution layer is used to capture feature information of different scales; the convolution layer includes multiple convolution kernels, each of which has a different size and step size; The pooling layer includes a downsampling module for downsampling the feature map; The feature fusion module is used to fuse the features corresponding to the data collected from different modalities; The classifier is used to output the probability value of whether there is a defect on the surface of the color-coated plate and the defect type; The deconvolution layer is used to map the position information of the defect back to the coordinate system of the original color-coated plate image, and output the defect detection result according to the specific position of the defect on the surface of the color-coated plate.

3. The color-coated plate defect detection method based on multimodal information according to claim 1 or 2, characterized in that: The quantitative parameters of the defect include one or more of the length, width, depth and area of ​​the defect.

4. The color-coated plate defect detection method based on multimodal information according to claim 1 is characterized in that: Aligning the multimodal information includes: According to the timestamp information, the laser scanning data, infrared thermal imaging data and hyperspectral imaging data of the same position of the color-coated plate collected at the same time are matched; According to the size information of the color-coated plate and the installation position of the equipment, the spatial coordinates of the laser scanning data, infrared thermal imaging data and hyperspectral imaging data are calibrated.

5. The color-coated plate defect detection method based on multimodal information according to claim 1 is characterized in that: Performing data fusion on the multimodal information includes: Extracting surface geometric features from the laser scanning data, extracting temperature gradient features and thermal anomaly region features from the infrared thermal imaging data, and extracting reflectance values ​​of characteristic spectral bands and slopes of spectral curves from the hyperspectral imaging data; The feature extraction results of the multimodal information are fused to obtain a fused feature vector.

6. The color-coated plate defect detection method based on multimodal information according to claim 1 is characterized in that: Preprocessing the multimodal information includes: denoising and normalization; The denoising process includes: selecting a corresponding filtering algorithm according to the data type of the multimodal information to remove outliers and random noise; Normalization processing includes: for numerical features, the maximum-minimum normalization method is used to map the feature values ​​to the [0,1] interval; for ratio features and dimensionless features, the standardization method is used to make the mean 0 and the variance 1.

7. The color-coated plate defect detection method based on multimodal information according to claim 1 is characterized in that: Also includes: According to the multimodal information, the laser scanning device, the infrared thermal imager and the hyperspectral imager are calibrated for accuracy, temperature and spectrum respectively.

8. A color-coated plate defect detection device based on multimodal information, characterized in that: include: An acquisition module is used to acquire multimodal information of the color-coated plate; wherein the multimodal information includes: laser scanning data, infrared thermal imaging data and hyperspectral imaging data; A data processing module, used for sequentially aligning, data fusion and preprocessing the multimodal information; The recognition module is used to input the preprocessed data into a trained improved convolutional neural network model to obtain defect detection results; wherein the improved convolutional neural network model is trained based on surface defect samples of color-coated plates; and the detection results include defect types and quantitative parameters of defects.

9. A color-coated plate defect detection system based on multimodal information, characterized in that: include: Laser scanning equipment, infrared thermal imager, hyperspectral imager and the color-coated plate defect detection device based on multimodal information as described in claim 8; The device performs the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

Images