Coating defect detection method, device, terminal equipment and storage medium based on neural network

Abstract

The present invention discloses a coating defect detection method, device, terminal equipment and storage medium based on neural network. The method comprises the following steps: transmitting a first ultrasonic signal array to a coating area, and determining an ultrasonic image of the coating area based on a returned second ultrasonic signal array; inputting the ultrasonic image into a first coating defect classification model, comparing the extracted first coating defect feature with a standard coating defect feature library, and outputting a first coating defect classification result; fusing the coating images at all angles to obtain and inputting the fused coating image into a second coating defect classification model, comparing the extracted second coating defect feature with a standard coating defect feature library, and outputting a second coating defect classification result; integrating the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect of the coating area; compared with the prior art, the technical solution of the present invention can improve the accuracy and reliability of defect detection.

Description

Technical Field

[0001] The present application relates to the technical field of image recognition processing, and in particular to a coating defect detection method, apparatus, terminal device and storage medium based on a neural network. Background Art

[0002] In the manufacturing process of composite materials, coating is a key technology, which involves coating paste polymer, molten polymer or polymer solution on a substrate such as paper, cloth or plastic film to form a composite material with specific properties; because in the traditional coating process, the control of coating quality depends largely on the personal experience and skill level of the operator, this dependence leads to the instability of product quality, because the difference in skills of different operators may lead to fluctuations in coating quality, therefore, defect detection of coating is required.

[0003] In the existing coating defect detection technology, defects on the coating surface are generally detected based on artificial vision. However, this method is inefficient and easily interfered by human factors, resulting in reduced detection accuracy. In addition, defects inside the composite coating, such as delamination, debonding, porosity, etc., cannot be discovered through artificial vision, which poses a potential threat to product quality. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a coating defect detection method, device, terminal equipment and storage medium based on a neural network, which can improve the accuracy and reliability of defect detection.

[0005] In order to solve the above technical problems, the present application provides a coating defect detection method based on a neural network, comprising:

[0006] Transmitting a first ultrasonic signal array to a coating area, and receiving a second ultrasonic signal array returned by the coating area, and generating an ultrasonic image of the coating area based on the second ultrasonic signal array;

[0007] Inputting the ultrasonic image into a pre-trained first coating defect classification model, so that the first coating defect classification model extracts defect features from the ultrasonic image, and compares the extracted first coating defect features with a standard coating defect feature library, and outputs a first coating defect classification result of the coating area;

[0008] Collecting coating images of the coating area at multiple angles, and fusing the coating images at all angles to obtain a fused coating image;

[0009] Inputting the fused coating image into a pre-trained second coating defect classification model, so that the second coating defect classification model extracts defect features from the fused coating image, and compares the extracted second coating defect features with a standard coating defect feature library, and outputs a second coating defect classification result of the coating area;

[0010] The first coating defect classification result and the second coating defect classification result are integrated to determine a predicted coating defect corresponding to the coating area.

[0011] In a possible implementation, generating an ultrasonic image of the coating area based on the second ultrasonic signal array specifically includes:

[0012] Acquire the signal amplitude corresponding to each second ultrasonic signal in the second ultrasonic signal array, and convert the signal amplitude into a brightness value;

[0013] The signal position corresponding to each second ultrasonic signal in the second ultrasonic signal array is determined, and the brightness values ​​are arranged based on the signal position to generate an ultrasonic image of the coating area.

[0014] In a possible implementation, the ultrasonic image is input into a pre-trained first coating defect classification model so that the first coating defect classification model extracts defect features from the ultrasonic image, specifically including:

[0015] Inputting the ultrasonic image into a pre-trained first coating defect classification model so that the first coating defect classification model obtains the brightness value corresponding to each pixel point in the ultrasonic image, and performs mean normalization processing on the brightness value to obtain a normalized brightness value;

[0016] Selecting the standard normalized brightness value corresponding to each pixel point, calculating the brightness difference coefficient based on the normalized brightness value and the standard normalized brightness value, and calculating the defect probability corresponding to each pixel point in the ultrasonic image based on the brightness difference coefficient, and generating a defect probability map corresponding to the ultrasonic image;

[0017] Performing defect edge extraction on the defect probability map to determine a first defect shape, calculating a first defect shape area corresponding to the first defect shape, and determining a first defect shape position of the first defect shape on the ultrasonic image;

[0018] The first defect shape, the first defect shape area and the first defect shape position are respectively vectorized to obtain a first defect shape vector, a first defect shape area vector and a first defect shape position vector, and the first defect shape vector, the first defect shape area vector and the first defect shape position vector are associated to obtain a first defect association vector, and the first defect association vector is used as the first coating defect feature of the ultrasonic image.

[0019] In a possible implementation, the coating images at all angles are fused to obtain a fused coating image, which specifically includes:

[0020] Converting the coating images at all angles into coating mask matrices, and acquiring coating depth images corresponding to the coating images at all angles, and obtaining image point clouds of the coating images at all angles based on the coating mask matrix and the coating depth image;

[0021] Determine the image centroids corresponding to the image point clouds of the coated image at all angles, and convert the image point clouds of the coated image at all angles into intermediate image point clouds according to the image centroids;

[0022] Selecting a reference intermediate image point cloud from all intermediate image point clouds, and registering the remaining intermediate image point clouds based on the reference intermediate image point cloud to obtain conversion point clouds corresponding to the remaining intermediate image point clouds;

[0023] Based on the preset coating image reference point cloud, the reference intermediate point cloud and all the conversion point clouds are fused to obtain a determined target image point cloud, and based on the target image point cloud, a fused coating image is obtained.

[0024] In a possible implementation, the fused coating image is input into a pre-trained second coating defect classification model so that the second coating defect classification model extracts defect features from the fused coating image, specifically including:

[0025] Inputting the fused coating image into a pre-trained second coating defect classification model, so that the second coating defect classification model performs grayscale processing on the fused coating image to obtain a fused coating grayscale image, extracting defect edges on the fused coating grayscale image to obtain a second defect shape, calculating a second defect shape area corresponding to the second defect shape, and determining a second defect shape position of the second defect shape on the fused coating image;

[0026] The second defect shape, the second defect shape area and the second defect shape position are respectively vectorized to obtain a second defect shape vector, a second defect shape area vector and a second defect shape position vector, and the second defect shape vector, the second defect shape area vector and the second defect shape position vector are associated to obtain a second defect association vector, and the second defect association vector is used as the second coating defect feature of the fused coating image.

[0027] The present invention provides a coating defect detection method based on a neural network, further comprising:

[0028] Performing a tapping process on the coated area, collecting voiceprint data corresponding to the coated area, and performing Fourier transform processing on the voiceprint data to obtain a voiceprint spectrum image;

[0029] Comparing the voiceprint spectrum image with a preset standard coating defect voiceprint feature library to determine a third coating defect classification result corresponding to the coating area;

[0030] The first coating defect classification result is corrected based on the third coating defect classification result to obtain a corrected first coating defect classification result, and the corrected first coating defect classification result is updated as the first coating defect classification result.

[0031] In a possible implementation, integrating the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect corresponding to the coating area specifically includes:

[0032] Using the first coating defect classification result as the predicted coating internal defect corresponding to the coating area; wherein the first coating defect classification result includes a first coating defect type, and a first coating defect shape area and the first coating defect shape position corresponding to the first coating defect type;

[0033] Using the second coating defect classification result as the predicted coating surface defect corresponding to the coating area; wherein the second coating defect classification result includes a second coating defect type, and a second coating defect shape area and a second coating defect shape position corresponding to the second coating defect type;

[0034] The predicted coating internal defects and the predicted coating surface defects are integrated to obtain the predicted coating defects corresponding to the coating area.

[0035] The present invention also provides a coating defect detection device based on a neural network, comprising: an ultrasonic image generation module, a first coating defect classification module, a coating image fusion module, a second coating defect classification module and a predicted coating defect determination module;

[0036] The ultrasonic image generation module is used to transmit a first ultrasonic signal array to the coating area, receive a second ultrasonic signal array returned by the coating area, and generate an ultrasonic image of the coating area based on the second ultrasonic signal array;

[0037] The first coating defect classification module is used to input the ultrasonic image into a pre-trained first coating defect classification model, so that the first coating defect classification model extracts defect features from the ultrasonic image, compares the extracted first coating defect features with a standard coating defect feature library, and outputs a first coating defect classification result of the coating area;

[0038] The coating image fusion module is used to collect coating images of the coating area at multiple angles, and fuse the coating images at all angles to obtain a fused coating image;

[0039] The second coating defect classification module is used to input the fused coating image into a pre-trained second coating defect classification model, so that the second coating defect classification model extracts defect features from the fused coating image, and compares the extracted second coating defect features with a standard coating defect feature library, and outputs a second coating defect classification result of the coating area;

[0040] The predicted coating defect determination module is used to integrate the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect corresponding to the coating area.

[0041] In a possible implementation, the ultrasonic image generation module is used to generate an ultrasonic image of the coating area based on the second ultrasonic signal array, specifically including:

[0042] Acquire the signal amplitude corresponding to each second ultrasonic signal in the second ultrasonic signal array, and convert the signal amplitude into a brightness value;

[0043] The signal position corresponding to each second ultrasonic signal in the second ultrasonic signal array is determined, and the brightness values ​​are arranged based on the signal position to generate an ultrasonic image of the coating area.

[0044] In a possible implementation, the first coating defect classification module is used to input the ultrasonic image into a pre-trained first coating defect classification model so that the first coating defect classification model extracts defect features from the ultrasonic image, specifically including:

[0045] Inputting the ultrasonic image into a pre-trained first coating defect classification model so that the first coating defect classification model obtains the brightness value corresponding to each pixel point in the ultrasonic image, and performs mean normalization processing on the brightness value to obtain a normalized brightness value;

[0046] Selecting the standard normalized brightness value corresponding to each pixel point, calculating the brightness difference coefficient based on the normalized brightness value and the standard normalized brightness value, and calculating the defect probability corresponding to each pixel point in the ultrasonic image based on the brightness difference coefficient, and generating a defect probability map corresponding to the ultrasonic image;

[0047] Performing defect edge extraction on the defect probability map to determine a first defect shape, calculating a first defect shape area corresponding to the first defect shape, and determining a first defect shape position of the first defect shape on the ultrasonic image;

[0048] The first defect shape, the first defect shape area and the first defect shape position are respectively vectorized to obtain a first defect shape vector, a first defect shape area vector and a first defect shape position vector, and the first defect shape vector, the first defect shape area vector and the first defect shape position vector are associated to obtain a first defect association vector, and the first defect association vector is used as the first coating defect feature of the ultrasonic image.

[0049] In a possible implementation, the coating image fusion module is used to fuse the coating images at all angles to obtain a fused coating image, specifically including:

[0050] Converting the coating images at all angles into coating mask matrices, and acquiring coating depth images corresponding to the coating images at all angles, and obtaining image point clouds of the coating images at all angles based on the coating mask matrix and the coating depth image;

[0051] Determine the image centroids corresponding to the image point clouds of the coated image at all angles, and convert the image point clouds of the coated image at all angles into intermediate image point clouds according to the image centroids;

[0052] Selecting a reference intermediate image point cloud from all intermediate image point clouds, and registering the remaining intermediate image point clouds based on the reference intermediate image point cloud to obtain conversion point clouds corresponding to the remaining intermediate image point clouds;

[0053] Based on the preset coating image reference point cloud, the reference intermediate point cloud and all the conversion point clouds are fused to obtain a determined target image point cloud, and based on the target image point cloud, a fused coating image is obtained.

[0054] In a possible implementation, the second coating defect classification module is used to input the fused coating image into a pre-trained second coating defect classification model so that the second coating defect classification model extracts defect features from the fused coating image, specifically comprising:

[0055] Inputting the fused coating image into a pre-trained second coating defect classification model, so that the second coating defect classification model performs grayscale processing on the fused coating image to obtain a fused coating grayscale image, extracting defect edges on the fused coating grayscale image to obtain a second defect shape, calculating a second defect shape area corresponding to the second defect shape, and determining a second defect shape position of the second defect shape on the fused coating image;

[0056] The second defect shape, the second defect shape area and the second defect shape position are respectively vectorized to obtain a second defect shape vector, a second defect shape area vector and a second defect shape position vector, and the second defect shape vector, the second defect shape area vector and the second defect shape position vector are associated to obtain a second defect association vector, and the second defect association vector is used as the second coating defect feature of the fused coating image.

[0057] The present application provides a coating defect detection device based on a neural network, further comprising: a coating defect classification result correction module;

[0058] The coating defect classification result correction module is used to perform a tapping process on the coating area, collect the voiceprint data corresponding to the coating area, and perform Fourier transform processing on the voiceprint data to obtain a voiceprint spectrum image;

[0059] Comparing the voiceprint spectrum image with a preset standard coating defect voiceprint feature library to determine a third coating defect classification result corresponding to the coating area;

[0060] The first coating defect classification result is corrected based on the third coating defect classification result to obtain a corrected first coating defect classification result, and the corrected first coating defect classification result is updated as the first coating defect classification result.

[0061] In a possible implementation, the predicted coating defect determination module is used to integrate the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect corresponding to the coating area, specifically including:

[0062] Using the first coating defect classification result as the predicted coating internal defect corresponding to the coating area; wherein the first coating defect classification result includes a first coating defect type, and a first coating defect shape area and the first coating defect shape position corresponding to the first coating defect type;

[0063] Using the second coating defect classification result as the predicted coating surface defect corresponding to the coating area; wherein the second coating defect classification result includes a second coating defect type, and a second coating defect shape area and a second coating defect shape position corresponding to the second coating defect type;

[0064] The predicted coating internal defects and the predicted coating surface defects are integrated to obtain the predicted coating defects corresponding to the coating area.

[0065] The present invention also provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the neural network-based coating defect detection method as described in any one of the above is implemented.

[0066] The present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the neural network-based coating defect detection method as described in any one of the above.

[0067] The coating defect detection method, device, terminal device and storage medium based on neural network in the embodiment of the present application have the following beneficial effects compared with the prior art:

[0068] By acquiring ultrasonic images and fused coating images at multiple angles, wherein the ultrasonic images can provide defect information inside the coating, and the fused coating images at multiple angles can provide defect information on the coating surface, the combination of the two can provide more comprehensive defect feature information of the coating area; and based on the pre-trained first coating defect classification model and the second coating defect classification model, in-depth defect feature extraction is performed on the ultrasonic image and the fused coating image respectively, which can increase the reliability of the defect detection results; finally, combining the first coating defect classification results and the second coating defect classification results, it is possible to reduce the misjudgment that may be introduced by a single detection method, thereby improving the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of an embodiment of a coating defect detection method based on a neural network provided in the present application;

[0070] Figure 2It is a structural schematic diagram of an embodiment of a coating defect detection device based on a neural network provided by the present application;

[0071] Figure 3 It is a structural schematic diagram of a terminal device provided by the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] Example 1, see Figure 1 , Figure 1 : is a flow chart of an embodiment of a coating defect detection method based on a neural network provided in the present application, such as Figure 1 As shown, the method includes steps 101 to 104, which are specifically as follows:

[0074] Step 101: transmitting a first ultrasonic signal array to a coating area, receiving a second ultrasonic signal array returned by the coating area, and generating an ultrasonic image of the coating area based on the second ultrasonic signal array.

[0075] In one embodiment, a plurality of ultrasonic guided wave transducers are selected, and based on the regional shape of the coating area, the plurality of ultrasonic guided wave transducers are evenly arranged at a certain distance to generate an ultrasonic guided wave transducer matrix corresponding to the regional shape, and each array element in the ultrasonic guided wave transducer matrix is ​​sequentially used as a transmitter to transmit a first ultrasonic signal to the coating area to form a first ultrasonic signal array, and all array elements in the ultrasonic guided wave transducer matrix are used as receivers of each first ultrasonic signal to receive a second ultrasonic signal returned from the coating area to obtain a second ultrasonic signal array.

[0076] Preferably, the ultrasonic guided wave transducer matrix may be a ring matrix, a linear matrix or a rectangular matrix, depending on the shape of the coating area to be detected.

[0077] In one embodiment, for the ultrasonic guided wave transducer matrix, a coordinate system is established with the geometric center of the ultrasonic guided wave transducer matrix as the origin, wherein the x-axis is along the length direction of the coating area and the z-axis is along the height direction of the coating area, a two-dimensional image of the coating area is constructed, and based on the number of ultrasonic guided wave transducers in the ultrasonic guided wave transducer matrix, the two-dimensional image of the coating area is divided into a number of pixel points; so that the second ultrasonic signal array returned by the coating area can be represented in the two-dimensional image of the coating area, wherein the coordinates of the two-dimensional image correspond one-to-one to the coordinate position of the coating area.

[0078] In one embodiment, when an ultrasonic image of the coating area is generated based on the second ultrasonic signal array, the signal amplitude corresponding to each second ultrasonic signal in the second ultrasonic signal array is obtained, and the signal amplitude is converted into a brightness value; the signal position corresponding to each second ultrasonic signal in the second ultrasonic signal array is determined, and based on the signal position, the brightness values ​​are arranged to generate an ultrasonic image of the coating area.

[0079] Specifically, when converting the signal amplitude into a brightness value, the signal amplitude is directly mapped into a brightness value range corresponding to the brightness value through linear conversion, thereby realizing conversion between the signal amplitude and the brightness value.

[0080] Specifically, the signal amplitude corresponding to each second ultrasonic signal in the second ultrasonic signal array is represented in the form of a brightness value on the two-dimensional image corresponding to the coating area, which can realize a comprehensive scan of the coating area and distinguish tiny defects inside the coating area; the size of the signal amplitude is reflected in the brightness of the color on the two-dimensional image, the darker positions in the two-dimensional image correspond to areas without defects in the coating area, and the highlighted positions in the two-dimensional image correspond to areas with defects in the coating area.

[0081] Step 102: Input the ultrasonic image into a pre-trained first coating defect classification model so that the first coating defect classification model extracts defect features from the ultrasonic image, compares the extracted first coating defect features with a standard coating defect feature library, and outputs a first coating defect classification result of the coating area.

[0082] In one embodiment, the pre-trained first coating defect classification model includes a first convolutional neural network model and a first support vector machine model, wherein the output of the first convolutional neural network model is connected to the input of the first support vector machine model.

[0083] Specifically, the first convolutional neural network model is used to extract defect features from the ultrasonic image to obtain a first coating defect feature, and input the first coating defect feature into a first support vector machine model.

[0084] Specifically, the first support vector machine model is used to receive the first coating defect feature, compare the first coating defect feature with a standard coating defect feature library, and output a first coating defect classification result of the coating area.

[0085] In one embodiment, the ultrasonic image is input into a pre-trained first coating defect classification model so that the first coating defect classification model obtains the brightness value corresponding to each pixel point in the ultrasonic image, and performs mean normalization on the brightness value to obtain a normalized brightness value.

[0086] Specifically, based on the brightness values ​​corresponding to each pixel point in the ultrasonic image, the brightness mean and brightness standard deviation corresponding to the ultrasonic image are calculated; when the brightness values ​​are mean-normalized, the brightness values ​​corresponding to each pixel point in the ultrasonic image are subtracted from the brightness mean to obtain the brightness difference, and the brightness difference is divided by the brightness standard deviation to obtain the normalized brightness value corresponding to each pixel point; by performing brightness normalization on the ultrasonic image, the brightness distribution of the ultrasonic image can be made uniform and the contrast of the image can be enhanced.

[0087] In one embodiment, the standard normalized brightness value corresponding to each pixel point is selected, and the brightness difference coefficient is calculated based on the normalized brightness value and the standard normalized brightness value. Based on the brightness difference coefficient, the defect probability corresponding to each pixel point in the ultrasonic image is calculated, and a defect probability map corresponding to the ultrasonic image is generated.

[0088] Specifically, the brightness difference coefficient is usually used to measure the difference between the brightness value of each pixel in the image and its standard brightness value; the brightness difference coefficient is calculated by substituting the normalized brightness value and the standard normalized brightness value into a preset brightness difference coefficient calculation formula; wherein the brightness difference coefficient calculation formula is as follows:

[0089] D(x,y)=|I(x,y)-S(x,y)| / S(x,y);

[0090] Where D(x,y) is the brightness difference coefficient of the ultrasonic image at the coordinate (x,y); I(x,y) is the normalized brightness value of the ultrasonic image at the coordinate (x,y); and S(x,y) is the standard normalized brightness value of the ultrasonic image at the coordinate (x,y).

[0091] Preferably, the standard normalized brightness value is the brightness value of a pixel point in an ultrasonic image corresponding to defect-free coating under an ideal state.

[0092] Specifically, the brightness difference coefficient D(x,y) represents the difference between the actual brightness value and the standard brightness value, relative to the ratio of the standard brightness value; if D(x,y) is large, it means that the brightness of the pixel point deviates greatly from the standard brightness, which may indicate a defect; if D(x,y) is close to 0, it means that the brightness of the pixel point is close to the standard brightness, and the possibility of a defect is small.

[0093] Specifically, based on the brightness difference coefficient, when calculating the defect probability corresponding to each pixel point in the ultrasonic image, the brightness difference coefficient threshold θ is obtained, and the brightness difference coefficient is compared with the brightness difference coefficient threshold. If the brightness difference coefficient is greater than the brightness difference coefficient threshold, the defect probability corresponding to the pixel point is set to 1; if the brightness difference coefficient is not greater than the brightness difference coefficient threshold, the defect probability corresponding to the pixel point is set to 0.

[0094] Specifically, for each pixel, the defect probability (x, y) is calculated using the following formula:

[0095]

[0096] Specifically, the calculated defect probability corresponding to each pixel is used as the defect probability value corresponding to each pixel point in the ultrasonic image to generate a defect probability map with the same size as the original ultrasonic image.

[0097] In one embodiment, defect edges are extracted from the defect probability map to determine a first defect shape, a first defect shape area corresponding to the first defect shape is calculated, and a first defect shape position of the first defect shape on the ultrasonic image is determined.

[0098] Specifically, a binary image algorithm is used to extract defect edges from the defect probability map to identify defect boundaries and obtain a first defect shape; wherein the binary image algorithm includes but is not limited to a Canny edge detection algorithm and a Roberts operator.

[0099] Preferably, the defect probability value of each pixel in the defect probability map is obtained based on the Roberts operator, and the change rate of the defect probability value of each pixel and its neighborhood is quantified to determine the edge point, and based on the edge point, the defect edge area is extracted to obtain the first defect shape.

[0100] Specifically, a connected region labeling algorithm is applied, wherein the connected region labeling algorithm includes but is not limited to bwlabel and connectedComponents, etc., which are used to mark and identify the first defect shape identified as a defect by the binary image algorithm, and for each marked first defect shape, the number of pixels in the first defect shape is calculated, and the number of pixels is used as the first defect shape area corresponding to the first defect shape.

[0101] Specifically, based on the area of ​​the first defect shape, the bounding box position corresponding to the first defect shape is determined, based on the upper left corner coordinates and lower right corner coordinates corresponding to the bounding box, the center point coordinates of the first defect shape are determined, and the center point coordinates are set as the first defect shape position of the first defect shape on the ultrasonic image.

[0102] In one embodiment, the first defect shape, the first defect shape area and the first defect shape position are respectively vectorized to obtain a first defect shape vector, a first defect shape area vector and a first defect shape position vector, and the first defect shape vector, the first defect shape area vector and the first defect shape position vector are associated to obtain a first defect association vector, and the first defect association vector is used as the first coating defect feature of the ultrasonic image.

[0103] Specifically, when the first defect shape is vectorized, a preset standard shape set is used, wherein the standard shape set includes multiple standard shapes and a vector value corresponding to each standard brick shape; the first defect shape and the preset standard shape set are matched to obtain a matching standard shape; and the vector value corresponding to the matching standard shape is used as the first defect shape vector.

[0104] Specifically, when the first defect shape area is vectorized, the first defect shape area is normalized to obtain a normalized first defect shape area, and the normalized first defect shape area is used as a first defect shape area vector;

[0105] Specifically, when vectorization is performed on the first defect shape position, the first defect shape position is directly used as the first defect shape area vector.

[0106] Specifically, the first defect association vector is formed by associating the first defect shape vector, the first defect shape area vector and the first defect shape position vector, wherein the first defect association vector can be expressed as [shape vector, area vector, position vector].

[0107] In one embodiment, a sample defect association vector library containing various known defect types is established as a standard coating defect feature library; this standard coating defect feature library can be obtained by collecting a large number of standard coating defect samples, and marking the defect type corresponding to each standard coating defect sample, and then based on the same feature extraction method as mentioned above, determining the sample defect association vector corresponding to each standard coating defect sample, integrating all sample defect association vectors, and obtaining the standard coating defect feature library.

[0108] In one embodiment, the extracted first coating defect feature is compared with a standard coating defect feature library, and when the first coating defect classification result of the coating area is output, the similarity between the extracted first coating defect feature vector and each standard coating defect feature in the standard coating defect feature library is calculated respectively, and the target standard coating defect feature corresponding to the maximum similarity value is obtained, and the defect type corresponding to the target standard coating defect feature is used as the first coating defect type in the first coating defect classification result, and the first defect shape area vector and the first defect shape position vector in the first defect association vector are used as the first coating defect shape area and the first coating defect shape position corresponding to the first coating defect type.

[0109] Preferably, when calculating the similarity between the extracted first coating defect feature vector and each standard coating defect feature in the standard coating defect feature library, a measurement method such as Euclidean distance or cosine similarity can be used for calculation.

[0110] In one embodiment, when multiple first coating defect features are extracted, multiple first coating defect classification results are finally obtained.

[0111] In one embodiment, the coated area is also tapped to collect the corresponding voiceprint data, and the voiceprint data is subjected to Fourier transform to obtain a voiceprint spectrum image.

[0112] Specifically, in order to stimulate the acoustic response of the coated area and thus obtain its voiceprint characteristics, the coated area is physically tapped, and the voiceprint data generated by the coated area is collected based on a data acquisition device; wherein the data acquisition device includes but is not limited to a high-precision microphone.

[0113] Specifically, the collected voiceprint data is subjected to Fourier transform, which is converted from the time domain to the frequency domain to obtain a voiceprint spectrum image.

[0114] In one embodiment, the voiceprint spectrum image is compared with a preset standard coating defect voiceprint feature library to determine a third coating defect classification result corresponding to the coating area.

[0115] Specifically, a standard coating defect voiceprint feature library is established, which contains standard voiceprint spectrum images corresponding to various known defect types. This standard coating defect feature library can be obtained by collecting defect types corresponding to a large number of standard coating defect samples, and then using the same feature extraction method as mentioned above to determine the standard voiceprint spectrum images corresponding to each standard coating defect sample, and integrating all standard voiceprint spectrum images to obtain the standard coating defect voiceprint feature library.

[0116] Specifically, the similarities between the voiceprint spectrum image and each standard voiceprint spectrum image in the standard coating defect voiceprint feature library are calculated respectively, the target standard voiceprint spectrum image corresponding to the maximum similarity value is obtained, and the defect type corresponding to the target standard voiceprint spectrum image is used as the third coating defect type in the third coating defect classification result.

[0117] In one embodiment, the first coating defect classification result is corrected based on the third coating defect classification result to obtain a corrected first coating defect classification result, and the corrected first coating defect classification result is updated to the first coating defect classification result.

[0118] Specifically, all first coating defect types in the first coating defect classification result are obtained, and the number of first coating defect types corresponding to the same defect type among all the first coating defect types is counted.

[0119] Specifically, all third coating defect types in the third coating defect classification result are obtained, and the number of second coating defect types corresponding to the same defect type in all the third coating defect types is counted.

[0120] Specifically, when it is determined that the number of first coating defect types corresponding to the same defect type in all the first coating defect types is greater than the number of second coating defect types corresponding to the same defect type in all the third coating defect types, the first coating defect classification result is directly used as the corrected first coating defect classification result.

[0121] Specifically, when it is determined that the number of first coating defect types corresponding to the same defect type in all the first coating defect types is not greater than the number of second coating defect types corresponding to the same defect type in all the third coating defect types, the difference between the number of first coating defect types corresponding to the same defect type in all the first coating defect types and the number of second coating defect types corresponding to the same defect type in the third coating defect types is calculated, and all first defect shape area vectors corresponding to the same defect type in all the first coating defect types are obtained, and all first defect shape area vectors are sorted in descending order, and based on the difference in number, the number of first defect shape area vectors arranged on the right side in the first defect shape area vector sequence are deleted to obtain a corrected first defect shape area vector sequence, and based on the corrected first defect shape area vector sequence, it is updated to the first coating defect classification result to obtain a corrected first coating defect classification result, and the corrected first coating defect classification result is updated to the first coating defect classification result.

[0122] Step 103: collecting coating images of the coating area at multiple angles, and fusing the coating images at all angles to obtain a fused coating image.

[0123] In one embodiment, since the shape of the coating area mainly depends on the shape of the coated object, that is, the coating area can be flat or curved, however, when the coating area is a curved area, the raised part may cause occlusion. Therefore, in order to reduce the impact of occlusion, collecting the coating images corresponding to the coating area from multiple angles can improve the accuracy of subsequent operations.

[0124] In one embodiment, when collecting coating images of the coating area at multiple angles, the center point of the coating area is used as the central axis, and the image acquisition device is controlled to collect images of the coating area from left to right at a preset rotation angle. At the same time, the image acquisition device is also controlled to collect images of the coating area from top to bottom at a preset rotation angle to obtain coating images of the coating area at multiple angles.

[0125] Preferably, image acquisition devices may be deployed above, to the left, to the right, in front and behind the coating area, respectively, so that image acquisition devices based on different orientations can capture the coating image of the coating area to obtain coating images at multiple angles.

[0126] In one embodiment, the coating images at all angles are converted into coating mask matrices, and coating depth images corresponding to the coating images at all angles are obtained at the same time. Based on the coating mask matrix and the coating depth image, image point clouds of the coating images at all angles are obtained.

[0127] Specifically, the coated areas involved in the coated images at all angles are set to non-zero values, and the non-coated areas involved in the coated images at all angles are set to zero values, so that the coated images at all angles are converted into their corresponding coating mask matrices.

[0128] Specifically, the coating depth image can be obtained by predicting the depth of each pixel in the coating image through a deep learning model; preferably, the training process of the deep learning model can obtain the depth information of the coating sample image by using a professional depth camera or lidar, or use an existing data set to generate a data set for training, the data set includes the coating sample image and its corresponding depth map; use the preprocessed data set to train the pre-constructed deep learning model, and during the model training process, use loss functions such as root mean square error or mean relative error to optimize model parameters, and data enhancement, regularization, dropout and other technologies can also be used to prevent model overfitting and improve the generalization ability of the model.

[0129] Specifically, based on the depth value corresponding to each pixel in the coating depth image, a depth matrix is ​​constructed, and the depth matrix and the coating mask matrix are convolved to obtain a target depth matrix, and the coating depth image corresponding to the target depth matrix is ​​used as the target coating depth image, and the depth of field value corresponding to the non-coating area involved in the target coating depth image is set to zero.

[0130] Specifically, determine the first pixel point in the target coating depth image whose depth of field value is non-zero, and for any of the first pixel points, obtain the pixel coordinates and depth of field value corresponding to the first pixel point; and convert the pixel coordinates and depth of field value corresponding to the first pixel point into three-dimensional coordinates in the world coordinate system corresponding to the first pixel point; determine the three-dimensional coordinates in the world coordinate system corresponding to the first pixel point as the three-dimensional point corresponding to the first pixel point; integrate the three-dimensional points corresponding to each of the first pixel points to obtain the image point cloud of the coating image at all angles.

[0131] In one embodiment, the image centroids corresponding to the image point clouds of the coating image at all angles are determined, and the image point clouds of the coating image at all angles are converted into intermediate image point clouds based on the image centroids.

[0132] Specifically, the average three-dimensional coordinates of all three-dimensional points in the image point cloud of the coating image at any angle are obtained respectively, and the average three-dimensional coordinates are used as the image centroid corresponding to the image point clouds of the coating image at all angles.

[0133] Specifically, the three-dimensional coordinates of each three-dimensional point in the image point cloud of the coating image at any angle are subtracted from the centroid to obtain the intermediate image point clouds corresponding to the coating images at all angles.

[0134] In one embodiment, a reference intermediate image point cloud is selected from all intermediate image point clouds, and based on the reference intermediate image point cloud, the remaining intermediate image point clouds are registered to obtain conversion point clouds corresponding to the remaining intermediate image point clouds.

[0135] Specifically, an intermediate image point cloud with a shooting angle taking the center point of the coating area as the central axis is selected from all intermediate image point clouds as a reference intermediate image point cloud.

[0136] Specifically, a target number of points are selected from the remaining intermediate image point cloud; and a preset KD-tree algorithm is used to select the nearest neighboring points corresponding to each of the target number of points from the reference intermediate image point cloud; the target number of points and the nearest neighboring points corresponding to each of the target number of points form a target number of nearest neighboring point pairs, and all target number of nearest neighboring point pairs are integrated to form a nearest neighboring point pair set; according to the nearest neighboring point pair set, a target rotation matrix and a target offset vector are determined; and based on the target rotation matrix and the target offset vector, the remaining intermediate image point clouds are respectively transformed to obtain initial transformed point clouds corresponding to each of the remaining intermediate image point clouds; the transformation error between the reference intermediate image point cloud and the initial transformed point cloud is obtained, and according to the transformation error, the transformed point clouds corresponding to each of the remaining intermediate image point clouds are determined.

[0137] In one embodiment, based on a preset coating image reference point cloud, the reference intermediate point cloud and all conversion point clouds are fused to obtain a determined target image point cloud, and based on the target image point cloud, a fused coating image is obtained.

[0138] Specifically, the conversion point clouds corresponding to the reference intermediate image point cloud and the remaining intermediate image point clouds are merged to obtain a complete coating image point cloud, and a verification point cloud is selected from the complete coating image point cloud, and the difference value between the verification point cloud and the preset coating image reference point cloud is calculated. When the difference value meets the preset difference threshold, the conversion point clouds corresponding to the reference intermediate image point cloud and the remaining intermediate image point clouds are fused to obtain a fused point cloud image, and the three-dimensional coordinates corresponding to each three-dimensional point in the fused point cloud image are mapped to a two-dimensional coordinate system to obtain a fused coating image.

[0139] Specifically, when calculating the difference value between the verification point cloud and the preset coating image reference point cloud, the chamfer distance between the verification point cloud and the preset coating image reference point cloud is obtained, and the chamfer distance is used as the difference value between the verification point cloud and the preset coating image reference point cloud.

[0140] Step 104: Input the fused coating image into a pre-trained second coating defect classification model, so that the second coating defect classification model extracts defect features from the fused coating image, and compares the extracted second coating defect features with a standard coating defect feature library, and outputs the second coating defect classification result of the coating area.

[0141] In one embodiment, the pre-trained second coating defect classification model includes a second convolutional neural network model and a second support vector machine model, wherein the output of the second convolutional neural network model is connected to the input of the second support vector machine model.

[0142] Specifically, the second convolutional neural network model is used to extract defect features from the ultrasonic image to obtain second coating defect features, and input the second coating defect features into the second support vector machine model.

[0143] Specifically, the second support vector machine model is used to receive the second coating defect feature, compare the second coating defect feature with a standard coating defect feature library, and output a second coating defect classification result of the coating area.

[0144] In one embodiment, the fused coating image is input into a pre-trained second coating defect classification model, so that the second coating defect classification model grayscales the fused coating image to obtain a fused coating grayscale image, performs defect edge extraction on the fused coating grayscale image to obtain a second defect shape, calculates the second defect shape area corresponding to the second defect shape, and determines the second defect shape position of the second defect shape on the fused coating image.

[0145] Specifically, the Canny edge detection operator is used to extract the defect edge of the fused coating grayscale image to realize the recognition of the defect boundary and obtain the second defect shape.

[0146] Preferably, the grayscale value of each pixel in the fused coated grayscale image can be obtained based on the Roberts operator, and the grayscale change rate of each pixel and its neighborhood can be quantified to determine the edge points. Based on the edge points, the defect edge area is extracted to obtain the second defect shape.

[0147] Specifically, a connected area labeling algorithm is applied, wherein the connected area labeling algorithm includes but is not limited to bwlabel and connectedComponents, etc., which is used to mark and identify the second defect shape identified as a defect by the edge detection algorithm, and for each marked second defect shape, the number of pixels in the second defect shape is calculated, and the number of pixels is used as the second defect shape area corresponding to the second defect shape.

[0148] Specifically, based on the area of ​​the second defect shape, the bounding box position corresponding to the second defect shape is determined, based on the upper left corner coordinates and lower right corner coordinates corresponding to the bounding box, the second center point coordinates of the second defect shape are determined, and the second center point coordinates are set as the second defect shape position of the second defect shape on the ultrasonic image.

[0149] In one embodiment, the second defect shape, the second defect shape area and the second defect shape position are respectively vectorized to obtain a second defect shape vector, a second defect shape area vector and a second defect shape position vector, and the second defect shape vector, the second defect shape area vector and the second defect shape position vector are associated to obtain a second defect association vector, and the second defect association vector is used as the second coating defect feature of the fused coating image.

[0150] Specifically, when the second defect shape is vectorized, a preset standard shape set is used, wherein the standard shape set includes multiple standard shapes and a vector value corresponding to each standard brick shape; the second defect shape and the preset standard shape set are matched to obtain a matching standard shape; and the vector value corresponding to the matching standard shape is used as the second defect shape vector.

[0151] Specifically, when the second defect shape area is vectorized, the second defect shape area is normalized to obtain a normalized second defect shape area, and the normalized second defect shape area is used as a second defect shape area vector;

[0152] Specifically, when the second defect shape position is vectorized, the second defect shape position is directly used as the second defect shape area vector.

[0153] Specifically, the second defect association vector is formed by associating the second defect shape vector, the second defect shape area vector and the second defect shape position vector, wherein the second defect association vector can be expressed as [shape vector, area vector, position vector].

[0154] In one embodiment, the extracted second coating defect feature is compared with a standard coating defect feature library, and when the second coating defect classification result of the coating area is output, the similarity between the extracted second coating defect feature vector and each standard coating defect feature in the standard coating defect feature library is calculated respectively, and the target standard coating defect feature corresponding to the maximum similarity value is obtained, and the defect type corresponding to the target standard coating defect feature is used as the second coating defect type in the second coating defect classification result, and the second defect shape area vector and the second defect shape position vector in the second defect association vector are used as the second coating defect shape area and the second coating defect shape position corresponding to the second coating defect type.

[0155] Preferably, when calculating the similarity between the extracted second coating defect feature vector and each standard coating defect feature in the standard coating defect feature library, a measurement method such as Euclidean distance or cosine similarity can also be used for calculation.

[0156] In one embodiment, when multiple second coating defect features are extracted, multiple second coating defect classification results are finally obtained.

[0157] Step 105: Integrate the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect corresponding to the coating area.

[0158] In one embodiment, the first coating defect classification result is used as the predicted coating internal defect corresponding to the coating area; wherein the first coating defect classification result includes a first coating defect type, and a first coating defect shape area and the first coating defect shape position corresponding to the first coating defect type.

[0159] In one embodiment, the second coating defect classification result is used as the predicted coating surface defect corresponding to the coating area; wherein the second coating defect classification result includes a second coating defect type, and a second coating defect shape area and a second coating defect shape position corresponding to the second coating defect type.

[0160] In one embodiment, the predicted coating internal defects and the predicted coating surface defects are integrated to obtain the predicted coating defects corresponding to the coating area.

[0161] Example 2, see Figure 2 , Figure 2 is a schematic diagram of the structure of an embodiment of a coating defect detection device based on a neural network provided by the present application, such as Figure 2As shown, the device includes an ultrasonic image generation module 201, a first coating defect classification module 202, a coating image fusion module 203, a second coating defect classification module 204 and a predicted coating defect determination module 205, which are specifically as follows:

[0162] The ultrasonic image generation module 201 is used to transmit a first ultrasonic signal array to the coating area, receive a second ultrasonic signal array returned by the coating area, and generate an ultrasonic image of the coating area based on the second ultrasonic signal array.

[0163] The first coating defect classification module 202 is used to input the ultrasonic image into a pre-trained first coating defect classification model so that the first coating defect classification model extracts defect features from the ultrasonic image, compares the extracted first coating defect features with a standard coating defect feature library, and outputs the first coating defect classification result of the coating area.

[0164] The coating image fusion module 203 is used to collect coating images of the coating area at multiple angles, and fuse the coating images at all angles to obtain a fused coating image.

[0165] The second coating defect classification module 204 is used to input the fused coating image into a pre-trained second coating defect classification model so that the second coating defect classification model extracts defect features from the fused coating image, compares the extracted second coating defect features with a standard coating defect feature library, and outputs the second coating defect classification result of the coating area.

[0166] The predicted coating defect determination module 205 is used to integrate the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect corresponding to the coating area.

[0167] In one embodiment, the ultrasonic image generation module 201 is used to generate an ultrasonic image of the coating area based on the second ultrasonic signal array, specifically including: obtaining the signal amplitude corresponding to each second ultrasonic signal in the second ultrasonic signal array, and converting the signal amplitude into a brightness value; determining the signal position corresponding to each second ultrasonic signal in the second ultrasonic signal array, and arranging the brightness values ​​based on the signal position to generate an ultrasonic image of the coating area.

[0168] In one embodiment, the first coating defect classification module 202 is used to input the ultrasonic image into a pre-trained first coating defect classification model so that the first coating defect classification model extracts defect features from the ultrasonic image, specifically including: inputting the ultrasonic image into the pre-trained first coating defect classification model so that the first coating defect classification model obtains the brightness value corresponding to each pixel in the ultrasonic image, and performs mean normalization processing on the brightness value to obtain a normalized brightness value; selecting the standard normalized brightness value corresponding to each pixel, calculating the brightness difference coefficient based on the normalized brightness value and the standard normalized brightness value, and calculating the defect feature corresponding to each pixel in the ultrasonic image based on the brightness difference coefficient. The defect probability is calculated, and a defect probability map corresponding to the ultrasonic image is generated; defect edges are extracted from the defect probability map to determine a first defect shape, a first defect shape area corresponding to the first defect shape is calculated, and a first defect shape position of the first defect shape on the ultrasonic image is determined; the first defect shape, the first defect shape area, and the first defect shape position are vectorized respectively to obtain a first defect shape vector, a first defect shape area vector, and a first defect shape position vector, and the first defect shape vector, the first defect shape area vector, and the first defect shape position vector are associated to obtain a first defect association vector, and the first defect association vector is used as the first coating defect feature of the ultrasonic image.

[0169] In one embodiment, the coating image fusion module 203 is used to fuse the coating images at all angles to obtain a fused coating image, specifically including: converting the coating images at all angles into a coating mask matrix, and simultaneously obtaining the coating depth images corresponding to each of the coating images at all angles, and obtaining the image point clouds of the coating images at all angles based on the coating mask matrix and the coating depth image; determining the image centroid corresponding to each of the image point clouds of the coating images at all angles, and converting the image point clouds of the coating images at all angles into intermediate image point clouds based on the image centroid; selecting a reference intermediate image point cloud from all intermediate image point clouds, and based on the reference intermediate image point cloud, registering the remaining intermediate image point clouds to obtain the conversion point clouds corresponding to each of the remaining intermediate image point clouds; based on a preset coating image reference point cloud, fusing the reference intermediate point cloud and all the conversion point clouds to obtain a determined target image point cloud, and obtaining a fused coating image based on the target image point cloud.

[0170] In one embodiment, the second coating defect classification module 204 is used to input the fused coating image into a pre-trained second coating defect classification model so that the second coating defect classification model extracts defect features from the fused coating image, specifically including: inputting the fused coating image into a pre-trained second coating defect classification model so that the second coating defect classification model performs grayscale processing on the fused coating image to obtain a fused coating grayscale image, extracting defect edges from the fused coating grayscale image to obtain a second defect shape, calculating a second defect shape area corresponding to the second defect shape, and determining a second defect shape position of the second defect shape on the fused coating image; respectively vectorizing the second defect shape, the second defect shape area, and the second defect shape position to obtain a second defect shape vector, a second defect shape area vector, and a second defect shape position vector, and associating the second defect shape vector, the second defect shape area vector, and the second defect shape position vector to obtain a second defect association vector, and using the second defect association vector as the second coating defect feature of the fused coating image.

[0171] A coating defect detection device based on a neural network provided in this embodiment also includes: a coating defect classification result correction module.

[0172] In one embodiment, the coating defect classification result correction module is used to perform a tapping process on the coating area, collect voiceprint data corresponding to the coating area, and perform Fourier transform processing on the voiceprint data to obtain a voiceprint spectrum image; compare the voiceprint spectrum image with a preset standard coating defect voiceprint feature library to determine a third coating defect classification result corresponding to the coating area; based on the third coating defect classification result, correct the first coating defect classification result to obtain a corrected first coating defect classification result, and update the corrected first coating defect classification result to the first coating defect classification result.

[0173] In one embodiment, the predicted coating defect determination module 205 is used to integrate the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect corresponding to the coating area, specifically including: using the first coating defect classification result as the predicted coating internal defect corresponding to the coating area; wherein the first coating defect classification result includes a first coating defect type, and a first coating defect shape area and the first coating defect shape position corresponding to the first coating defect type; using the second coating defect classification result as the predicted coating surface defect corresponding to the coating area; wherein the second coating defect classification result includes a second coating defect type, and a second coating defect shape area and the second coating defect shape position corresponding to the second coating defect type; integrating the predicted coating internal defect and the predicted coating surface defect to obtain the predicted coating defect corresponding to the coating area.

[0174] The coating defect detection device based on neural network can implement the coating defect detection method based on neural network in the above method embodiment. The options in the above method embodiment are also applicable to this embodiment and will not be described in detail here.

[0175] Figure 3 A schematic diagram of the structure of a terminal device. Figure 3 As shown, the terminal device 3 of this embodiment includes: at least one processor 301 ( Figure 3 Only one is shown in the figure) a processor, a memory 302, and a computer program 303 stored in the memory 302 and executable on at least one processor 301, and when the processor 301 executes the computer program 303, the steps in any of the above method embodiments are implemented.

[0176] The terminal device 3 may be a computing device such as a smart phone, a laptop, a tablet computer, and a desktop computer. The terminal device may include but is not limited to a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of the terminal device 3 and does not constitute a limitation on the terminal device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0177] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0178] In some embodiments, the memory 302 may be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. In other embodiments, the memory 302 may also be an external storage device of the terminal device 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 3. Further, the memory 302 may also include both an internal storage unit of the terminal device 3 and an external storage device. The memory 302 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program, etc. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0179] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0180] In several embodiments provided in the present application, it is understood that each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes also be executed in the opposite order, depending on the functions involved.

[0181] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a terminal device to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0182] In summary, the present application provides a coating defect detection method, apparatus, terminal device and storage medium based on a neural network, which transmits a first ultrasonic signal array to the coating area, and determines the ultrasonic image of the coating area based on the returned second ultrasonic signal array; inputs the ultrasonic image into a first coating defect classification model, compares the extracted first coating defect feature with a standard coating defect feature library, and outputs a first coating defect classification result; fuses the coating images at all angles to obtain and inputs the fused coating image into a second coating defect classification model, compares the extracted second coating defect feature with a standard coating defect feature library, and outputs a second coating defect classification result; integrates the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect of the coating area; compared with the prior art, the technical solution of the present invention can improve the accuracy and reliability of defect detection.

[0183] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present application. These improvements and substitutions should also be regarded as the scope of protection of the present application.

Claims

1. A coating defect detection method based on neural network, characterized in that: include: Transmitting a first ultrasonic signal array to a coating area, and receiving a second ultrasonic signal array returned by the coating area, and generating an ultrasonic image of the coating area based on the second ultrasonic signal array; Inputting the ultrasonic image into a pre-trained first coating defect classification model, so that the first coating defect classification model extracts defect features from the ultrasonic image, and compares the extracted first coating defect features with a standard coating defect feature library, and outputs a first coating defect classification result of the coating area; Collecting coating images of the coating area at multiple angles, and fusing the coating images at all angles to obtain a fused coating image; Inputting the fused coating image into a pre-trained second coating defect classification model, so that the second coating defect classification model extracts defect features from the fused coating image, and compares the extracted second coating defect features with a standard coating defect feature library, and outputs a second coating defect classification result of the coating area; The first coating defect classification result and the second coating defect classification result are integrated to determine a predicted coating defect corresponding to the coating area.

2. A coating defect detection method based on a neural network as claimed in claim 1, characterized in that: Generating an ultrasonic image of the coating area based on the second ultrasonic signal array specifically includes: Acquire the signal amplitude corresponding to each second ultrasonic signal in the second ultrasonic signal array, and convert the signal amplitude into a brightness value; The signal position corresponding to each second ultrasonic signal in the second ultrasonic signal array is determined, and the brightness values ​​are arranged based on the signal position to generate an ultrasonic image of the coating area.

3. A coating defect detection method based on a neural network as claimed in claim 1, characterized in that: Inputting the ultrasonic image into a pre-trained first coating defect classification model so that the first coating defect classification model extracts defect features from the ultrasonic image, specifically comprising: Inputting the ultrasonic image into a pre-trained first coating defect classification model so that the first coating defect classification model obtains the brightness value corresponding to each pixel point in the ultrasonic image, and performs mean normalization processing on the brightness value to obtain a normalized brightness value; Selecting the standard normalized brightness value corresponding to each pixel point, calculating the brightness difference coefficient based on the normalized brightness value and the standard normalized brightness value, and calculating the defect probability corresponding to each pixel point in the ultrasonic image based on the brightness difference coefficient, and generating a defect probability map corresponding to the ultrasonic image; Performing defect edge extraction on the defect probability map to determine a first defect shape, calculating a first defect shape area corresponding to the first defect shape, and determining a first defect shape position of the first defect shape on the ultrasonic image; The first defect shape, the first defect shape area and the first defect shape position are respectively vectorized to obtain a first defect shape vector, a first defect shape area vector and a first defect shape position vector, and the first defect shape vector, the first defect shape area vector and the first defect shape position vector are associated to obtain a first defect association vector, and the first defect association vector is used as the first coating defect feature of the ultrasonic image.

4. A coating defect detection method based on a neural network as claimed in claim 1, characterized in that: The coating images at all angles are fused to obtain a fused coating image, which specifically includes: Converting the coating images at all angles into coating mask matrices, and acquiring coating depth images corresponding to the coating images at all angles, and obtaining image point clouds of the coating images at all angles based on the coating mask matrix and the coating depth image; Determine the image centroids corresponding to the image point clouds of the coated image at all angles, and convert the image point clouds of the coated image at all angles into intermediate image point clouds according to the image centroids; Selecting a reference intermediate image point cloud from all intermediate image point clouds, and registering the remaining intermediate image point clouds based on the reference intermediate image point cloud to obtain conversion point clouds corresponding to the remaining intermediate image point clouds; Based on the preset coating image reference point cloud, the reference intermediate point cloud and all the conversion point clouds are fused to obtain a determined target image point cloud, and based on the target image point cloud, a fused coating image is obtained.

5. A coating defect detection method based on a neural network as claimed in claim 1, characterized in that: Inputting the fused coating image into a pre-trained second coating defect classification model so that the second coating defect classification model extracts defect features from the fused coating image, specifically comprising: Inputting the fused coating image into a pre-trained second coating defect classification model, so that the second coating defect classification model performs grayscale processing on the fused coating image to obtain a fused coating grayscale image, extracting defect edges on the fused coating grayscale image to obtain a second defect shape, calculating a second defect shape area corresponding to the second defect shape, and determining a second defect shape position of the second defect shape on the fused coating image; The second defect shape, the second defect shape area and the second defect shape position are respectively vectorized to obtain a second defect shape vector, a second defect shape area vector and a second defect shape position vector, and the second defect shape vector, the second defect shape area vector and the second defect shape position vector are associated to obtain a second defect association vector, and the second defect association vector is used as the second coating defect feature of the fused coating image.

6. A coating defect detection method based on a neural network as claimed in claim 1, characterized in that: Also includes: Performing a tapping process on the coated area, collecting voiceprint data corresponding to the coated area, and performing Fourier transform processing on the voiceprint data to obtain a voiceprint spectrum image; Comparing the voiceprint spectrum image with a preset standard coating defect voiceprint feature library to determine a third coating defect classification result corresponding to the coating area; The first coating defect classification result is corrected based on the third coating defect classification result to obtain a corrected first coating defect classification result, and the corrected first coating defect classification result is updated as the first coating defect classification result.

7. A coating defect detection method based on a neural network as claimed in claim 1, characterized in that: Integrating the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect corresponding to the coating area specifically includes: Using the first coating defect classification result as the predicted coating internal defect corresponding to the coating area; wherein the first coating defect classification result includes a first coating defect type, and a first coating defect shape area and the first coating defect shape position corresponding to the first coating defect type; Using the second coating defect classification result as the predicted coating surface defect corresponding to the coating area; wherein the second coating defect classification result includes a second coating defect type, and a second coating defect shape area and a second coating defect shape position corresponding to the second coating defect type; The predicted coating internal defects and the predicted coating surface defects are integrated to obtain the predicted coating defects corresponding to the coating area.

8. A coating defect detection device based on a neural network, characterized in that: include: An ultrasonic image generation module, a first coating defect classification module, a coating image fusion module, a second coating defect classification module and a predicted coating defect determination module; The ultrasonic image generation module is used to transmit a first ultrasonic signal array to the coating area, receive a second ultrasonic signal array returned by the coating area, and generate an ultrasonic image of the coating area based on the second ultrasonic signal array; The first coating defect classification module is used to input the ultrasonic image into a pre-trained first coating defect classification model, so that the first coating defect classification model extracts defect features from the ultrasonic image, compares the extracted first coating defect features with a standard coating defect feature library, and outputs a first coating defect classification result of the coating area; The coating image fusion module is used to collect coating images of the coating area at multiple angles, and fuse the coating images at all angles to obtain a fused coating image; The second coating defect classification module is used to input the fused coating image into a pre-trained second coating defect classification model, so that the second coating defect classification model extracts defect features from the fused coating image, and compares the extracted second coating defect features with a standard coating defect feature library, and outputs a second coating defect classification result of the coating area; The predicted coating defect determination module is used to integrate the first coating defect classification result and the second coating defect classification result to determine the predicted coating defect corresponding to the coating area.

9. A terminal device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the coating defect detection method based on neural network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the coating defect detection method based on a neural network as described in any one of claims 1 to 7.

Images