A defect detection method, apparatus, electronic device, and storage medium

Abstract

This application relates to the field of defect detection technology, and more particularly to a defect detection method, apparatus, electronic device, and storage medium for improving defect detection efficiency and accuracy. The method includes: acquiring a target image corresponding to an object to be detected, and a normal vector image corresponding to the target image, wherein the target image is obtained by full-light imaging of the object to be detected; performing at least three rounds of feature extraction based on the target image and the normal vector image, and concatenating two features to be fused obtained from the last round of feature extraction to obtain a target fusion feature; and performing defect detection on the target fusion feature on the target image to obtain at least one candidate defect region. Since this application obtains the target fusion feature based on the target image and the normal vector image, and then obtains the candidate defect region based on the target fusion feature, manual defect detection is unnecessary, effectively improving the efficiency and accuracy of defect detection.

Description

Technical Field

[0001] This application relates to the field of defect detection technology, and in particular to a defect detection method, apparatus, electronic device and storage medium. Background Technology

[0002] In modern industrial manufacturing, it is inevitable that some defective parts will appear among the various components produced by enterprises. On the one hand, in order to ensure product quality, enterprises must detect defective parts for subsequent processing; on the other hand, finding defective parts and analyzing the characteristics and proportions of various defects is crucial for improving production processes and increasing production line yield.

[0003] In related technologies, companies often employ workers to manually inspect products for defects. However, this method has several drawbacks. First, some defects are very small and difficult to observe, requiring a significant amount of time to inspect each one, resulting in low efficiency. Second, manual inspection relies solely on visual judgment, and the subjective differences among workers can lead to false positives and false negatives, resulting in large errors and low accuracy.

[0004] Therefore, improving the efficiency and accuracy of defect detection has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a defect detection method, apparatus, electronic device, and storage medium to improve the efficiency and accuracy of defect detection.

[0006] This application provides a defect detection method, including:

[0007] Obtain the image to be detected corresponding to the object to be detected, and the normal vector image corresponding to the image to be detected. The image to be detected is obtained by taking a full-light image of the object to be detected. The pixel value of each pixel in the normal vector image is the normal vector coordinate of the corresponding pixel in the image to be detected. The normal vector coordinate is used to represent the orientation of the object point on the surface of the object to be detected corresponding to the pixel.

[0008] At least three rounds of feature extraction are performed based on the image to be detected and the normal vector image, and the two features to be fused obtained in the last round of feature extraction are concatenated to obtain the target fusion feature;

[0009] Based on the target fusion features, defect detection is performed on the image to be detected to obtain at least one candidate defect region.

[0010] This application provides a defect detection device, comprising:

[0011] The acquisition unit is used to acquire a detection image corresponding to the detection object and a normal vector image corresponding to the detection image. The detection image is obtained by taking a full-light image of the detection object. The pixel value of each pixel in the normal vector image is the normal vector coordinate of the corresponding pixel in the detection image. The normal vector coordinate is used to characterize the orientation of the object point on the surface of the detection object corresponding to the pixel.

[0012] An extraction unit is used to perform at least three rounds of feature extraction based on the image to be detected and the normal vector image, and to concatenate the two features to be fused obtained in the last round of feature extraction to obtain the target fusion feature;

[0013] The detection unit is used to perform defect detection on the image to be detected based on the target fusion features to obtain at least one candidate defect region.

[0014] Optionally, the apparatus further includes a determining unit for:

[0015] For each candidate defect region, perform the following operations:

[0016] The depth value of each pixel is determined based on the normal vector coordinates of each pixel within a candidate defect region.

[0017] If any of the depth values ​​is greater than a preset depth threshold, then the candidate defect region is taken as the target defect region.

[0018] Optionally, the determining unit is specifically used for:

[0019] Based on the position of each pixel within a candidate defect region, the pixels are divided to obtain each boundary pixel and each intermediate pixel.

[0020] Set the depth value of each boundary pixel to the corresponding preset value;

[0021] The depth value of each pixel is determined based on the normal vector coordinates of each intermediate pixel and the depth value of the associated point, wherein the associated point of an intermediate pixel is a pixel whose positional relationship with the intermediate pixel conforms to a preset positional relationship.

[0022] Optionally, the extraction unit is specifically used for:

[0023] In one round of feature extraction, the following steps are performed:

[0024] Based on the feature extraction network, features are extracted from the two input data respectively to obtain the input features corresponding to the two input data;

[0025] Based on the feature fusion network, feature fusion is performed on the two input features respectively to obtain the initial fused features corresponding to the two input features;

[0026] Wherein, if the first round of feature extraction is the first round of feature extraction, then the two input data are the image to be detected and the normal vector map; otherwise, the two input data are the two initial fusion features output from the previous round of feature extraction.

[0027] Optionally, the feature extraction network includes an image feature extraction layer and a normal vector feature extraction layer, and the feature fusion layer includes an image feature fusion layer and a normal vector feature fusion layer;

[0028] The extraction unit is specifically used for:

[0029] Based on the image feature extraction layer, feature extraction is performed on the image input data to obtain image fusion features. The image input data is the image to be detected or the initial fusion features output by the image feature fusion layer in the previous round of feature extraction.

[0030] Based on the normal vector feature extraction layer, feature extraction is performed on the normal vector input data to obtain normal vector fusion features. The normal vector input data is the normal vector image or the initial fusion features output by the normal vector feature fusion layer in the previous round of feature extraction.

[0031] Optionally, the extraction unit is specifically used for:

[0032] Based on the image feature fusion layer, the image fusion feature and the normal vector fusion feature are fused to obtain the first fusion feature;

[0033] The first fusion feature is concatenated with the image fusion feature to obtain the initial fusion feature corresponding to the image fusion feature;

[0034] Based on the normal vector feature fusion layer, the image fusion feature and the normal vector fusion feature are fused to obtain a second fusion feature;

[0035] The second fusion feature is concatenated with the normal vector fusion feature to obtain the initial fusion feature corresponding to the normal vector fusion feature.

[0036] Optionally, the acquisition unit is specifically used to: after acquiring the image to be detected and before acquiring the normal vector image, obtain the normal vector coordinates of each pixel in the image to be detected in the following manner:

[0037] At least three polarized light sources selected from the candidate light source set are used to photograph the object to be detected to obtain at least three polarized light images, each of the candidate light sources in the candidate light source set corresponding to a different direction;

[0038] Based on the pixel values ​​of each pixel in the at least three polarized images, the normal vector coordinates corresponding to each pixel in the image to be detected are obtained.

[0039] Optionally, the acquisition unit is specifically used for:

[0040] The pixel values ​​of each pixel in the image to be detected are replaced with the corresponding normal vector coordinates, and the image is drawn to obtain the normal vector image.

[0041] Optionally, the extraction unit is specifically used for:

[0042] The image fusion features are subjected to feature mapping to obtain the image fusion vector;

[0043] The normal vector fusion features are subjected to feature mapping to obtain a first fusion vector and a second fusion vector;

[0044] Image feature fusion is performed on the image fusion vector, the first fusion vector, and the second fusion vector to obtain the first fusion feature.

[0045] Optionally, the extraction unit is specifically used for:

[0046] The image fusion features are subjected to feature mapping to obtain a third fusion vector and a fourth fusion vector;

[0047] Perform feature mapping on the fused normal vector features to obtain the fused normal vector;

[0048] The third fusion vector, the fourth fusion vector, and the normal vector fusion vector are fused using normal vector features to obtain the second fusion feature.

[0049] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described defect detection methods.

[0050] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described defect detection methods.

[0051] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described defect detection methods.

[0052] The beneficial effects of this application are as follows:

[0053] The defect detection method, apparatus, electronic device, and storage medium provided in this application first acquire the image to be detected corresponding to the object to be detected, and the normal vector image corresponding to the image to be detected; then, based on the image to be detected and the normal vector image, at least three rounds of feature extraction are performed, which can perform deep fusion of the image to be detected and the normal vector image; the two features to be fused obtained in the last round of feature extraction are concatenated, and the obtained target fusion feature has a strong feature representation capability; on this basis, defect detection is performed on the image to be detected based on the target fusion feature, which can obtain at least one candidate defect region, improve defect detection efficiency, and eliminate the subjective differences existing in manual detection, thereby improving defect detection accuracy.

[0054] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0055] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0056] Figure 1 This is a normal vector image in an embodiment of this application;

[0057] Figure 2 This is a schematic diagram of an insulating tape according to an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of an application scenario in an embodiment of this application;

[0059] Figure 4 This is a flowchart illustrating the implementation of a defect detection method in an embodiment of this application.

[0060] Figure 5A This is a schematic diagram of an image to be detected in an embodiment of this application;

[0061] Figure 5BThis is another normal vector image in the embodiments of this application;

[0062] Figure 6A This is a schematic diagram of a set of polarized images in an embodiment of this application;

[0063] Figure 6B This is a schematic diagram of another image to be detected in an embodiment of this application;

[0064] Figure 7 This is a schematic diagram of a feature extraction method in an embodiment of this application;

[0065] Figure 8 This is a schematic diagram of another feature extraction method in the embodiments of this application;

[0066] Figure 9 This is a schematic diagram of a feature fusion method in an embodiment of this application;

[0067] Figure 10 This is a schematic diagram of the structure of a cross-attention module in an embodiment of this application;

[0068] Figure 11 This is a schematic diagram of the structure of a deep cross-attention feature fusion detector according to an embodiment of this application;

[0069] Figure 12A This is a schematic diagram of a method for obtaining candidate defect regions in an embodiment of this application;

[0070] Figure 12B This is a schematic diagram of another method for obtaining candidate defect regions in an embodiment of this application;

[0071] Figure 13 This is a vector visualization diagram in an embodiment of this application;

[0072] Figure 14 This is a schematic diagram of a height field in an embodiment of this application;

[0073] Figure 15 This is a schematic diagram of an auxiliary diagram in an embodiment of this application;

[0074] Figure 16 This is a flowchart illustrating a pinhole detection method in an embodiment of this application;

[0075] Figure 17 This is a logical schematic diagram of a defect detection method in an embodiment of this application;

[0076] Figure 18 This is a schematic diagram of the structure of a defect detection device according to an embodiment of this application;

[0077] Figure 19This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application;

[0078] Figure 20 This is a schematic diagram of the hardware structure of another electronic device using an embodiment of this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0080] The following describes some of the concepts involved in the embodiments of this application.

[0081] Normal vector coordinates: These refer to the coordinates of the normal vector of each pixel in the image to be detected. The direction of the normal vector can represent the orientation of the point. Therefore, the orientation of the object point on the surface of the object to be detected corresponding to the pixel can be represented by the normal vector coordinates of the pixel.

[0082] Normal vector image: refers to an image drawn based on the normal vector coordinates of pixels, such as... Figure 1 As shown, this is a normal vector image in an embodiment of this application. When the object to be detected has a defect, the normal vector direction of the pixels in the defective region will be significantly different from the normal vector direction of the non-defective region. Therefore, after drawing the normal vector image based on the normal vector coordinates, the defective region can be reflected from the normal vector image, such as... Figure 1 The area within the circle in the image needs to be explained. Figure 1 The normal vector image in the image is actually a color image, but here we use the grayscale image corresponding to the color image to represent it.

[0083] Feature extraction: This includes two parts: feature extraction and feature fusion. The purpose of feature extraction is to extract features from the image to be detected and the normal vector image respectively, and then fuse the extracted features to perform deep feature fusion between the image to be detected and the normal vector image, so as to obtain fused features with stronger expressive power, thereby improving the defect detection accuracy when performing defect detection based on fused features.

[0084] Feature extraction: Extracting features from an image or existing features to obtain the desired features. In this embodiment, the objects of feature extraction include the image to be detected, the normal vector image, and the fused features output from each round of feature extraction.

[0085] Feature fusion: refers to the fusion of different features to obtain fused features, which can produce features with stronger expressive power. In the embodiments of this application, feature extraction and feature fusion are combined to achieve deep fusion of different features and enhance feature expressive power.

[0086] Photometric stereo technology: a technique for reconstructing surface normal vectors by observing objects under different lighting conditions. In this application embodiment, photometric stereo technology is used to obtain the normal vector coordinates of each pixel in the image to be detected.

[0087] Object detection: This is used to determine the location and category of objects in an input image. Input images often contain many objects, and object detection aims to identify the objects within the input image and determine their locations. Object detection is a core task in computer vision. In this embodiment, the detection target for defect detection is the defect region; defect detection is also a type of object detection.

[0088] The embodiments of this application relate to artificial intelligence (AI), natural language processing (NLP), and machine learning (ML) technologies, and are designed based on computer vision technology and machine learning in artificial intelligence.

[0089] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.

[0090] Artificial intelligence (AI) studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology mainly includes computer vision, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation. With the research and advancement of AI technology, it is being researched and applied in multiple fields, such as smart homes, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, autonomous driving, robotics, and smart healthcare. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role.

[0091] Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0092] Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. The text separation model in this application embodiment is trained using machine learning or deep learning techniques. Based on the training method of the text separation model in this application embodiment, the recognition accuracy of printed text can be improved.

[0093] Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding those of previous related technologies.

[0094] The design concept of the embodiments of this application is briefly introduced below:

[0095] In modern industrial manufacturing, it is inevitable that some defective parts will appear among the various components produced by enterprises. On the one hand, in order to ensure product quality, enterprises must detect defective parts for subsequent processing; on the other hand, finding defective parts and analyzing the characteristics and proportions of various defects is crucial for improving production processes and increasing production line yield.

[0096] In related technologies, companies often employ workers to manually inspect products for defects. However, this method has several drawbacks. First, some defects are very small and difficult to observe, requiring a significant amount of time to inspect each one, resulting in low efficiency. Second, manual inspection relies solely on visual judgment, and the subjective differences among workers can lead to false positives and false negatives, resulting in large errors and low accuracy.

[0097] For example, such as Figure 2 The diagram shown is a schematic of an insulating tape according to an embodiment of this application. During the quality inspection of insulating tape, a defect called a "pinhole" exists. Although its area is extremely small, its depth is very deep, posing a significant safety hazard and making it a defect that cannot be overlooked during quality inspection. In traditional quality inspection processes, companies often rely on manual observation to determine the presence of pinholes on the insulating tape. This process presents challenges for quality inspectors, including high workload and repetitive tasks, leading to high employee turnover. For companies, it results in high testing costs (personnel costs) and low quality inspection efficiency.

[0098] Furthermore, "pinholes" and "small pits" are very similar in appearance, differing only in depth. Deeper pinholes can damage the internal structure of the insulating tape, posing a safety hazard; therefore, insulating tape with pinholes must be discarded. Small pits, on the other hand, are shallow and, if small in area and few in number, do not require treatment. Manual quality inspection based solely on visual inspection is very difficult and subject to significant subjective differences between workers, resulting in substantial inspection errors.

[0099] Therefore, improving the efficiency and accuracy of defect detection has become an urgent problem to be solved.

[0100] In view of this, embodiments of this application provide a defect detection method, apparatus, electronic device, and storage medium. First, a detection image corresponding to the object to be detected and a normal vector image corresponding to the detection image are acquired. Then, at least three rounds of feature extraction are performed based on the detection image and the normal vector image, enabling deep fusion of the detection image and the normal vector image. The two features to be fused obtained in the last round of feature extraction are concatenated, and the resulting target fused feature has a strong feature representation capability. Based on this, defect detection is performed on the detection image based on the target fused feature, thereby obtaining at least one candidate defect region, improving defect detection efficiency, eliminating subjective differences in manual detection, and improving defect detection accuracy.

[0101] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0102] like Figure 3 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes two terminal devices 310 and one server 320.

[0103] In this embodiment, the terminal device 310 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have a client installed related to defect detection. This client can be software (e.g., a browser, detection software), a webpage, or a mini-program. The server 320 is the backend server corresponding to the software, webpage, or mini-program, or a server specifically used for defect detection; this application does not impose specific limitations. The server 320 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0104] It should be noted that the defect detection method in this application embodiment can be executed by an electronic device, which can be a server 320 or a terminal device 310. That is, the method can be executed by the server 320 or the terminal device 310 alone, or by both the server 320 and the terminal device 310. For example, when executed by both the server 320 and the terminal device 310, the terminal device 310 acquires the image to be detected corresponding to the object to be detected and the normal vector image corresponding to the image to be detected. Then, it sends the image to be detected and the normal vector image to the server 320. The server 320 first performs at least three rounds of feature extraction based on the image to be detected and the normal vector image, and concatenates the two features to be fused obtained in the last round of feature extraction to obtain the target fusion feature. Then, it performs defect detection on the image to be detected based on the target fusion feature to obtain at least one candidate defect region. The server 320 sends the candidate defect region to the terminal device 310, and the terminal device 310 displays the location information of the candidate defect region to the user.

[0105] In one alternative implementation, the terminal device 310 and the server 320 can communicate via a communication network.

[0106] In one alternative implementation, the communication network is a wired network or a wireless network.

[0107] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0108] In this embodiment of the application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as disclosed in the defect detection method of this embodiment, the image to be detected, the normal vector image, and the network parameters of the feature extraction network and the feature fusion network can all be stored on the blockchain.

[0109] Furthermore, the embodiments of this application can be applied to various scenarios, including not only defect detection scenarios, but also scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. For example, when the embodiments of this application are applied to artificial intelligence scenarios, they can be used to detect whether there are pinholes on insulating tape, thereby improving the efficiency of pinhole detection.

[0110] The following describes the defect detection method provided by the exemplary embodiments of this application in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0111] In this embodiment, the example mainly uses insulating tape as the object to be tested and the candidate defect area as the area containing pinholes. It should be noted that the defect detection method in this application is not only applicable to the detection of pinholes in insulating tape, but also applicable to other defects in industrial defect detection, such as detecting whether there are pits, holes and dirt on the surface of industrial products. This application does not make specific limitations here.

[0112] See Figure 4 The diagram shown is a flowchart of an implementation of a defect detection method provided in this application. Taking a server as the executing entity, the specific implementation process of this method includes the following steps S41-S43:

[0113] S41: The server obtains the image to be detected corresponding to the object to be detected, and the normal vector image corresponding to the image to be detected;

[0114] To perform defect detection on the object to be inspected, the first step is to acquire an image of the object to be inspected by taking a full-light photograph of it, such as... Figure 5AThe diagram shown is a schematic of an image to be inspected in an embodiment of this application. It can be seen that some dirty areas are highly visible in the image to be inspected, while industrial products exhibit a variety of defects, such as deep pinholes, dents, and shallow dirt. Deep pinholes and dents are not highly visible in the image to be inspected. Therefore, a normal vector image corresponding to the image to be inspected can also be obtained. The pixel value of each pixel in the normal vector image is the normal vector coordinate of the corresponding pixel in the image to be inspected. The normal vector coordinate is used to characterize the orientation of the object point on the surface of the object to be inspected corresponding to the pixel. For object points in areas with pinholes or dents, their orientation will be significantly different from other object points, and the normal vector coordinates of the corresponding pixel will also be different from the normal vector coordinates of other pixels. The red, green, and blue color channel values ​​(RGB values) of each pixel in the obtained normal vector image represent the surface normal vector direction (x, y, z coordinates) of that pixel location.

[0115] like Figure 5B As shown, Figure 5A The normal vector image corresponding to the image to be detected in the image is compared with the normal vector image in the image to be detected. Figure 5A and Figure 5B It is known that the normal vector image is far superior to the image to be detected for defects caused by depth, such as pinholes and dents. However, for another type of defect without depth, the image to be detected is superior to the normal vector image. In this problem, it is impossible to determine all defects using only the original image (the image to be detected) or the normal vector image. Existing technical solutions suffer from high computational cost and poor performance due to a lack of targeted design. Therefore, this application proposes to acquire both the image to be detected and the normal vector image, and to perform defect detection based on both images, thereby improving the accuracy and efficiency of defect detection.

[0116] After acquiring the image of the object to be detected, in one optional implementation, the normal vector coordinates of each pixel in the image to be detected are obtained in the following way:

[0117] First, at least three polarized light sources selected from the candidate light source set are used to capture images of the object to be detected, obtaining at least three polarized images. Then, based on the pixel values ​​of each pixel in the at least three polarized images, the normal vector coordinates of each pixel in the image to be detected are obtained.

[0118] Specifically, each candidate light source in the candidate light source set corresponds to a different direction, therefore the at least three selected polarizing light sources are also in different directions. When photographing the object to be detected, the camera and the object remain stationary, and the selected polarizing light sources are used to illuminate the image to be detected, obtaining polarized images. For example, the candidate light source set includes light source 1, light source 2, light source 3, and light source 4, where the emission direction of light source 1 is in front of the object to be detected, the emission direction of light source 2 is behind the object to be detected, the emission direction of light source 3 is to the left of the object to be detected, and the emission direction of light source 4 is to the right of the object to be detected. Light sources 1-3 are selected to photograph the insulating tape, obtaining polarized image 1 corresponding to light source 1, polarized image 2 corresponding to light source 2, and polarized image 3 corresponding to light source 3. Each pixel in the polarized image corresponds one-to-one with each pixel in the image to be detected. Based on the pixel values ​​of each pixel in at least three polarized images, the normal vector coordinates corresponding to each pixel in the image to be detected can be obtained.

[0119] Based on the above method, the normal vector coordinates of each pixel in the image to be detected are obtained. Then, a normal vector image can be drawn based on the normal vector coordinates, which improves the visibility of defects in the object to be detected and improves the accuracy of defect detection.

[0120] Optionally, during the image capture process, only one light source can be used. By adjusting the relative position of the light source and the object to be inspected, the effect of illuminating the object from different directions can be achieved. For example, by selecting light source 1 to illuminate the front, back, and left of the insulating tape, the same effect as that achieved by selecting light sources 1-3 can be obtained.

[0121] like Figure 6A As shown, this is a schematic diagram of a set of polarized images in an embodiment of this application, including four polarized images (1)-(4) obtained by taking polarized photos of insulating tape at four angles, as shown. Figure 6B The image shown is a schematic diagram of another image to be detected in an embodiment of this application, which is an image to be detected obtained by taking a full-light photograph of insulating tape.

[0122] In one optional implementation, after obtaining the normal vector coordinates of each pixel in the image to be detected, the pixel values ​​are replaced with the corresponding normal vector coordinates, and the image is drawn to obtain a normal vector image.

[0123] Specifically, the normal vector coordinates of the pixels are used as RGB values ​​for image drawing. For example, if the image to be detected contains pixels 1, 2, 3, and 4, and the normal vector coordinates of pixel 1 are (64, 255, 0), the normal vector coordinates of pixel 2 are (64, 255, 0), the normal vector coordinates of pixel 3 are (0, 64, 255), and the normal vector coordinates of pixel 4 are (64, 255, 0), then the normal vector image also contains 4 pixels. The RGB value of pixel 11 is (64, 255, 0), the RGB value of pixel 22 is (64, 255, 0), the RGB value of pixel 33 is (0, 64, 255), and the RGB value of pixel 4 is (64, 255, 0).

[0124] S42: The server performs at least three rounds of feature extraction based on the image to be detected and the normal vector image, and concatenates the two features to be fused obtained in the last round of feature extraction to obtain the target fusion feature;

[0125] Specifically, after obtaining the image to be detected and the normal vector image, at least three rounds of feature extraction are performed. In one round of feature extraction, features can be extracted from the image to be detected and the normal vector image separately. For example, features can be extracted from the image of insulating tape and the corresponding normal vector image separately to obtain two features. The two features obtained in the last round of feature extraction are then used as two features to be fused. Alternatively, in one round of feature extraction, features can be extracted from the image to be detected and the normal vector image separately, and then the extracted features can be fused to obtain two fused features. The two fused features obtained in the last round of feature extraction are then used as two features to be fused. In addition, any feature extraction method can be applied to the embodiments of this application, and no specific limitation is made here.

[0126] The two features to be fused obtained in the last round are concatenated. The concatenation method includes, but is not limited to, splicing, addition, multiplication, etc. The target fused feature obtained after concatenation has a strong feature representation ability.

[0127] In one alternative implementation, during a round of feature extraction, the following steps S421-S422 are performed:

[0128] S421: Based on the feature extraction network, feature extraction is performed on the two input data respectively to obtain the input features corresponding to each of the two input data;

[0129] S422: Based on the feature fusion network, feature fusion is performed on the two input features to obtain the initial fused features corresponding to each of the two input features.

[0130] Specifically, if the above-mentioned feature extraction is the first round of feature extraction, then the two input data are the image to be detected and the normal vector map; otherwise, the two input data are the two initial fused features output from the previous round of feature extraction. Based on the above, it can be seen that the feature extraction in this application includes two steps: feature extraction and feature fusion. Figure 7 The diagram illustrates a feature extraction method according to an embodiment of this application. In the first round of feature extraction, taking the image to be detected as insulating tape image 1 and the normal vector image as the normal vector image 1 corresponding to insulating tape image 1 as an example, the feature extraction network extracts features from the insulating tape image 1 to obtain input feature 1. The feature extraction network then extracts features from the normal vector image 1 to obtain input feature 2. Next, the feature fusion network fuses input feature 1 and input feature 2 to obtain the initial fused feature 1 corresponding to input feature 1. Finally, the feature fusion network fuses input feature 2 and input feature 1 to obtain the initial fused feature 2 corresponding to input feature 2. The feature extraction process in other rounds is the same as above and will not be described in detail here.

[0131] Based on the above approach, in one round of feature extraction, features are extracted from the input data, and the obtained features are fused. This can fully combine the expressive power of each feature of the input data to obtain a more expressive fused feature, and then perform defect detection based on the fused feature, thereby improving the efficiency of defect detection.

[0132] The feature extraction network includes an image feature extraction layer and a normal vector feature extraction layer, and the feature fusion layer includes an image feature fusion layer and a normal vector feature fusion layer. In an optional implementation, step S421 can be implemented as follows:

[0133] S4211: Based on the image feature extraction layer, feature extraction is performed on the image input data to obtain image fusion features;

[0134] S4212: Based on the normal vector feature extraction layer, feature extraction is performed on the normal vector input data to obtain normal vector fusion features.

[0135] Specifically, in this application, the image feature extraction layer in the feature extraction network is used to extract features from the image input data, where the image input data is the image to be detected or the initial fusion features output by the image feature fusion layer in the previous round of feature extraction. The normal vector feature extraction layer is used to extract features from the normal vector input data, where the normal vector input data is the normal vector image or the initial fusion features output by the normal vector feature fusion layer in the previous round of feature extraction. Figure 8The diagram shown is a schematic of another feature extraction method in this application embodiment. In the second round of feature extraction, taking the image input data as the initial fusion feature 1 listed above and the normal vector input data as the initial fusion feature 2 listed above as an example, the initial fusion feature 1 is input into the image feature extraction layer for feature extraction to obtain image fusion feature 1, and the initial fusion feature 2 is input into the normal vector feature extraction layer for feature extraction to obtain normal vector fusion feature 1.

[0136] After obtaining the image fusion features and normal vector fusion features based on steps S4211 and S4212, in an optional implementation, step S422 can be implemented as follows:

[0137] S4221: Based on the image feature fusion layer, image feature fusion features and normal vector fusion features are fused to obtain the first fused feature;

[0138] S4222: Concatenate the first fusion feature with the image fusion feature to obtain the initial fusion feature corresponding to the image fusion feature;

[0139] S4223: Based on the normal vector feature fusion layer, the image fusion feature and the normal vector fusion feature are fused to obtain the second fusion feature;

[0140] S4224: Concatenate the second fusion feature with the normal vector fusion feature to obtain the initial fusion feature corresponding to the normal vector fusion feature.

[0141] Specifically, steps S4221-S4224 are used to perform feature fusion on the image fusion features and the normal vector fusion features, such as... Figure 9 The diagram shown is a schematic of a feature fusion method in an embodiment of this application. Taking image fusion feature 1 and normal vector fusion feature 1 as an example, the image feature fusion layer performs image feature fusion on image fusion feature 1 and normal vector fusion feature 1 to obtain a first fusion feature 1. The image fusion feature 1 and the first fusion feature 1 are concatenated to obtain an initial fusion feature 1. The normal vector feature fusion layer performs normal vector feature fusion on image fusion feature 1 and normal vector fusion feature 1 to obtain a second fusion feature 1. The normal vector fusion feature 1 and the second fusion feature 1 are concatenated to obtain an initial fusion feature 2.

[0142] It should be noted that there is no specific order of steps S4221-S4222 and S4223-S4224. Steps S4221-S4222 can be executed first, or steps S4223-S4224 can be executed first, or steps S4221-S4222 and steps S4223-S4224 can be executed simultaneously. This application does not make any specific restrictions here.

[0143] Based on the above method, deep feature fusion is performed on the image fusion features and normal vector fusion features output by feature extraction. The resulting initial fusion features fully combine the features of the image to be detected and the normal vector image, enhancing the feature representation capability. Defect detection is then performed based on the initial fusion features output by the last round of feature extraction, improving the efficiency of defect detection.

[0144] In an optional implementation, step S4221 can be performed as follows:

[0145] First, feature mapping is performed on the image fusion features to obtain the image fusion vector; then, feature mapping is performed on the normal vector fusion features to obtain the first fusion vector and the second fusion vector; finally, image feature fusion is performed on the image fusion vector, the first fusion vector and the second fusion vector to obtain the first fusion feature.

[0146] Specifically, a convolutional layer is used to map the image fusion features into an image fusion vector, and two convolutional layers are used to map the normal vector fusion features into a first fusion vector and a second fusion vector. Then, cross-attention is calculated on these three vectors to obtain the second fusion feature. Additionally, the calculated attention feature (i.e., the second fusion feature) can be fed into a fully connected layer to further increase its non-linearity and feature representation ability.

[0147] Based on the above approach, different data features can be deeply integrated, greatly enhancing the feature representation capability.

[0148] In an optional implementation, step S4223 can be performed as follows:

[0149] First, feature mapping is performed on the image fusion features to obtain the third fusion vector and the fourth fusion vector; then, feature mapping is performed on the normal vector fusion features to obtain the normal vector fusion vector; finally, normal vector feature fusion is performed on the third fusion vector, the fourth fusion vector, and the normal vector fusion vector to obtain the second fusion feature.

[0150] Specifically, two convolutional layers are used to map the image fusion features into a third and fourth fusion vector, and one convolutional layer is used to map the normal vector fusion features into a normal vector fusion vector. Then, cross-attention is calculated on these three vectors to obtain the first fusion feature. Additionally, the calculated attention feature (i.e., the first fusion feature) can be fed into a fully connected layer to further increase its non-linearity and feature representation ability.

[0151] like Figure 10 The diagram shown is a schematic of a cross-attention module in an embodiment of this application, used for feature fusion of image fusion features and normal vector fusion features. In the cross-attention module, a convolutional layer is used to map feature A into a feature vector. Figure 10In the Q layer, the two convolutional layers map feature B into two feature vectors (Q). Figure 10 The process involves first calculating the cross-attention values ​​(Q, K, and V), then feeding the calculated attention features into a fully connected layer to further enhance their nonlinearity and feature representation capabilities. Finally, the output feature C of the fully connected layer is concatenated with feature A to obtain the fused feature. The correspondence between features A and B, and image fusion features and normal vector fusion features, is as follows: for the image feature fusion layer, feature A is the image fusion feature, and feature B is the normal vector fusion feature; for the normal vector map feature fusion layer, feature A is the normal vector fusion feature, and feature B is the image fusion feature. The cross-attention calculation for Q, K, and V can be performed based on the following formula:

[0152]

[0153] Where, d K This represents the dimension of K.

[0154] S43: The server performs defect detection on the image to be detected based on the target fusion features, and obtains at least one candidate defect region.

[0155] Specifically, the target fusion features are fed into the detection head for defect classification and localization to obtain candidate defect regions.

[0156] like Figure 11 The diagram shown is a schematic of a deep cross-attention feature fusion detector according to an embodiment of this application. The image to be detected and the normal vector image are input into two convolutional networks (each containing three convolutional layers) for feature extraction. The inter-layer connections between the two convolutional networks employ the following... Figure 10 The cross-attention module shown performs deep feature fusion to obtain features with stronger expressive power. The two fused features output by the last cross-attention module are concatenated to obtain the target fused feature. Then, the detection head performs defect detection on the target fused feature to obtain candidate defect regions.

[0157] It should be noted that the number of feature extractor layers in the deep cross-attention feature fusion detector can be increased from 3 layers to more, and other types of fusion mechanisms can also be designed for the cross-attention module. The detection head can be any object detection model, and no specific restrictions are imposed here.

[0158] The deep cross-attention feature fusion detector in this embodiment can deeply fuse different data features, greatly enhancing feature representation capabilities. It has the advantages of high performance (i.e., high detection accuracy) and low computational cost.

[0159] In this embodiment, the image to be detected and the normal vector image corresponding to the object to be detected are first obtained. Then, at least three rounds of feature extraction are performed based on the image to be detected and the normal vector image, which enables deep fusion of the image to be detected and the normal vector image. The two features to be fused obtained in the last round of feature extraction are concatenated, and the resulting target fusion feature has a strong feature representation capability. Based on this, defect detection is performed on the image to be detected based on the target fusion feature, which can obtain at least one candidate defect region, improve defect detection efficiency, eliminate the subjective differences in manual detection, and improve defect detection accuracy.

[0160] In one alternative implementation, after acquiring the image to be detected and the normal vector image, as follows: Figure 12A The diagram illustrates a method for obtaining candidate defect regions in an embodiment of this application. The image to be detected and the normal vector image are input into two feature extraction networks, respectively. The features output by the two feature extraction networks are then input into two detection heads. The detection results from the two detection heads are fused to obtain the candidate defect regions. Based on this method, there is no interaction between the two detection heads, and each detector can only see one data feature. The feature representation capability is very weak, resulting in very poor detection performance. The final detection accuracy is 7.1 percentage points lower than steps S41-S43 in this application. Furthermore, this technical solution actually uses two detection heads, resulting in huge computational resource overhead and severely limited processing speed.

[0161] In one alternative implementation, after acquiring the image to be detected and the normal vector image, as follows: Figure 12B The diagram illustrates another method for obtaining candidate defect regions in this application. Feature extraction is performed on the image to be detected and the normal vector image, respectively. The extracted features are concatenated and then fed into the detection head for defect detection to obtain candidate defect regions. Based on this method, obtaining candidate defect regions simply involves concatenating the two features without fusing them. In the defect detection scenario of this application, the two types of data are complementary; that is, for most defects, the normal vector image is more visible than the image to be detected, but for a small number of defects, they are only visible in the image to be detected and not in the normal vector image. Simple concatenation only ensures that the detector obtains both features simultaneously, but it cannot effectively combine their characteristics to maximize their effectiveness. The final detection accuracy is 5.2 percentage points lower than the scheme in steps S41-S43 of this application.

[0162] Compared to existing technologies, defect detection using steps S41-S43 is significantly more accurate. However, for challenging defects such as tiny pinholes, false alarms are common; for example, a small dent might be mistaken for a pinhole, even though a pinhole is much deeper. Therefore, based on this characteristic, this application uses vector visualization to visualize the normal vector map of suspected "pinhole" areas, such as... Figure 13 As shown, this is a vector visualization diagram in an embodiment of this application. It can be seen that it has strong continuity. Therefore, the depth value of each pixel in the candidate defect area can be calculated based on the following method. The target defect area can be determined from the candidate defect area based on the depth value, thereby further improving the defect detection accuracy.

[0163] In one optional implementation, the following steps S24-S25 are performed for each candidate defect region:

[0164] S24: Determine the depth value of each pixel based on its normal vector coordinates within a candidate defect region;

[0165] S25: If there is a depth value among the depth values ​​that is greater than the preset depth threshold, then a candidate defect region will be selected as the target defect region.

[0166] Specifically, different target defect areas can be filtered out by adjusting the preset depth threshold. For example, if you want to filter out defects such as pinholes, holes, and deep pits, you can set the preset depth threshold to be larger. If you also want to filter out small pits and shallow defects, you can set the preset depth threshold to be smaller. Taking an image to be detected containing three candidate defect areas as an example, with a preset depth threshold of 0.01 mm, if none of the pixels in candidate defect area 1 have a depth value greater than 0.01 mm, one pixel in candidate defect area 2 has a depth value greater than 0.01 mm, and ten pixels in candidate defect area 3 have a depth value greater than 0.01 mm, then both candidate defect areas 2 and 3 will be considered as target defect areas.

[0167] Defect detection based on the above methods can replace 80%-90% of quality inspection manpower, greatly reducing costs and improving efficiency for enterprises. Using the defect detection method in this application for pinhole detection can eliminate the subjective differences present in traditional manual quality inspection, greatly improving the accuracy of pinhole detection.

[0168] In an alternative implementation, step S24 can be performed as follows:

[0169] Step S241: Based on the position of each pixel within a candidate defect region, divide each pixel to obtain each boundary pixel and each intermediate pixel;

[0170] Step S242: Set the depth value of each boundary pixel to the corresponding preset value;

[0171] Step S243: Determine the depth value of each pixel based on its respective normal vector coordinates and the depth value of the associated point.

[0172] Specifically, the associated point of an intermediate pixel is a pixel whose positional relationship with the intermediate pixel conforms to a preset positional relationship. For example, the preset positional relationship could be the nearest pixel to the upper left of the intermediate pixel, the nearest pixel to the lower right of the intermediate pixel, or the nearest pixel to the upper right of the intermediate pixel; no specific limitation is made here. The depth value of the intermediate pixel is obtained based on the depth value of the associated point. For each pixel in a candidate defect region, there will inevitably be some points without associated points, i.e., boundary pixels. The depth value of the boundary pixels is set to the corresponding preset value. The depth values ​​of the boundary pixels can be set to the same preset value, or different preset values ​​can be set for different boundary pixels; no specific limitation is made here.

[0173] For the middle pixel in the candidate defect region, the surface depth value of each pixel can be calculated using the following formula:

[0174]

[0175] Where f(x+1, y+1) is the depth value of the middle pixel, f(x, y) is the depth value of the associated point of this middle pixel, and the normal vector coordinates of this middle pixel are (n x n y n z ).

[0176] Based on the depth value calculated using the above method, and combined with the theoretical formula derived in this application, a fast and accurate depth calculation can be performed using the normal vector. By combining the candidate defect region location and depth calculation results in this application, the presence of pinholes in the insulating tape can be accurately and quantitatively determined, and quality inspection results can be output. This method features low computational complexity, short processing time, and high accuracy in depth calculation.

[0177] Alternatively, the depth value of each pixel can be calculated using the global integral (Frankot-Chellappa) algorithm for 3D reconstruction.

[0178] Optionally, deep learning methods can be used to classify and determine candidate defect regions, and a large number of pinhole sample images of different depth levels (500 each) can be collected for model training.

[0179] In this embodiment of the application, the above formula for calculating the depth value of pixels within the candidate region is derived through the following theoretical analysis:

[0180] First, the height field is represented as z = f(x, y), as follows: Figure 14 As shown, based on the principle of path integral, we obtain:

[0181]

[0182] That is, the height difference between two points is equal to the path integral of the directional derivative.

[0183] Since the integral of the above equation is independent of the path, taking a special path such as (a, b) → (x, b) → (x, y), we have:

[0184]

[0185] according to Figure 15 The auxiliary point (s, y) in the equation has the following relationship:

[0186]

[0187] Combining formulas (1, (2), and (3), we can obtain:

[0188]

[0189] Then, based on the gradient and normal vector (n) x n y n z The relationship between )

[0190]

[0191] We can obtain:

[0192]

[0193] Transform the variable-upper-level integral into an indefinite integral:

[0194]

[0195] That is:

[0196]

[0197] By discretizing the pixels on the image, we can obtain:

[0198]

[0199] like Figure 16 The diagram shown is a flowchart of a pinhole detection method according to an embodiment of this application, including the following steps:

[0200] S1601: Take polarized photos of insulating tape under different light sources to obtain polarized images under different light sources, and take full-light photos of insulating tape to obtain the original image;

[0201] S1602: Synthesize the normal vector image based on the polarized light images under light sources from different directions;

[0202] S1603: Input the original image and normal vector image into the deep cross-attention feature fusion detector to detect defects and obtain candidate defect regions;

[0203] S1604: Determine whether the defect type of the candidate defect area is a pinhole. If yes, proceed to step S1605; otherwise, proceed to step S1606.

[0204] S1605: Calculate the depth value of each pixel within the candidate defect region;

[0205] S1606: Output the defect type and location of the candidate defect region;

[0206] S1607: Determine whether there is a pixel whose depth value is greater than a preset depth threshold. If yes, proceed to step S1608; otherwise, proceed to step S1609.

[0207] S1608: The candidate defect area is determined to be a pinhole;

[0208] S1609: The candidate defect area is determined to be non-pinhole.

[0209] Based on the above method, images of the target under different light sources are first acquired. Then, based on the image data acquired under different light sources, the normal vector map of each point on the target surface is calculated. Next, the normal vector map and the original image are fed into a feature fusion and detector for defect inspection. The normal vectors of the suspected pinhole local areas are integrated to calculate the depth. Finally, the calculated depth is used to determine whether the defect is a pinhole. Users can accurately determine in real time whether there are pinhole defects in insulating tape.

[0210] like Figure 17The diagram shown is a logical schematic of a defect detection method in an embodiment of this application. Taking the object to be detected as a waterproof material and the candidate defect area as the area containing pinholes as an example, firstly, the waterproof material is photographed under full light to obtain the image to be detected. Then, the waterproof material is photographed under polarized light from four angles to obtain four polarized images. Then, a normal vector image is synthesized based on the four polarized images using photometric stereo technology. Subsequently, the image to be detected and the normal vector image are subjected to the first feature extraction to obtain image fusion feature 1 and normal vector fusion feature 2. Then, the image fusion feature 1 and the normal vector fusion image 2 are subjected to the first feature fusion to obtain initial fusion feature 1 and initial fusion feature 2. The second feature extraction is performed on feature 2 to obtain image fusion feature 2 and normal vector fusion feature 2. Then, the second feature fusion is performed on image fusion feature 2 and normal vector fused image 2 to obtain initial fusion feature 11 and initial fusion feature 22. The third feature extraction is performed on initial fusion feature 11 and initial fusion feature 22 to obtain image fusion feature 3 and normal vector fusion feature 3. Then, the third feature fusion is performed on image fusion feature 3 and normal vector fused image 3 to obtain initial fusion feature 111 and initial fusion feature 222. Finally, the initial fusion feature 111 and initial fusion feature 222 are concatenated to obtain the target fusion feature. Based on the target fusion feature, defect detection is performed to obtain candidate defect regions.

[0211] This application proposes a multi-source, cross-modal defect detection algorithm centered on a cross-attention module. It deeply fuses features from different, complementary data sources to enhance feature representation and achieve superior detection accuracy. Furthermore, after obtaining preliminary detection results, for defects prone to false positives, such as pinholes, this application also designs a dedicated post-processing mechanism to perform depth calculations on suspected pinhole regions, significantly reducing the model's false positive rate without sacrificing accuracy.

[0212] Based on the same inventive concept, embodiments of this application also provide a defect detection device. For example... Figure 18 As shown, this is a structural schematic diagram of the defect detection device 1800, which may include:

[0213] The acquisition unit 1801 is used to acquire the image to be detected corresponding to the object to be detected and the normal vector image corresponding to the image to be detected. The image to be detected is obtained by taking a full-light image of the object to be detected. The pixel value of each pixel in the normal vector image is the normal vector coordinate of the corresponding pixel in the image to be detected. The normal vector coordinate is used to represent the orientation of the object point on the surface of the object to be detected corresponding to the pixel.

[0214] The extraction unit 1802 is used to perform at least three rounds of feature extraction based on the image to be detected and the normal vector image, and to concatenate the two features to be fused obtained in the last round of feature extraction to obtain the target fusion feature;

[0215] The detection unit 1803 is used to perform defect detection on the image to be detected based on the target fusion features, and obtain at least one candidate defect region.

[0216] In this embodiment, the image to be detected and the normal vector image corresponding to the object to be detected are first obtained. Then, at least three rounds of feature extraction are performed based on the image to be detected and the normal vector image, which enables deep fusion of the image to be detected and the normal vector image. The two features to be fused obtained in the last round of feature extraction are concatenated, and the resulting target fusion feature has a strong feature representation capability. Based on this, defect detection is performed on the image to be detected based on the target fusion feature, which can obtain at least one candidate defect region, improve defect detection efficiency, eliminate the subjective differences in manual detection, and improve defect detection accuracy.

[0217] Optionally, the device further includes a determining unit 1804, for:

[0218] For each candidate defect region, perform the following operations:

[0219] Based on the normal vector coordinates of each pixel within a candidate defect region, determine the depth value of each pixel.

[0220] If any depth value is greater than the preset depth threshold, then a candidate defect region will be selected as the target defect region.

[0221] Optionally, the determining unit 1804 is specifically used for:

[0222] Based on the position of each pixel within a candidate defect region, each pixel is divided to obtain each boundary pixel and each intermediate pixel.

[0223] Set the depth value of each boundary pixel to its corresponding preset value;

[0224] The depth value of each pixel is determined based on the normal vector coordinates of each intermediate pixel and the depth value of the associated point. The associated point of an intermediate pixel is a pixel whose positional relationship with an intermediate pixel conforms to a preset positional relationship.

[0225] Optionally, extraction unit 1802 is specifically used for:

[0226] In one round of feature extraction, the following steps are performed:

[0227] Based on the feature extraction network, features are extracted from the two input data respectively to obtain the input features corresponding to each of the two input data;

[0228] Based on the feature fusion network, the two input features are fused separately to obtain the initial fused features corresponding to each of the two input features;

[0229] If a feature extraction round is the first round of feature extraction, then the two input data are the image to be detected and the normal vector map; otherwise, the two input data are the two initial fused features output from the previous round of feature extraction.

[0230] Optionally, the feature extraction network includes an image feature extraction layer and a normal vector feature extraction layer, and the feature fusion layer includes an image feature fusion layer and a normal vector feature fusion layer;

[0231] Extraction unit 1802 is specifically used for:

[0232] Based on the image feature extraction layer, features are extracted from the image input data to obtain image fusion features. The image input data is the image to be detected or the initial fusion features output by the image feature fusion layer in the previous round of feature extraction.

[0233] The normal vector feature extraction layer extracts features from the input normal vector data to obtain normal vector fusion features. The input normal vector data is the normal vector image or the initial fusion features output by the normal vector feature fusion layer in the previous round of feature extraction.

[0234] Optionally, extraction unit 1802 is specifically used for:

[0235] Image feature fusion is performed on image fusion features and normal vector fusion features based on the image feature fusion layer to obtain the first fusion feature;

[0236] The first fusion feature is concatenated with the image fusion feature to obtain the initial fusion feature corresponding to the image fusion feature;

[0237] Based on the normal vector feature fusion layer, the image fusion feature and the normal vector fusion feature are fused to obtain the second fusion feature;

[0238] The second fusion feature is concatenated with the normal vector fusion feature to obtain the initial fusion feature corresponding to the normal vector fusion feature.

[0239] Optionally, the acquisition unit 1801 is specifically used to: after acquiring the image to be detected and before acquiring the normal vector image, obtain the normal vector coordinates of each pixel in the image to be detected in the following manner:

[0240] At least three polarized light sources selected from the candidate light source set are used to capture images of the object to be detected, and at least three polarized light images are obtained. Each candidate light source in the candidate light source set corresponds to a different direction.

[0241] Based on the pixel values ​​of each pixel in at least three polarized images, the normal vector coordinates of each pixel in the image to be detected are obtained.

[0242] Optionally, the acquisition unit 1801 is specifically used for:

[0243] The pixel values ​​of each pixel in the image to be detected are replaced with the corresponding normal vector coordinates, and the image is drawn to obtain the normal vector image.

[0244] Optionally, the extraction unit 1802 is specifically used for:

[0245] Feature mapping is performed on the image fusion features to obtain the image fusion vector;

[0246] Perform feature mapping on the fused features of the normal vectors to obtain the first fused vector and the second fused vector;

[0247] Image feature fusion is performed on the image fusion vector, the first fusion vector, and the second fusion vector to obtain the first fusion feature.

[0248] Optionally, the extraction unit 1802 is specifically used for:

[0249] The image fusion features are feature-mapped to obtain the third and fourth fusion vectors;

[0250] Perform feature mapping on the normal vector fusion features to obtain the normal vector fusion vector;

[0251] The third fusion vector, the fourth fusion vector, and the normal vector fusion vector are fused using normal vector features to obtain the second fusion feature.

[0252] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0253] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0254] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 3 The server 320 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 19 As shown, it includes a memory 1901, a communication module 1903, and one or more processors 1902.

[0255] The memory 1901 is used to store computer programs executed by the processor 1902. The memory 1901 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0256] Memory 1901 may be volatile memory, such as random-access memory (RAM); memory 1901 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1901 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1901 may be a combination of the above-described memories.

[0257] Processor 1902 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1902 is used to implement the above-mentioned defect detection method when calling the computer program stored in memory 1901.

[0258] The communication module 1903 is used to communicate with terminal devices and other servers.

[0259] This application embodiment does not limit the specific connection medium between the memory 1901, communication module 1903, and processor 1902. This application embodiment... Figure 19 The memory 1901 and the processor 1902 are connected via a bus 1904, which is in the middle of the memory. Figure 19 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1904 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 19It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0260] The memory 1901 stores a computer storage medium, which in turn stores computer-executable instructions for implementing the defect detection method of this application embodiment. The processor 1902 is used to execute the aforementioned defect detection method, such as... Figure 4 As shown.

[0261] In another embodiment, the electronic device may also be other electronic devices, such as... Figure 3 The terminal device 310 is shown. In this embodiment, the electronic device can be structured as follows: Figure 20 As shown, it includes components such as: communication component 2010, memory 2020, display unit 2030, camera 2040, sensor 2050, audio circuit 2060, Bluetooth module 2070, and processor 2080.

[0262] The communication component 2010 is used to communicate with the server. In some embodiments, it may include a Circuit-Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. Electronic devices can use the WiFi module to help users send and receive information.

[0263] The memory 2020 can be used to store software programs and data. The processor 2080 executes various functions of the terminal device 310 and performs data processing by running the software programs or data stored in the memory 2020. The memory 2020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 2020 stores an operating system that enables the terminal device 310 to run. In this application, the memory 2020 may store the operating system and various applications, and may also store a computer program that executes the defect detection method of the embodiments of this application.

[0264] The display unit 2030 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 310, forming a graphical user interface (GUI). Specifically, the display unit 2030 may include a display screen 2032 disposed on the front of the terminal device 310. The display screen 2032 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 2030 can be used to display the defect detection user interface, etc., as described in the embodiments of this application.

[0265] The display unit 2030 can also be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 310. Specifically, the display unit 2030 may include a touch screen 2031 disposed on the front of the terminal device 310, which can collect touch operations of the user on or near it, such as clicking a button, dragging a scroll box, etc.

[0266] The touchscreen 2031 can be placed over the display screen 2032, or the touchscreen 2031 and the display screen 2032 can be integrated to realize the input and output functions of the terminal device 310. After integration, it can be referred to as a touch display screen. In this application, the display unit 2030 can display the application program and the corresponding operation steps.

[0267] Camera 2040 can be used to capture still images, which users can then post comments on via the application. There can be one or multiple cameras 2040. An object is projected onto a photosensitive element through a lens, generating an optical image. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the processor 2080 to be converted into a digital image signal.

[0268] The terminal device may also include at least one sensor 2050, such as an accelerometer 2051, a proximity sensor 2052, a fingerprint sensor 2053, and a temperature sensor 2054. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0269] Audio circuitry 2060, speaker 2061, and microphone 2062 provide an audio interface between the user and terminal device 310. Audio circuitry 2060 converts received audio data into electrical signals, which are then transmitted to speaker 2061, where they are converted into sound signals for output. Terminal device 310 may also be equipped with volume buttons for adjusting the volume of the sound signal. Conversely, microphone 2062 converts collected sound signals into electrical signals, which are then received by audio circuitry 2060, converted back into audio data, and output to communication component 2010 for transmission to, for example, another terminal device 310, or to memory 2020 for further processing.

[0270] The Bluetooth module 2070 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 2070, thereby exchanging data.

[0271] The processor 2080 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 2020 and calling data stored in the memory 2020. In some embodiments, the processor 2080 may include one or more processing units; the processor 2080 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 2080. In this application, the processor 2080 can run the operating system, applications, user interface display and touch response, as well as the defect detection method of this application embodiment. Furthermore, the processor 2080 is coupled to the display unit 2030.

[0272] In some possible implementations, various aspects of the defect detection method provided in this application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps in the defect detection method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 4 The steps are shown in the figure.

[0273] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0274] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0275] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0276] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0277] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0278] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0279] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0280] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0281] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0282] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0283] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0284] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0285] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A defect detection method, characterized in that, The method includes: Obtain the image to be detected corresponding to the object to be detected, and the normal vector image corresponding to the image to be detected. The image to be detected is obtained by taking a full-light image of the object to be detected. The pixel value of each pixel in the normal vector image is the normal vector coordinate of the corresponding pixel in the image to be detected. The normal vector coordinate is used to represent the orientation of the object point on the surface of the object to be detected corresponding to the pixel. At least three rounds of feature extraction are performed based on the image to be detected and the normal vector image, and the two features to be fused obtained in the last round of feature extraction are concatenated to obtain the target fusion feature; Based on the target fusion features, defect detection is performed on the image to be detected to obtain at least one candidate defect region; For each candidate defect region, perform the following operations: Based on the position of each pixel within a candidate defect region, the pixels are divided to obtain each boundary pixel and each intermediate pixel. Set the depth value of each boundary pixel to the corresponding preset value; The depth value of each pixel is determined based on the normal vector coordinates of each intermediate pixel and the depth value of the associated point, wherein the associated point of an intermediate pixel is a pixel whose positional relationship with the intermediate pixel conforms to a preset positional relationship; If any of the depth values ​​is greater than a preset depth threshold, then the candidate defect region is taken as the target defect region.

2. The method as described in claim 1, characterized in that, In one round of feature extraction, the following steps are performed: Based on the feature extraction network, features are extracted from the two input data respectively to obtain the input features corresponding to the two input data; Based on the feature fusion network, feature fusion is performed on the two input features respectively to obtain the initial fused features corresponding to the two input features; Wherein, if the first round of feature extraction is the first round of feature extraction, then the two input data are the image to be detected and the normal vector map; otherwise, the two input data are the two initial fusion features output from the previous round of feature extraction.

3. The method as described in claim 2, characterized in that, The feature extraction network includes an image feature extraction layer and a normal vector feature extraction layer, and the feature fusion network includes an image feature fusion layer and a normal vector feature fusion layer. The step of extracting features from the two input data using a feature extraction network to obtain the input features corresponding to each of the two input data includes: Based on the image feature extraction layer, feature extraction is performed on the image input data to obtain image fusion features. The image input data is the image to be detected or the initial fusion features output by the image feature fusion layer in the previous round of feature extraction. Based on the normal vector feature extraction layer, feature extraction is performed on the normal vector input data to obtain normal vector fusion features. The normal vector input data is the normal vector image or the initial fusion features output by the normal vector feature fusion layer in the previous round of feature extraction.

4. The method as described in claim 3, characterized in that, The feature fusion network performs feature fusion on the two input features respectively to obtain the initial fused features corresponding to the two input features, including: Based on the image feature fusion layer, the image fusion feature and the normal vector fusion feature are fused to obtain the first fusion feature; The first fusion feature is concatenated with the image fusion feature to obtain the initial fusion feature corresponding to the image fusion feature; Based on the normal vector feature fusion layer, the image fusion feature and the normal vector fusion feature are fused to obtain a second fusion feature; The second fusion feature is concatenated with the normal vector fusion feature to obtain the initial fusion feature corresponding to the normal vector fusion feature.

5. The method as described in claim 1, characterized in that, After acquiring the image to be detected and before acquiring the normal vector image, the normal vector coordinates of each pixel in the image to be detected are obtained in the following way: At least three polarized light sources selected from the candidate light source set are used to photograph the object to be detected to obtain at least three polarized light images, each of the candidate light sources in the candidate light source set corresponding to a different direction; Based on the pixel values ​​of each pixel in the at least three polarized images, the normal vector coordinates corresponding to each pixel in the image to be detected are obtained.

6. The method as described in claim 5, characterized in that, After obtaining the normal vector coordinates corresponding to each pixel in the image to be detected based on the pixel values ​​of each pixel in the at least three polarized images, the normal vector image is obtained by the following method: The pixel values ​​of each pixel in the image to be detected are replaced with the corresponding normal vector coordinates, and the image is drawn to obtain the normal vector image.

7. The method as described in claim 4, characterized in that, The step of fusing image features based on the image feature fusion layer and the normal vector fusion feature to obtain a first fused feature includes: The image fusion features are subjected to feature mapping to obtain the image fusion vector; The normal vector fusion features are subjected to feature mapping to obtain a first fusion vector and a second fusion vector; Image feature fusion is performed on the image fusion vector, the first fusion vector, and the second fusion vector to obtain the first fusion feature.

8. The method as described in claim 4, characterized in that, The step of fusing the image fusion features and the normal vector fusion features based on the normal vector feature fusion layer to obtain the second fusion feature includes: The image fusion features are subjected to feature mapping to obtain a third fusion vector and a fourth fusion vector; Perform feature mapping on the fused normal vector features to obtain the fused normal vector; The third fusion vector, the fourth fusion vector, and the normal vector fusion vector are fused using normal vector features to obtain the second fusion feature.

9. A defect detection device, characterized in that, include: The acquisition unit is used to acquire a detection image corresponding to the detection object and a normal vector image corresponding to the detection image. The detection image is obtained by taking a full-light image of the detection object. The pixel value of each pixel in the normal vector image is the normal vector coordinate of the corresponding pixel in the detection image. The normal vector coordinate is used to characterize the orientation of the object point on the surface of the detection object corresponding to the pixel. An extraction unit is used to perform at least three rounds of feature extraction based on the image to be detected and the normal vector image, and to concatenate the two features to be fused obtained in the last round of feature extraction to obtain the target fusion feature; The detection unit is used to perform defect detection on the image to be detected based on the target fusion features to obtain at least one candidate defect region; The determination unit is used to perform the following operations for each candidate defect region: Based on the position of each pixel within a candidate defect region, the pixels are divided to obtain each boundary pixel and each intermediate pixel. Set the depth value of each boundary pixel to the corresponding preset value; The depth value of each pixel is determined based on the normal vector coordinates of each intermediate pixel and the depth value of the associated point, wherein the associated point of an intermediate pixel is a pixel whose positional relationship with the intermediate pixel conforms to a preset positional relationship; If any of the depth values ​​is greater than a preset depth threshold, then the candidate defect region is taken as the target defect region.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 8.

12. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 8.

Images