An image processing method and related apparatus

Abstract

The application provides an image processing method and related device. The method comprises: acquiring a target detection image and a template image; performing feature comparison on the target detection image and the template image according to a twin network to generate a feature difference image; performing feature reorganization on the feature difference image to obtain a feature difference reorganization image; performing feature extraction on the target detection image to obtain a first target feature image; processing pixel point features of the first target feature image and pixel point features of the feature difference reorganization image through a mutual attention mechanism network to obtain a defect probability score of each pixel point in the target detection image, wherein the defect probability score is used to represent the similarity of pixel points at the same position; and finally, determining the position of defects in the target detection image according to the defect probability score of each pixel point in the target detection image. The method provided by the application can accurately locate defects on a PCB.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] Printed circuit boards (PCBs) are a crucial component of modern electronic devices, serving as information carriers that integrate various electronic components. They have wide applications in the electronics field, and their quality directly impacts product performance. PCB defect detection is a vital step in the PCB production line and a key factor in evaluating PCB quality, directly affecting the performance and safety of electronic products.

[0003] With the continuous development of computer vision technology, defect detection on PCB circuits using computer vision technology has gradually replaced manual inspection of PCB circuit boards using magnifying glasses or calibrated microscopes. Currently, commonly used PCB defect recognition methods based on object detection take the image to be detected as input and train on a large amount of data to find all possible defect types. This requires even larger-scale labeled data to directly learn the patterns of different types of defects. However, because the yield rate on actual production lines is already relatively high, defect collection is quite difficult, especially for a few uncommon defect types. This method cannot accurately locate defects on the PCB circuit board, resulting in poor defect localization accuracy. Summary of the Invention

[0004] This application provides an image processing method and related apparatus. By comparing the pixel features in the target detection image and the template image using a twin network and a mutual attention mechanism network, the method identifies pixels with defects in the target detection image and thus determines the location of the defects in the target detection image. This method can accurately locate defects on PCB circuit boards.

[0005] One aspect of this application provides an image processing method, comprising:

[0006] Acquire the target detection image and template image, wherein the target detection image and template image correspond to the same type of printed circuit board;

[0007] Based on the Siamese network, feature comparison is performed between the target detection image and the template image to generate a feature difference image;

[0008] Feature reconstruction is performed on the feature difference image to obtain the feature difference reconstructed image. Feature reconstruction refers to rearranging the positions of pixels in the feature difference image.

[0009] Feature extraction is performed on the target detection image to obtain the first target feature image;

[0010] The pixel features of the first target feature image and the pixel features of the reconstructed image are processed by a mutual attention mechanism network to obtain the defect probability score of each pixel in the target detection image. The defect probability score is used to characterize the similarity of pixels at the same position.

[0011] The location of defects in the target detection image is determined based on the defect probability score of each pixel in the target detection image.

[0012] Another aspect of this application provides an image processing model training method, comprising:

[0013] Acquire training detection images and training template images, wherein the training detection images and training template images correspond to the same type of printed circuit board;

[0014] Based on the Siamese network, feature comparison is performed between the training detection image and the training template image to generate a training feature difference image;

[0015] The training feature difference image is reconstructed to obtain the training feature difference reconstructed image. Here, training feature reconstruction refers to rearranging the positions of pixels in the training feature difference image.

[0016] Feature extraction is performed on the training detection images to obtain the first training feature image;

[0017] The pixel features of the first training feature image and the pixel features of the reconstructed image are processed by the mutual attention mechanism network to obtain the defect probability score of each pixel in the training detection image. The defect probability score is used to characterize the similarity of pixels at the same position.

[0018] The predicted location of the defect in the training detection image is determined based on the defect probability score of each pixel in the training detection image.

[0019] A loss function is generated based on the predicted location of defects in the training detection images and the location labels of defects in the training detection images;

[0020] The Siamese network and the mutual attention mechanism network are trained based on the loss function.

[0021] Another aspect of this application provides an image processing apparatus, comprising:

[0022] The image acquisition module is used to acquire target detection images and template images, wherein the target detection images and template images correspond to the same type of printed circuit board;

[0023] The feature comparison module is used to compare the features of the target detection image and the template image based on the Siamese network and generate a feature difference image.

[0024] The feature reconstruction module is used to reconstruct the feature difference image to obtain a feature difference reconstructed image. Feature reconstruction refers to rearranging the positions of pixels in the feature difference image.

[0025] The feature extraction module is used to extract features from the target detection image to obtain the first target feature image;

[0026] The defect probability score calculation module is used to process the pixel features of the first target feature image and the pixel features of the reconstructed image through the mutual attention mechanism network to obtain the defect probability score of each pixel in the target detection image. The defect probability score is used to characterize the similarity of pixels at the same position.

[0027] The defect location determination module is used to determine the location of defects in the target detection image based on the defect probability score of each pixel in the target detection image.

[0028] In another implementation of this application embodiment, the defect probability score calculation module is further used for:

[0029] Obtain the pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the reconstructed image based on the feature difference;

[0030] The pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the feature difference reconstructed image are processed by the mutual attention mechanism to obtain the defect probability score of each pixel in the target detection image.

[0031] In another implementation of this application embodiment, the defect probability score calculation module is further used for:

[0032] Using the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value, the pixel features corresponding to each pixel in the feature difference reconstructed image are processed through a mutual attention mechanism to obtain the target pixel features.

[0033] The defect probability score of each pixel in the target detection image is calculated based on the characteristics of the target pixel.

[0034] In another implementation of this application embodiment, the defect probability score calculation module is further used for:

[0035] Using the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value, the first pixel features are extracted from the pixel features corresponding to each pixel in the feature difference reconstructed image through a mutual attention mechanism.

[0036] Obtain the weight of the pixel corresponding to the feature of the first pixel;

[0037] The target pixel features are calculated based on the features of the first pixel and the weights corresponding to the pixels.

[0038] In another implementation of this application embodiment, the feature comparison module is further used for:

[0039] The first feature extraction subnetwork in the Siamese network is used to extract features from the target detection image to generate a second target feature image;

[0040] The template image is used to extract features from the template image based on the second feature extraction subnetwork in the Siamese network, thereby generating a template feature image;

[0041] The feature images of the second target and the template are compared to generate a feature difference image.

[0042] In another implementation of this application embodiment, the feature reorganization module is further configured to:

[0043] The target positions of each pixel and its corresponding P pixels in the feature difference image are arranged to generate a feature difference reconstructed image, where P is an integer greater than 1.

[0044] In another implementation of this application's embodiments, the defect location determination module is further configured to:

[0045] The target detection image is processed by L convolutional networks with different dilation rates to obtain L convolutional feature images, where L is an integer greater than 1;

[0046] Perform feature concatenation on L convolutional feature images to generate a stitched image;

[0047] Based on the stitched image, determine the pixel points corresponding to copper wire defects in the target detection image;

[0048] The location of the defect in the target detection image is determined based on the pixel corresponding to the copper wire defect in the target detection image and the defect probability score of each pixel in the target detection image.

[0049] In another implementation of this application embodiment, the feature comparison module is further used for:

[0050] The target detection image is subjected to M feature extractions based on the first pyramid feature extraction subnetwork in the Siamese network, generating M second target feature images. The pyramid feature extraction subnetwork includes M feature extraction layers, and the input of each feature layer is the output of the previous feature layer. M is an integer greater than or equal to 1.

[0051] Based on the second pyramid feature extraction subnetwork in the Siamese network, M feature extractions are performed on the template image to generate M template feature images;

[0052] Based on the feature comparison between the M second target feature images and the M template feature images, M feature difference images are generated.

[0053] In another implementation of this application embodiment, the feature extraction module is further configured to: perform M feature extractions on the target detection image to obtain M first target feature images, wherein the input of each feature extraction is the output of the previous feature extraction;

[0054] The defect probability score calculation module is also used for:

[0055] The pixel features of the M first target feature images and the pixel features of the M feature difference reconstructed images are processed by a mutual attention mechanism network to obtain the M initial defect probability scores corresponding to each pixel in the target detection image.

[0056] Based on the M initial defect probability scores corresponding to each pixel in the target detection image, determine the defect probability score of each pixel in the target detection image.

[0057] Another aspect of this application provides an image processing model training apparatus, comprising:

[0058] The training image acquisition module is used to acquire training detection images and training template images, wherein the training detection images and training template images correspond to the same type of printed circuit board.

[0059] The training feature comparison module is used to compare the features of the training detection image and the training template image based on the Siamese network, and generate a training feature difference image.

[0060] The training feature reorganization module is used to reorganize the training feature difference image to obtain the training feature difference reorganized image. Here, training feature reorganization refers to rearranging the positions of pixels in the training feature difference image.

[0061] The training feature extraction module is used to extract features from the training detection images to obtain the first training feature image;

[0062] The training defect probability score calculation module is used to process the pixel features of the first training feature image and the pixel features of the reconstructed image by the training feature difference through a mutual attention mechanism network to obtain the defect probability score of each pixel in the training detection image. The defect probability score is used to characterize the similarity of pixels at the same position.

[0063] The training defect location determination module is used to determine the predicted location of defects in the training detection image based on the defect probability score of each pixel in the training detection image.

[0064] The loss function calculation module is used to generate a loss function based on the predicted location of the defect in the training detection image and the location label of the defect in the training detection image;

[0065] The model training module is used to train the Siamese network and the mutual attention mechanism network according to the loss function.

[0066] Another aspect of this application provides a computer device, comprising:

[0067] Memory, transceiver, processor, and bus system;

[0068] The memory is used to store programs;

[0069] The processor is used to execute programs in memory, including methods for performing the aspects mentioned above;

[0070] Bus systems are used to connect memory and processor to enable communication between them.

[0071] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0072] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0073] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0074] This application provides an image processing method and related apparatus. The method includes: first, acquiring a target detection image and a template image, wherein the target detection image and the template image correspond to the same type of printed circuit board; second, performing feature comparison on the target detection image and the template image according to a Siamese network to generate a feature difference image; third, performing feature recombination on the feature difference image to obtain a feature difference recombined image, wherein feature recombination refers to rearranging the positions of pixels in the feature difference image; next, performing feature extraction on the target detection image to obtain a first target feature image; then, processing the pixel features of the first target feature image and the pixel features of the feature difference recombined image through a mutual attention mechanism network to obtain a defect probability score for each pixel in the target detection image, wherein the defect probability score is used to characterize the similarity of pixels at the same position; finally, determining the position of the defect in the target detection image based on the defect probability score of each pixel in the target detection image. This application provides an image processing method and related apparatus. By comparing the pixel features in the target detection image and the template image through a three-branch network structure composed of a twin network and a mutual attention mechanism network, the method identifies the pixels with defects in the target detection image and thus determines the location of the defects in the target detection image, which can accurately locate defects on the PCB circuit board. Attached Figure Description

[0075] Figure 1 This is a schematic diagram illustrating a defect in the holes of a PCB board with missing solder, provided in a certain embodiment of this application.

[0076] Figure 2 A schematic diagram illustrating a defect in the copper lines of a PCB board provided in a certain embodiment of this application;

[0077] Figure 3 A schematic diagram of the architecture of an image processing system provided in a certain embodiment of this application;

[0078] Figure 4 A flowchart of an image processing method provided in one embodiment of this application;

[0079] Figure 5 This is a target detection image provided in one embodiment of this application;

[0080] Figure 6 Template image provided for one embodiment of this application;

[0081] Figure 7 A flowchart of an image processing method provided in another embodiment of this application;

[0082] Figure 8 A flowchart of an image processing method provided in another embodiment of this application;

[0083] Figure 9 A flowchart of an image processing method provided in another embodiment of this application;

[0084] Figure 10 A flowchart of an image processing method provided in another embodiment of this application;

[0085] Figure 11 A model diagram of a three-branch network structure provided in one embodiment of this application;

[0086] Figure 12 A flowchart of an image processing method provided in another embodiment of this application;

[0087] Figure 13 A schematic diagram illustrating feature recombination provided in a certain embodiment of this application;

[0088] Figure 14 A flowchart of an image processing method provided in another embodiment of this application;

[0089] Figure 15 A structural diagram of an edge-sensitive change prediction module provided in a certain embodiment of this application;

[0090] Figure 16 A flowchart of an image processing method provided in another embodiment of this application;

[0091] Figure 17 This is a schematic diagram of an image processing method provided in one embodiment of this application;

[0092] Figure 18 This is a ResNet50 network structure diagram provided in one embodiment of this application;

[0093] Figure 19 A flowchart of an image processing method provided in another embodiment of this application;

[0094] Figure 20 A flowchart illustrating an image processing model training method provided in one embodiment of this application;

[0095] Figure 21 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this application;

[0096] Figure 22 This is a schematic diagram of the structure of an image processing model training device provided in a certain embodiment of this application;

[0097] Figure 23 This is a schematic diagram of a server structure provided in one embodiment of this application. Detailed Implementation

[0098] This application provides an image processing method that compares the pixel features in a target detection image and a template image using a three-branch network structure composed of a Siamese network and a mutual attention mechanism network to determine the pixels with defects in the target detection image, thereby determining the location of defects in the target detection image and accurately locating defects on a PCB circuit board.

[0099] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0100] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0101] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0102] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0103] As a crucial component of modern electronic devices, the PCB (Printed Circuit Board) serves as the information carrier integrating various electronic components and has wide applications in the electronics field. Its quality directly impacts product performance. PCB defect detection is a vital step in the PCB production line and a key factor in evaluating PCB quality, directly affecting the performance and safety of electronic products. Figure 1 and Figure 2 The diagram shows two common types of defects. Figure 1 The defect is due to missing solder in the hole. Figure 2 The defect is due to missing parts on the copper wire.

[0104] Traditional PCB visual inspection methods rely on magnifying glasses or calibrated microscopes, with operators visually inspecting the circuit boards to determine their quality and when corrections are needed. Its disadvantages include human subjectivity, long inspection times, slow speed, high long-term costs, and difficulties in data collection. With increasing PCB production volume and continuous advancements in manufacturing technology, larger, higher-density, and higher-precision PCBs are becoming the development trend, demanding increasingly stringent specifications. Traditional visual inspection with magnifying glasses and microscopes is no longer sufficient to filter all defects and cannot meet modern production needs. Utilizing machine vision inspection technology to promptly detect defects present during the soldering process has become the primary means of PCB defect detection.

[0105] With the continuous development of computer vision, machine vision inspection technology can not only detect typical defects such as short circuits and open circuits, but also effectively detect various other defects, such as vias, excess copper, holes, burrs, gaps, misaligned holes, insufficient etching, and incorrect drilling. This effectively avoids the drawbacks of manual inspection, such as high labor intensity, poor stability, and low efficiency, enabling high-efficiency inspection 24 / 7. Furthermore, even if the PCB type and defect standards change, artificial intelligence technology can achieve optimization through incremental learning and domain adaptation.

[0106] The following are two methods for defect detection on PCB boards based on machine vision in related technologies:

[0107] One type is based on object detection. These methods directly use mature object detection frameworks such as Faster R-CNN and Cascaded R-CNN to detect defect regions. They take the image to be detected as input and train on a large amount of data to find all possible defect types. The second type is based on weakly supervised object localization. These methods do not require bounding boxes; they only need to know whether a defect exists in an image. They utilize mature recognition models as the backbone network, using algorithms like CAM to extract the most discriminative regions in the image, and then obtain the bounding rectangle of the defect by finding the maximum connected component.

[0108] The drawback of these two methods is that they require large-scale labeled data to directly learn the patterns of different types of defects. However, since the yield on actual production lines is relatively high, defect collection is difficult, especially for a few uncommon types of defects, where the method cannot accurately locate uncommon types of defects.

[0109] The image processing method provided in this application uses a three-branch-based image processing model. Two branches, similar to a Siamese network, are used to extract and compare features from the template image and the target detection image, respectively. For the differences in features, the third branch dynamically references the discriminative content in the feature differences through a mutual attention mechanism network, thereby enabling the third branch to output the region that is actually a defect, thus improving the accuracy of locating defects on the PCB circuit board.

[0110] For easier understanding, please refer to Figure 3 , Figure 3 This is an application environment diagram of the image processing method in the embodiments of this application, such as... Figure 3As shown, the image processing method in this embodiment is applied to an image processing system. The image processing system includes a server and terminal devices. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0111] The server first acquires the target detection image and the template image, where the target detection image and the template image correspond to the same type of printed circuit board. Next, the server performs feature comparison on the target detection image and the template image using a Siamese network to generate a feature difference image. Then, the server performs feature reconstruction on the feature difference image to obtain a feature difference reconstructed image, where feature reconstruction refers to rearranging the positions of pixels in the feature difference image. Next, the server extracts features from the target detection image to obtain a first target feature image. Then, the server processes the pixel features of the first target feature image and the pixel features of the feature difference reconstructed image using a mutual attention mechanism network to obtain a defect probability score for each pixel in the target detection image, where the defect probability score characterizes the similarity of pixels at the same position. Finally, the server determines the location of the defect in the target detection image based on the defect probability score of each pixel.

[0112] The image processing method in this application will be described from the server's perspective below. Please refer to [link / reference needed]. Figure 4 The image processing method provided in this application includes steps S110 to S160. Specifically:

[0113] S110. Obtain the target detection image and template image.

[0114] The target detection image and the template image correspond to the same type of printed circuit board.

[0115] It is understandable that the process involves acquiring the printed circuit board (PCB) and its template circuit of a given circuit board, where the PCB can be obtained by printing the PCB from the template circuit. Image acquisition is then performed on the PCB to obtain a target detection image, and image acquisition is performed on the template circuit to obtain a template image. For example... Figure 5 and Figure 6 As shown, Figure 5The image obtained from the printed circuit board (PCB) of a certain circuit board is a target detection image. Due to certain problems during the PCB printing process, the copper lines on the PCB have defects (such as...). Figure 5 (within the marker box) Figure 6 This is a template image obtained from the acquisition of a template circuit of a certain circuit board.

[0116] S120. Based on the Siamese network, the feature difference image is generated by comparing the target detection image and the template image.

[0117] It should be noted that Siamese Network, also known as a conjoined network, is a special type of deep learning network architecture. Unlike deep models with a single image input, this neural network learns to distinguish and identify the differences between two inputs. They often have the same configuration, i.e., the same parameters and weights, and parameter updates are performed jointly on both subnetworks.

[0118] Understandably, the first feature extraction subnetwork in the Siamese network extracts features from the target detection image to generate a target feature image; the second feature extraction subnetwork in the Siamese network extracts features from the template image to generate a template feature image. A feature difference image is then generated by comparing the target and template feature images. This feature comparison can be achieved by subtracting the target and template feature images, and the feature difference image is generated based on the result. This feature difference image retains the pixel features corresponding to pixels at the same positions in the target detection image and the template image that have different pixel features.

[0119] S130. Perform feature reconstruction on the feature difference image to obtain the feature difference reconstructed image.

[0120] Feature recombination refers to rearranging the positions of pixels in the feature difference image.

[0121] Understandably, the target positions of each pixel and its corresponding P pixels in the feature difference image are arranged to generate a feature difference reconstructed image. The P pixels corresponding to each pixel can be the eight pixels surrounding that pixel.

[0122] S140. Perform feature extraction on the target detection image to obtain the first target feature image.

[0123] S150. The pixel features of the first target feature image and the pixel features of the reconstructed image are processed by the mutual attention mechanism network to obtain the defect probability score of each pixel in the target detection image.

[0124] Among them, the defect probability score is used to characterize the similarity of pixels at the same location.

[0125] It is understandable that the pixel features of the first target feature image and the pixel features of the feature difference reconstructed image are compared at the same position through the mutual attention mechanism network to find pixels with a similarity higher than a preset value. The similarity of pixels at the same position is the defect probability score of that pixel.

[0126] S160. Determine the location of the defect in the target detection image based on the defect probability score of each pixel in the target detection image.

[0127] Understandably, the location of a defect is determined based on the positions of pixels in the target detection image whose defect probability scores are higher than a preset value. A defect probability score map is obtained based on the defect probability score of each pixel in the target detection image. This map is then binarized, and the largest connected component is found to determine the bounding box, thereby determining the defect detection box.

[0128] This application provides an image processing method that compares the pixel features in the target detection image and the template image using a three-branch network structure composed of a Siamese network and a mutual attention mechanism network to determine the pixels with defects in the target detection image, thereby determining the location of the defects in the target detection image, which can accurately locate defects on the PCB circuit board.

[0129] In this application Figure 4 In one optional embodiment of the image processing method provided in the corresponding embodiment, please refer to... Figure 7 Step S150 further includes sub-steps S151 to S152. Specifically:

[0130] S151. Obtain the pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the feature difference reconstructed image.

[0131] It is understandable that the first target feature image is composed of pixel features corresponding to each pixel, and the pixel features corresponding to each pixel in the first target feature image are obtained. The feature difference reconstructed image is composed of pixel features corresponding to each pixel, and the pixel features corresponding to each pixel in the feature difference reconstructed image are obtained.

[0132] S152. The pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the feature difference reconstructed image are processed by the mutual attention mechanism to obtain the defect probability score of each pixel in the target detection image.

[0133] Understandably, for pixels at the same location, the pixel features in the first target feature image and the pixel features in the feature difference reconstructed image corresponding to the pixel are processed through the mutual attention mechanism to obtain the similarity between the pixel features in the first target feature image and the pixel features in the feature difference reconstructed image. The defect probability score corresponding to the pixel is obtained by calculating the similarity.

[0134] This application provides an image processing method that uses a mutual attention mechanism to calculate the similarity between the pixel features of each pixel in the first target feature image and the pixel features of the same pixel in the feature difference reconstructed image, thereby obtaining the defect probability score corresponding to each pixel. This improves the accuracy of the defect probability score calculation and lays the foundation for improving the location of defects in the target detection image.

[0135] In this application Figure 7 In one optional embodiment of the image processing method provided in the corresponding embodiment, please refer to... Figure 8 Sub-step S151 further includes sub-steps S1511 to S1512. Specifically:

[0136] S1511. Using the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value, the pixel features corresponding to each pixel in the feature difference reconstructed image are processed through a mutual attention mechanism to obtain the target pixel features.

[0137] It is understandable that the first target feature image is F, and the feature difference reconstructed image is D. For each pixel i on F, the pixel feature corresponding to each pixel i is used as the key, and the pixel i on D and P pixels that have a corresponding relationship with pixel i are used as the value. The useful information in the feature difference reconstructed image is dynamically extracted through the mutual attention mechanism to obtain the first pixel feature. The first pixel feature is multiplied by a learnable parameter to obtain the target pixel feature.

[0138] S1512. Calculate the defect probability score of each pixel in the target detection image based on the target pixel features.

[0139] It is understandable that the defect probability score of each pixel in the target detection image can be calculated based on the characteristics of the target pixel.

[0140] This application provides an image processing method that uses the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value. Through a mutual attention mechanism, the similarity between the pixel features of each pixel in the first target feature image and the pixel features of the same pixel in the feature difference reconstructed image is calculated to obtain the defect probability score corresponding to each pixel. This improves the accuracy of the defect probability score calculation and lays the foundation for improving the location of defects in the target detection image.

[0141] In this application Figure 8 In one optional embodiment of the image processing method provided in the corresponding embodiment, please refer to... Figure 9 Sub-step S1511 further includes sub-steps S15111 to S15113. Specifically:

[0142] S15111. Using the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value, extract the first pixel features from the pixel features corresponding to each pixel in the feature difference reconstructed image through a mutual attention mechanism.

[0143] Understandably, the first target feature image is F, and the feature difference reconstructed image is D. For each pixel i in F, using the pixel feature corresponding to each pixel i as the key, and using pixel i in D and P pixels corresponding to pixel i as the value, a mutual attention mechanism is used to dynamically extract useful information from the feature difference reconstructed image to obtain the first pixel feature. The first pixel feature can be represented by the following formula:

[0144] ;

[0145] in, and For each pixel in the first target feature image, the pixel feature is represented. To reconstruct the pixel features corresponding to each pixel in the image using feature difference, The dimension or number of channels of the input feature.

[0146] S15112. Obtain the weight of the pixel corresponding to the feature of the first pixel.

[0147] It is understandable that the weight of a pixel is a learnable parameter or the number of channels used to balance the relative weights of detail information and contextual information.

[0148] S15113. Based on the features of the first pixel and the weights corresponding to the pixels, the features of the target pixel are calculated.

[0149] It is understandable that the product is calculated as the feature of the target pixel by performing a multiplication operation based on the features of the first pixel and the weights corresponding to the pixels.

[0150] This application provides an image processing method that uses the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value. Through a mutual attention mechanism, the similarity between the pixel features of each pixel in the first target feature image and the pixel features of the same pixel in the feature difference reconstructed image is calculated to obtain the defect probability score corresponding to each pixel. This improves the accuracy of the defect probability score calculation and lays the foundation for improving the location of defects in the target detection image.

[0151] In this application Figure 4 In one optional embodiment of the image processing method provided in the corresponding embodiment, please refer to... Figure 10 Step S120 further includes sub-steps S121 to S123. Specifically:

[0152] S121. Based on the first feature extraction subnetwork in the Siamese network, feature extraction is performed on the target detection image to generate a second target feature image.

[0153] S122. Extract features from the template image based on the second feature extraction subnetwork in the Siamese network to generate a template feature image.

[0154] It is understood that the image processing method provided in this application uses a three-branch network structure, where two branches are Siamese networks and one branch is a mutual attention mechanism network. The inputs to the two branches of the Siamese network are the target detection image and the template image, respectively. By extracting features from the template image and the image to be detected and comparing the features, some potential defective regions can be discovered, which can then be used as a reference for the third branch network.

[0155] Sub-steps S121 and S122 can be parallel steps or steps with a sequential relationship. They can be executed either S121 first and then S122, or vice versa. This embodiment of the application does not impose such limitations. Features are extracted from the target detection image through a first feature extraction subnetwork of one branch of the Siamese network to generate a second target feature image. Features are extracted from the template image through a second feature extraction subnetwork of another branch of the Siamese network to generate a template feature image.

[0156] S123. Perform feature comparison on the second target feature image and the template feature image to generate a feature difference image.

[0157] Understandably, the target feature image and the template feature image are compared by subtraction to obtain the regions that differ between the two images, generating a feature difference image. The feature difference image is then fed into the third branch, where the mutual attention mechanism network dynamically extracts the discriminative parts of the feature difference through an adaptive change extraction module, thereby assisting its own judgment.

[0158] For easier understanding, please refer to Figure 11 , Figure 11 This is a model diagram of a three-branch network structure provided in an embodiment of this application. The first and second branches form a Siamese network, and the third branch is a mutual attention mechanism network. The input to the first branch is the target detection image (e.g., ...). Figure 5 The input to the second branch is a template image (such as...). Figure 6 The input to the third branch is the target detection image (e.g., Figure 5 The first branch extracts features from the target detection image using the first feature extraction subnetwork to generate a second target feature image. The second branch then extracts features from the feature image using the second feature extraction subnetwork to generate a template feature image. The second target feature image and the template feature image are compared to generate a feature difference image. The third branch extracts features from the target detection image using the third feature extraction subnetwork to obtain the first target feature image. Finally, the mutual attention mechanism network in the third branch processes the pixel features of the first target feature image and the pixel features of the feature difference image to obtain the defect probability score for each pixel in the target detection image.

[0159] This application provides an image processing method that adopts a model based on a three-branch network structure, in which two branches are Siamese networks and one branch is a mutual attention mechanism network. The design of the three-branch network structure can not only have the advantages of the mainstream Siamese network design, but also avoid the problem of poor parameter adaptability caused by weight sharing in Siamese networks. This allows the branch responsible for defect localization to be more specialized, and achieves more accurate identification of defect regions by dynamically checking the feature differences between Siamese networks.

[0160] In this application Figure 4 In one optional embodiment of the image processing method provided in the corresponding embodiment, please refer to... Figure 12 Step S130 further includes sub-step S131. Specifically:

[0161] S131. Arrange the target positions of each pixel and its corresponding P pixels in the feature difference image to generate a feature difference reconstructed image.

[0162] Where P is an integer greater than 1.

[0163] Understandably, for each pixel i in the feature difference image, pixel i and its corresponding P pixels are arranged at their target positions to generate a reconstructed feature difference image. The P pixels corresponding to pixel i can be the 8 pixels adjacent to pixel i. By reconstructing pixel i and its 8 adjacent pixels, contextual information can be taken into account.

[0164] For easier understanding, please refer to Figure 13 , Figure 13 This diagram illustrates the feature reorganization provided in this application embodiment. Due to the influence of the manufacturing process, minor normal differences inevitably appear between the target detection image and the template image, such as removable stains. However, some of these are difficult to distinguish directly from defects. To filter out these non-defect-like differences during defect detection and localization, this application embodiment uses an adaptive change extraction module to dynamically extract truly meaningful difference regions from the differences between the two, thereby filtering out non-real defects. Specifically, for the feature difference image output by the Siamese network, this module first reorganizes the feature difference image to take into account contextual information, that is, copies the features corresponding to the 8 pixels surrounding each pixel to a new channel dimension. That is, assuming the original feature dimension is n × c × h × w, the dimension of the reorganized feature difference image is n × 9 × c × h × w, where the newly added channel contains the feature difference corresponding to the current pixel and its 8 surrounding pixels.

[0165] This application provides an image processing method that, in order to distinguish between real and non-real defects, reconstructs the feature difference image output by a Siamese network by recombining the pixels in the feature difference image to generate a reconstructed feature difference image. This dynamically extracts the truly meaningful difference regions from the differences between the two, thereby filtering out non-real defects. Furthermore, a module based on a mutual attention mechanism adaptively extracts discriminative regions from the feature difference, effectively avoiding interference from normal abnormal changes such as dirt caused by the production process, thus achieving high accuracy in defect detection.

[0166] In this application Figure 4 In one optional embodiment of the image processing method provided in the corresponding embodiment, please refer to... Figure 14 Step S160 further includes sub-steps S161 to S164. Specifically:

[0167] S161. The target detection image is processed by L convolutional networks with different dilation rates to obtain L convolutional feature images.

[0168] Where L is an integer greater than 1.

[0169] S162. Perform feature stitching on L convolutional feature images to generate a stitched image.

[0170] S163. Based on the stitched image, determine the pixel points corresponding to copper wire defects in the target detection image.

[0171] It is understandable that PCBs are typically covered with conductors (usually copper wires). To achieve better electrical performance, these conductors are usually distributed horizontally or vertically, resulting in some defects on the PCB being distributed along these conductors. Therefore, the distribution information of the conductors is important for auxiliary defect identification. The method provided in this application uses an edge-sensitive change prediction module composed of L convolutional networks with different void ratios to assist in the identification of defects on the conductors. Figure 15 As shown, Figure 15 This is a structural diagram of the edge-sensitive change prediction module provided in this application embodiment, where c is a connection layer. It models vertical or horizontal edges by parallelly listing irregularly shaped convolutions of 1×k or k×1 shapes. To expand the modeling range, the method provided in this application embodiment uses three sets of irregular convolution modules with different dilatancy rates in sequence, from top to bottom: dilatancy rates of 1, 2, and 3. Finally, the outputs of the three modules in the concatenation process are simultaneously concatenated as the output of the entire module. It should be noted that the edge-sensitive change prediction module is not limited to... Figure 15 The form shown focuses on stacking 1×k type convolutions, which helps the network extract edge information in the horizontal and vertical directions, while also trying to integrate features from different receptive fields.

[0172] S164. Determine the location of the defect in the target detection image based on the pixel corresponding to the copper wire defect in the target detection image and the defect probability score of each pixel in the target detection image.

[0173] It is understandable that the location of the defect in the target detection image is determined by combining the copper wire defect in the target detection image with the defect probability score of each pixel in the target detection image.

[0174] This application provides an image processing method that, based on the horizontal and vertical wiring characteristics on a PCB, uses an edge-sensitive change prediction module composed of L convolutional networks with different void ratios to assist in identifying defect areas using contextual information, thereby achieving more accurate defect localization.

[0175] In this application Figure 4 In one optional embodiment of the image processing method provided in the corresponding embodiment, please refer to... Figure 16 Step S120 further includes sub-steps S1201 to S1203. Specifically:

[0176] S1201. Based on the first pyramid feature extraction subnetwork in the Siamese network, perform M feature extractions on the target detection image to generate M second target feature images.

[0177] The pyramid feature extraction subnetwork consists of M feature extraction layers, where the input of each feature layer is the output of the previous feature layer, and M is an integer greater than or equal to 1.

[0178] S1202. Based on the second pyramid feature extraction subnetwork in the Siamese network, perform M feature extractions on the template image to generate M template feature images.

[0179] It is understood that the image processing method provided in this application uses a three-branch network structure, where two branches are Siamese networks and one branch is a mutual attention mechanism network. The inputs to the two branches of the Siamese network are the target detection image and the template image, respectively. By extracting features from the template image and the image to be detected, and comparing the extracted second target feature image with the template feature image, defects can be located, thus providing a reference for the third branch network. The feature extraction sub-networks of the first and second branches of the Siamese network have a pyramid structure, meaning that the input for the subsequent feature extraction is the output of the previous feature extraction.

[0180] Please see Figure 17 , Figure 17 This is a schematic diagram of the image processing method provided in an embodiment of this application. The Siamese network used in the image processing method provided in this embodiment adopts ResNet50 as the base network, and the first and second branches have the same network structure. Each branch of the network consists of 5 blocks, each block being composed of consecutive residual units. For each residual unit, channel downsampling is first performed, then a 3×3 convolution is used for feature transformation, followed by channel upsampling to restore the original channel size, and finally, the output features are obtained by connecting it to the input residual. Figure 18 As shown, spatial downsampling is achieved between each ConvBlock through Maxpooling or convolutions with a stride of 2, which increases the network's receptive field and local translation invariance. Simultaneously, the number of baseline channels in each block increases with the depth of the network, allowing for the extraction of richer semantic information.

[0181] S1203. Based on the feature comparison between the M second target feature images and the M template feature images, generate M feature difference images.

[0182] Understandably, the first and second branches perform feature extraction operations on the target detection image and template image in parallel. Then, for the second target feature image and template feature image output from each stage of the first and second branches, the regions of difference between the two feature images are determined by subtraction. Performing feature comparison at each stage can simultaneously take into account differences at the fine-grained detail level and high-level semantic level, thereby enabling more comprehensive identification of various defects. Then, the feature difference image from each stage is fed into the corresponding stage of the third branch. The third branch network dynamically extracts the discriminative parts of the feature difference through an adaptive change extraction module, thereby assisting its own judgment and realizing the localization of defects in the target detection image.

[0183] This application provides an image processing method in which a pyramid-structured feature extraction subnetwork in a Siamese network performs feature extraction operations on the target detection image and the template image in parallel. By subtracting the output of the second target feature image and the template feature image at each stage, feature difference images are generated for each stage. Performing feature comparison at each stage can simultaneously take into account differences at the fine-grained detail level and differences at the high-level semantic level, thereby enabling more comprehensive identification of various defects and improving the accuracy of defect localization in the target detection image.

[0184] In this application Figure 16 In one optional embodiment of the image processing method provided in the corresponding embodiment, please refer to... Figure 19 Step S140 further includes sub-steps S1401 to S1402. Step S150 includes sub-steps S1501 to S1502. Specifically:

[0185] S1401. Perform feature extraction M times on the target detection image to obtain M first target feature images.

[0186] In this process, the input for each feature extraction is the output of the previous feature extraction.

[0187] It is understandable that the feature extraction subnetwork in the third branch that performs feature extraction on the target detection image is also a pyramid structure feature extraction subnetwork, and the number of layers in the pyramid structure feature extraction subnetwork in the third branch is the same as the number of layers in the pyramid structure feature extraction subnetwork in the first and second branches.

[0188] S1501. The pixel features of the M first target feature images and the pixel features of the M feature difference reconstructed images are processed by the mutual attention mechanism network to obtain the M initial defect probability scores corresponding to each pixel in the target detection image.

[0189] Understandably, the third branch takes the target detection image as input, extracts features in stages, and references the feature difference image of the corresponding stage in the Siamese network at each stage. Let F be the first target feature image extracted by the third branch at stage j (1≤j<M). j The reconstructed image based on the feature difference output by the Siamese network in stage j is D. j For F j For each pixel i, using the pixel feature corresponding to each pixel i as the key, and using D j The first pixel feature is obtained by dynamically extracting useful information from the reconstructed image using a mutual attention mechanism, based on the value of pixel i and P corresponding pixels i. The first pixel feature is then multiplied by a learnable parameter and added to the pixel feature of the third branch to obtain the target pixel feature of the j-th stage. The target pixel feature of the j-th stage is the initial defect probability score corresponding to the j-th stage. Feature extraction in subsequent stages is then carried out based on the initial defect probability score corresponding to the j-th stage.

[0190] S1502. Determine the defect probability score of each pixel in the target detection image based on the M initial defect probability scores corresponding to each pixel in the target detection image.

[0191] It is understandable that, assuming the first target feature image extracted by the third branch in the Mth stage is F M The reconstructed image based on the feature difference output by the Siamese network in stage j is D. M For F M For each pixel i, using the pixel feature corresponding to each pixel i as the key, and using D M The first pixel feature is obtained by dynamically extracting useful information from the reconstructed image through a mutual attention mechanism. The first pixel feature is then multiplied by a learnable parameter and added to the pixel feature of the third branch to obtain the target pixel feature of the Mth stage. The target pixel feature of the Mth stage is the defect probability score.

[0192] This application provides an image processing method that adaptively extracts discriminative regions from feature differences using a pyramid-structured feature extraction network. This design can effectively avoid interference from normal abnormal changes such as dirt caused by the production process, and improve the accuracy of locating defects in target detection images.

[0193] The image processing method in this application will be described from the server's perspective below. Please refer to [link / reference needed]. Figure 20 The image processing model training method provided in this application includes steps S210 to S280. Specifically:

[0194] S210. Obtain the training detection image and the training template image.

[0195] The training detection images and training template images correspond to the same type of printed circuit board.

[0196] It is understandable that the process involves acquiring a printed circuit board (PCB) and its template circuit, where the PCB can be obtained by printing the PCB from the template circuit. Image acquisition is then performed on the PCB to obtain training detection images, and image acquisition is performed on the template circuit to obtain training template images.

[0197] S220. Based on the Siamese network, perform feature comparison between the training detection image and the training template image to generate a training feature difference image.

[0198] Understandably, the first feature extraction subnetwork in the Siamese network extracts features from the training detection image to generate a training feature image; the second feature extraction subnetwork in the Siamese network extracts features from the training template image to generate a training template feature image. A feature comparison is then performed between the training feature image and the training template feature image to generate a training feature difference image. This feature comparison can be achieved by subtracting the training feature image from the training template feature image, and the training feature difference image is generated based on the result. This training feature difference image retains the pixel features corresponding to pixels at the same positions in the training detection image and the training template image that have different pixel features.

[0199] S230. Perform feature reconstruction on the training feature difference image to obtain the training feature difference reconstructed image.

[0200] Among them, training feature recombination refers to rearranging the positions of pixels in the training feature difference image.

[0201] Understandably, the target positions of each pixel and its corresponding P pixels in the training feature difference image are arranged to generate the training feature difference reconstructed image. The P pixels corresponding to each pixel can be the 8 pixels surrounding that pixel.

[0202] S240. Extract features from the training detection image to obtain the first training feature image.

[0203] S250. The pixel features of the first training feature image and the pixel features of the reconstructed image are processed by the mutual attention mechanism network to obtain the defect probability score of each pixel in the training detection image.

[0204] Among them, the defect probability score is used to characterize the similarity of pixels at the same location.

[0205] It is understandable that the pixel features of the first training feature image and the pixel features of the reconstructed image are compared at the same position through the mutual attention mechanism network to find pixels with a similarity higher than a preset value. The similarity of pixels at the same position is the defect probability score of that pixel.

[0206] S260. Determine the predicted location of the defect in the training detection image based on the defect probability score of each pixel in the training detection image.

[0207] It is understandable that the location of a defect is determined based on the location of pixels in the training detection image whose defect probability score is higher than a preset value.

[0208] S270. Generate a loss function based on the predicted location of the defect in the training detection image and the location label of the defect in the training detection image.

[0209] Understandably, for each pixel, the probability of it belonging to each defect category is predicted; that is, the classification logits feature is first obtained, and then model-supervised learning is performed using cross-entropy loss. The loss function can be expressed by the following formula:

[0210] ;

[0211] ;

[0212] in, Represents pixels The defect probability score, For pixels The classification logits feature, To train the detection of the total number of pixels in the image, Represents pixels Defect tags.

[0213] S280. Train the Siamese network and the mutual attention mechanism network according to the loss function.

[0214] This application provides an image processing model training method. The image processing model adopts a three-branch network structure. Two branches of the Siamese network are trained using a weight-sharing mode, primarily for extracting deep features. The third branch is trained independently based on the differences in feature outputs between the first two branches at each stage. This avoids the poor parameter adaptability problem caused by shared weights in Siamese networks, enabling the third branch, responsible for defect localization, to achieve more accurate defect region identification by dynamically checking feature differences between the Siamese networks. Practical experience shows that this application significantly improves PCB defect detection. Furthermore, compared to methods that directly detect defects, it effectively reduces the need for training samples, thus facilitating rapid algorithm iteration in practical applications.

[0215] The image processing apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 21 . Figure 21 This is a schematic diagram of one embodiment of the image processing apparatus 10 in this application. The image processing apparatus 10 includes:

[0216] The image acquisition module 110 is used to acquire target detection images and template images. The target detection images and template images correspond to the same type of printed circuit board.

[0217] The feature comparison module 120 is used to compare the features of the target detection image and the template image based on the Siamese network and generate a feature difference image.

[0218] The feature reconstruction module 130 is used to reconstruct the feature difference image to obtain the feature difference reconstructed image.

[0219] Feature recombination refers to rearranging the positions of pixels in the feature difference image.

[0220] The feature extraction module 140 is used to extract features from the target detection image to obtain the first target feature image.

[0221] The defect probability score calculation module 150 is used to process the pixel features of the first target feature image and the pixel features of the reconstructed image through the mutual attention mechanism network to obtain the defect probability score of each pixel in the target detection image.

[0222] Among them, the defect probability score is used to characterize the similarity of pixels at the same location.

[0223] The defect location determination module 160 is used to determine the location of the defect in the target detection image based on the defect probability score of each pixel in the target detection image.

[0224] This application provides an image processing device that compares the pixel features in a target detection image and a template image using a three-branch network structure composed of a twin network and a mutual attention mechanism network to determine the pixels with defects in the target detection image, thereby determining the location of the defects in the target detection image, and can accurately locate defects on a PCB circuit board.

[0225] In this application Figure 21 In an optional embodiment of the image processing apparatus provided in the corresponding embodiment, the defect probability score calculation module 150 is further configured to:

[0226] Obtain the pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the feature difference reconstructed image.

[0227] The pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the feature difference reconstructed image are processed by the mutual attention mechanism to obtain the defect probability score of each pixel in the target detection image.

[0228] This application provides an image processing apparatus that calculates the similarity between the pixel features of each pixel in the first target feature image and the pixel features of the same pixel in the feature difference reconstructed image through a mutual attention mechanism, thereby obtaining the defect probability score corresponding to each pixel. This improves the accuracy of the defect probability score calculation and lays the foundation for improving the location of defects in the target detection image.

[0229] In this application Figure 21 In an optional embodiment of the image processing apparatus provided in the corresponding embodiment, the defect probability score calculation module 150 is further configured to:

[0230] Using the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value, the pixel features corresponding to each pixel in the feature difference reconstructed image are processed through a mutual attention mechanism to obtain the target pixel features.

[0231] The defect probability score of each pixel in the target detection image is calculated based on the characteristics of the target pixel.

[0232] This application provides an image processing apparatus that uses the pixel features corresponding to each pixel in a first target feature image as the key and the pixel features corresponding to each pixel in a feature difference reconstructed image as the value. Through a mutual attention mechanism, it calculates the similarity between the pixel features of each pixel in the first target feature image and the pixel features of the same pixel in the feature difference reconstructed image, thereby obtaining a defect probability score corresponding to each pixel. This improves the accuracy of the defect probability score calculation and lays the foundation for improving the location of defects in target detection images.

[0233] In this application Figure 21 In an optional embodiment of the image processing apparatus provided in the corresponding embodiment, the defect probability score calculation module 150 is further configured to:

[0234] Using the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value, the first pixel feature is extracted from the pixel features corresponding to each pixel in the feature difference reconstructed image through a mutual attention mechanism.

[0235] Obtain the weight of the pixel corresponding to the feature of the first pixel.

[0236] The target pixel features are calculated based on the features of the first pixel and the weights corresponding to the pixels.

[0237] This application provides an image processing apparatus that uses the pixel features corresponding to each pixel in a first target feature image as the key and the pixel features corresponding to each pixel in a feature difference reconstructed image as the value. Through a mutual attention mechanism, it calculates the similarity between the pixel features of each pixel in the first target feature image and the pixel features of the same pixel in the feature difference reconstructed image, thereby obtaining a defect probability score corresponding to each pixel. This improves the accuracy of the defect probability score calculation and lays the foundation for improving the location of defects in target detection images.

[0238] In this application Figure 21 In an optional embodiment of the image processing apparatus provided in the corresponding embodiment, the feature comparison module 120 is further configured to:

[0239] The first feature extraction subnetwork in the Siamese network is used to extract features from the target detection image to generate a second target feature image.

[0240] The template image is used to extract features from the template image by using the second feature extraction subnetwork in the Siamese network, thereby generating a template feature image.

[0241] The feature images of the second target and the template are compared to generate a feature difference image.

[0242] This application provides an image processing device that adopts a model based on a three-branch network structure, wherein two branches are Siamese networks and one branch is a mutual attention mechanism network. The design of the three-branch network structure can not only have the advantages of the mainstream Siamese network design, but also avoid the problem of poor parameter adaptability caused by weight sharing in Siamese networks. This allows the branch responsible for defect localization to be more specialized, and achieves more accurate identification of defect regions by dynamically checking the feature differences between Siamese networks.

[0243] In this application Figure 21 In an optional embodiment of the image processing apparatus provided in the corresponding embodiment, the feature reconstruction module 130 is further configured to:

[0244] Arrange the target positions of each pixel and its corresponding P pixels in the feature difference image to generate a feature difference reconstructed image.

[0245] Where P is an integer greater than 1.

[0246] This application provides an image processing apparatus that, in order to distinguish between real and non-real defects, reconstructs the pixel points in the feature difference image output by a Siamese network to generate a reconstructed feature difference image. This dynamically extracts the truly meaningful difference regions from the differences between the two, thereby filtering out non-real defects. Furthermore, a module based on a mutual attention mechanism adaptively extracts discriminative regions from the feature difference, effectively avoiding interference from normal abnormal changes such as dirt caused by the production process, thus achieving high accuracy in defect detection.

[0247] In this application Figure 21 In an optional embodiment of the image processing apparatus provided in the corresponding embodiment, the defect location determination module 160 is further configured to:

[0248] The target detection image is processed by L convolutional networks with different dilation rates to obtain L convolutional feature images.

[0249] Where L is an integer greater than 1.

[0250] The L convolutional feature images are concatenated to generate a stitched image.

[0251] Based on the stitched image, determine the pixel points corresponding to copper wire defects in the target detection image.

[0252] The location of the defect in the target detection image is determined based on the pixel corresponding to the copper wire defect in the target detection image and the defect probability score of each pixel in the target detection image.

[0253] This application provides an image processing device that, based on the horizontal and vertical wiring characteristics on a PCB, uses an edge-sensitive change prediction module composed of L convolutional networks with different void ratios to assist in identifying defect areas using contextual information, thereby achieving more accurate location of defects.

[0254] In this application Figure 21 In an optional embodiment of the image processing apparatus provided in the corresponding embodiment, the feature comparison module 120 is further configured to:

[0255] Based on the first pyramid feature extraction subnetwork in the Siamese network, M feature extractions are performed on the target detection image to generate M second target feature images.

[0256] The pyramid feature extraction subnetwork consists of M feature extraction layers, where the input of each feature layer is the output of the previous feature layer, and M is an integer greater than or equal to 1.

[0257] Based on the second pyramid feature extraction subnetwork in the Siamese network, M feature extractions are performed on the template image to generate M template feature images.

[0258] Based on the feature comparison between the M second target feature images and the M template feature images, M feature difference images are generated.

[0259] This application provides an image processing apparatus in which a pyramid-structured feature extraction subnetwork in a Siamese network performs feature extraction operations on the target detection image and the template image in parallel. By subtracting the output of the second target feature image and the template feature image at each stage, feature difference images are generated for each stage. Performing feature comparison at each stage can simultaneously take into account differences at the fine-grained detail level and differences at the high-level semantic level, thereby enabling more comprehensive identification of various defects and improving the accuracy of defect localization in the target detection image.

[0260] In this application Figure 21 In an optional embodiment of the image processing apparatus provided in the corresponding embodiment, the feature extraction module 140 is further configured to perform M feature extractions on the target detection image to obtain M first target feature images.

[0261] In this process, the input for each feature extraction is the output of the previous feature extraction.

[0262] The defect probability score calculation module 150 is also used for:

[0263] The pixel features of the M first target feature images and the pixel features of the M feature difference reconstructed images are processed by a mutual attention mechanism network to obtain the M initial defect probability scores corresponding to each pixel in the target detection image.

[0264] Based on the M initial defect probability scores corresponding to each pixel in the target detection image, determine the defect probability score of each pixel in the target detection image.

[0265] This application provides an image processing apparatus that adaptively extracts discriminative regions from feature differences using a pyramid-structured feature extraction network. This design can effectively avoid interference from normal abnormal changes such as dirt caused by the production process, thereby improving the accuracy of locating defects in target detection images.

[0266] The image processing model training device in this application is described in detail below. Please refer to [link / reference]. Figure 22 . Figure 19 This is a schematic diagram of one embodiment of the image processing apparatus 20 in this application. The image processing model training apparatus 20 includes:

[0267] The training image acquisition module 210 is used to acquire training detection images and training template images, wherein the training detection images and training template images correspond to the same type of printed circuit board.

[0268] The training feature comparison module 220 is used to compare the features of the training detection image and the training template image based on the Siamese network, and generate a training feature difference image.

[0269] The training feature recombination module 230 is used to reconstruct the training feature difference image to obtain a training feature difference reconstructed image. Here, training feature recombination refers to rearranging the positions of pixels in the training feature difference image.

[0270] The training feature extraction module 240 is used to extract features from the training detection image to obtain the first training feature image;

[0271] The training defect probability score calculation module 250 is used to process the pixel features of the first training feature image and the pixel features of the reconstructed image by the training feature difference through a mutual attention mechanism network to obtain the defect probability score of each pixel in the training detection image. The defect probability score is used to characterize the similarity of pixels at the same position.

[0272] The training defect location determination module 260 is used to determine the predicted location of the defect in the training detection image based on the defect probability score of each pixel in the training detection image.

[0273] The loss function calculation module 270 is used to generate a loss function based on the predicted location of the defect in the training detection image and the location label of the defect in the training detection image;

[0274] The model training module 280 is used to train the Siamese network and the mutual attention mechanism network according to the loss function.

[0275] This application provides an image processing model training device. The image processing model adopts a three-branch network structure. Two branches of the Siamese network are trained using a weight-sharing mode, primarily for extracting deep features. The third branch is trained independently based on the differences in feature outputs between the first two branches at each stage. This avoids the poor parameter adaptability problem caused by shared weights in Siamese networks, enabling the third branch, responsible for defect localization, to achieve more accurate defect region identification by dynamically checking feature differences between the Siamese networks. Practical experience shows that this application significantly improves PCB defect detection. Furthermore, compared to methods that directly detect defects, it effectively reduces the need for training samples, thus facilitating rapid algorithm iteration in practical applications.

[0276] Figure 23This is a schematic diagram of a server structure provided in an embodiment of this application. The server 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations stored in the storage media 330 on the server 300.

[0277] Server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0278] The steps performed by the server in the above embodiments can be based on this Figure 23 The server structure shown.

[0279] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0280] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0281] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0282] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0283] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0284] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Acquire a target detection image and a template image, wherein the target detection image and the template image correspond to the same type of printed circuit board; Based on the Siamese network, feature comparison is performed between the target detection image and the template image to generate a feature difference image; The feature difference image is reconstructed to obtain a feature difference reconstructed image, wherein the feature reconstruction refers to rearranging the positions of the pixels in the feature difference image; Feature extraction is performed on the target detection image to obtain a first target feature image; The pixel features of the first target feature image and the pixel features of the feature difference reconstructed image are processed by a mutual attention mechanism network to obtain the defect probability score of each pixel in the target detection image, wherein the defect probability score is used to characterize the similarity of pixels at the same position. The location of the defect in the target detection image is determined based on the defect probability score of each pixel in the target detection image.

2. The image processing method as described in claim 1, characterized in that, The step of processing the pixel features of the first target feature image and the pixel features of the reconstructed image using a mutual attention mechanism network to obtain the defect probability score of each pixel in the target detection image includes: Obtain the pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the feature difference reconstructed image; The pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the feature difference reconstructed image are processed by the mutual attention mechanism to obtain the defect probability score of each pixel in the target detection image.

3. The image processing method as described in claim 2, characterized in that, The step of processing the pixel features corresponding to each pixel in the first target feature image and the pixel features corresponding to each pixel in the feature difference reconstructed image through a mutual attention mechanism to obtain the defect probability score of each pixel in the target detection image includes: Using the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value, the pixel features corresponding to each pixel in the feature difference reconstructed image are processed through a mutual attention mechanism to obtain the target pixel features. The defect probability score of each pixel in the target detection image is calculated based on the target pixel features.

4. The image processing method as described in claim 3, characterized in that, The step of processing the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value through a mutual attention mechanism to obtain the target pixel features includes: Using the pixel features corresponding to each pixel in the first target feature image as the key and the pixel features corresponding to each pixel in the feature difference reconstructed image as the value, the first pixel feature is extracted from the pixel features corresponding to each pixel in the feature difference reconstructed image through a mutual attention mechanism. Obtain the weight of the pixel corresponding to the feature of the first pixel; The target pixel features are calculated based on the features of the first pixel and the weights corresponding to the pixels.

5. The image processing method as described in claim 1, characterized in that, The step of comparing the features of the target detection image and the template image based on the Siamese network to generate a feature difference image includes: The target detection image is feature extracted based on the first feature extraction subnetwork in the Siamese network to generate a second target feature image; The template image is used to extract features based on the second feature extraction subnetwork in the Siamese network to generate a template feature image; The second target feature image and the template feature image are compared to generate a feature difference image.

6. The image processing method as described in claim 1, characterized in that, The step of performing feature reconstruction on the feature difference image to obtain a feature difference reconstructed image includes: The target positions of each pixel and its corresponding P pixels in the feature difference image are arranged to generate a feature difference reconstructed image, where P is an integer greater than 1.

7. The image processing method as described in claim 1, characterized in that, Determining the location of the defect in the target detection image based on the defect probability score of each pixel in the target detection image includes: The target detection image is processed by L convolutional networks with different dilation rates to obtain L convolutional feature images, where L is an integer greater than 1; The L convolutional feature images are concatenated to generate a concatenated image; Based on the stitched image, determine the pixel points corresponding to copper wire defects in the target detection image; The location of the defect in the target detection image is determined based on the pixel corresponding to the copper wire defect in the target detection image and the defect probability score of each pixel in the target detection image.

8. The image processing method as described in claim 1, characterized in that, The step of comparing the features of the target detection image and the template image based on the Siamese network to generate a feature difference image includes: The target detection image is subjected to M feature extractions based on the first pyramid feature extraction subnetwork in the Siamese network to generate M second target feature images. The pyramid feature extraction subnetwork includes M feature extraction layers, and the input of each feature layer is the output of the previous feature layer. M is an integer greater than or equal to 1. The template image is subjected to M feature extraction operations based on the second pyramid feature extraction subnetwork in the twin network to generate M template feature images; Based on the feature comparison between the M second target feature images and the M template feature images, M feature difference images are generated.

9. The image processing method as described in claim 8, characterized in that, The step of extracting features from the target detection image to obtain a first target feature image includes: The target detection image is subjected to M feature extractions to obtain M first target feature images, wherein the input of each feature extraction is the output of the previous feature extraction; The step of processing the pixel features of the first target feature image and the pixel features of the reconstructed image using a mutual attention mechanism network to obtain the defect probability score of each pixel in the target detection image includes: The pixel features of the M first target feature images and the pixel features of the M feature difference reconstructed images are processed by a mutual attention mechanism network to obtain M initial defect probability scores corresponding to each pixel in the target detection image. Based on the M initial defect probability scores corresponding to each pixel in the target detection image, the defect probability score of each pixel in the target detection image is determined.

10. A method for training an image processing model, characterized in that, include: Acquire training detection images and training template images, wherein the training detection images and the training template images correspond to the same type of printed circuit board; Based on the Siamese network, feature comparison is performed between the training detection image and the training template image to generate a training feature difference image; The training feature difference image is reconstructed to obtain a training feature difference reconstructed image, wherein the training feature reconstruction refers to rearranging the positions of pixels in the training feature difference image; Feature extraction is performed on the training detection image to obtain the first training feature image; The pixel features of the first training feature image and the pixel features of the reconstructed image by the training feature difference are processed by a mutual attention mechanism network to obtain the defect probability score of each pixel in the training detection image, wherein the defect probability score is used to characterize the similarity of pixels at the same position. The predicted location of the defect in the training detection image is determined based on the defect probability score of each pixel in the training detection image; A loss function is generated based on the predicted location of the defect in the training detection image and the location label of the defect in the training detection image; The Siamese network and the mutual attention mechanism network are trained according to the loss function.

11. An image processing apparatus, characterized in that, include: An image acquisition module is used to acquire a target detection image and a template image, wherein the target detection image and the template image correspond to the same type of printed circuit board; The feature comparison module is used to compare the features of the target detection image and the template image based on the Siamese network to generate a feature difference image; The feature recombination module is used to reconstruct the feature difference image to obtain a feature difference reconstructed image, wherein the feature recombination refers to rearranging the positions of the pixels in the feature difference image; The feature extraction module is used to extract features from the target detection image to obtain a first target feature image; The defect probability score calculation module is used to process the pixel features of the first target feature image and the pixel features of the feature difference reconstructed image through a mutual attention mechanism network to obtain the defect probability score of each pixel in the target detection image, wherein the defect probability score is used to characterize the similarity of pixels at the same position. The defect location determination module is used to determine the location of the defect in the target detection image based on the defect probability score of each pixel in the target detection image.

12. An image processing model training device, characterized in that, include: The training image acquisition module is used to acquire training detection images and training template images, wherein the training detection images and the training template images correspond to the same type of printed circuit board; The training feature comparison module is used to compare the features of the training detection image and the training template image based on the Siamese network, and generate a training feature difference image. The training feature reorganization module is used to reorganize the training feature difference image to obtain a training feature difference reorganized image, wherein the training feature reorganization refers to rearranging the positions of pixels in the training feature difference image; The training feature extraction module is used to extract features from the training detection image to obtain a first training feature image; The training defect probability score calculation module is used to process the pixel features of the first training feature image and the pixel features of the training feature difference reconstructed image through a mutual attention mechanism network to obtain the defect probability score of each pixel in the training detection image, wherein the defect probability score is used to characterize the similarity of pixels at the same position. The training defect location determination module is used to determine the predicted location of the defect in the training detection image based on the defect probability score of each pixel in the training detection image; The loss function calculation module is used to generate a loss function based on the predicted location of the defect in the training detection image and the location label of the defect in the training detection image; The model training module is used to train the Siamese network and the mutual attention mechanism network according to the loss function.

13. A computer device, characterized in that, include: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, including executing the image processing method as described in any one of claims 1 to 9 or the image processing model training method as described in claim 10; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

14. A computer-readable storage medium comprising instructions, when executed on a computer, causing the computer to perform the image processing method as claimed in any one of claims 1 to 9 or the image processing model training method as claimed in claim 10.

15. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor using the image processing method as described in any one of claims 1 to 9 or the image processing model training method as described in claim 10.

Images