Circuit board defect detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a circuit board defect detection method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: extracting depth features and mode features of circuit board defects in a video frame image through a circuit board defect detection model, carrying out pixel-by-pixel affinity and mode-by-mode affinity processing on the video frame image, and carrying out circuit board defect identification on the video frame image based on a target defect depth feature map and fusion affinity features. According to the embodiment of the invention, depth feature compensation is carried out on the depth features through the extracted mode features of the circuit board, more comprehensive defect depth features are extracted, pixel-by-pixel and mode-by-mode affinity processing is carried out on the image, pixel-level and mode-level features under different time sequences are captured, and the image quality is improved. Subsequent circuit board defect identification on the video frame image based on the target defect depth feature map and the fusion affinity feature is facilitated, and the accuracy of circuit board defect detection is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly relates to a circuit board defect detection method and device, an electronic device, and a storage medium. Background Art

[0002] Currently, traditional circuit board defect detection methods usually extract features from a single-frame static image using a deep learning model (such as a convolutional neural network, CNN) for circuit board defect segmentation, or generate a depth map of the circuit board defect area by introducing a depth estimation module and combine it with the original circuit board image for defect localization. For example, in the industrial production process, for the circuit board defect detection task, a convolutional neural network extracts circuit board features from a single-frame circuit board image and segments the circuit board defect area from the circuit board features to achieve circuit board defect detection. However, this method can only detect static circuit board images and lacks the utilization of depth information, and it is easy to misdetect normal areas as defect areas. Although the depth information can be utilized through the depth estimation module, it is also easy to mislabel normal areas with large depth differences as defect areas, resulting in a low accuracy of circuit board defect detection. Therefore, how to improve the accuracy of circuit board defect detection has become an urgent problem to be solved. Summary of the Invention

[0003] The main purpose of the embodiments of this application is to propose a circuit board defect detection method and device, an electronic device, and a storage medium, aiming to improve the accuracy of circuit board defect detection.

[0004] To achieve the above object, in the first aspect of the embodiments of this application, a circuit board defect detection method is proposed, and the method includes:

[0005] Obtain a circuit board video of a target circuit board in industrial production and extract video frame images from the circuit board video;

[0006] Extract circuit board depth defect features from the video frame images through a pre-trained circuit board defect detection model to obtain an initial defect depth feature map;

[0007] Extract circuit board defect pattern features from the initial defect depth feature map to obtain a defect pattern feature map;

[0008] Fuse defect features of the defect pattern feature map and the initial defect depth feature map to obtain a target defect depth feature map;

[0009] Perform per-pixel affinity processing on the video frame images to obtain per-pixel affinity features;

[0010] Perform per-pattern affinity processing on the defect pattern feature map to obtain per-pattern affinity features;

[0011] Fuse the per-pixel affinity feature and the per-pattern affinity feature to obtain a fused affinity feature;

[0012] Based on the target defect depth feature map and the fused affinity feature, perform circuit board defect recognition on the video frame image to obtain the target defect data of the target circuit board.

[0013] In some embodiments, the extracting circuit board defect mode features from the initial defect depth feature map to obtain a defect mode feature map includes:

[0014] Extract features of different channel dimensions from the initial defect depth feature map to obtain a channel dimension feature map;

[0015] Perform defect mode feature recognition on the channel dimension feature map to obtain the defect mode feature map.

[0016] In some embodiments, the performing per-pixel affinity processing on the video frame image to obtain a per-pixel affinity feature includes:

[0017] Obtain the target frame image of the video frame image and obtain the pixel values of the target frame image;

[0018] Extract the adjacent frame image of the target frame image from the video frame image and obtain the pixel values of the adjacent frame image;

[0019] Extract features from the pixel values of the target frame image to obtain a target frame image feature;

[0020] Extract features from the pixel values of the adjacent frame image to obtain an adjacent frame image feature;

[0021] Perform pixel feature integration on the target frame image features and the adjacent frame image features of all video frames one by one to obtain the per-pixel affinity feature.

[0022] In some embodiments, the fusing the per-pixel affinity feature and the per-pattern affinity feature to obtain a fused affinity feature includes:

[0023] Obtain the pixel feature weight of the per-pixel affinity feature and obtain the pattern feature weight of the per-pattern affinity feature;

[0024] Determine an enhanced per-pixel affinity feature according to the pixel feature weight and the corresponding per-pixel affinity feature;

[0025] Determine the enhanced per-pattern affinity feature according to the pattern feature weight and the corresponding per-pattern affinity feature;

[0026] Determine the fused affinity feature according to the enhanced per-pixel affinity feature and the enhanced per-pattern affinity feature.

[0027] In some embodiments, the target defect data includes a target defect category and a target defect location. The method for identifying circuit board defects in the video frame image based on the target defect depth feature map and the fused affinity feature to obtain the target defect data of the target circuit board includes:

[0028] Identify the target defect category in the video frame image based on the target defect depth feature map and the fused affinity feature to obtain the target defect category;

[0029] Identify the target defect location in the video frame image based on the target defect category to obtain the target defect location;

[0030] Determine the target defect category and the target defect location as the target defect data of the target circuit board.

[0031] In some embodiments, the method for extracting the initial defect depth feature map of the circuit board depth defect from the video frame image by using a pre-trained circuit board defect detection model includes:

[0032] Obtain the image pixel values of the video frame image;

[0033] Perform defect depth convolution processing on each frame of the image pixel values through the pre-trained circuit board defect detection model to obtain the initial defect depth feature map.

[0034] In some embodiments, before extracting the initial defect depth feature map of the circuit board depth defect from the video frame image by using a pre-trained circuit board defect detection model, the method further includes:

[0035] Obtain the original circuit board defect detection model and the training data set; the training data set includes a plurality of training video frame images and the true defect data corresponding to each training video frame image;

[0036] Extract the circuit board depth defect feature map from the plurality of training video frame images through the original circuit board defect detection model to obtain the training defect depth feature map;

[0037] Extract the circuit board defect pattern feature map from the training defect depth feature map to obtain the training defect pattern feature map;

[0038] Perform defect feature fusion on the training defect pattern feature map and the training defect depth feature map to obtain a target training defect depth feature map;

[0039] Perform per-pixel affinity processing on multiple training video frame images to obtain training per-pixel affinity features;

[0040] Perform per-pattern affinity processing on the training defect pattern feature map to obtain training per-pattern affinity features;

[0041] Perform affinity feature fusion on the training per-pixel affinity features and the training per-pattern affinity features to obtain fused training affinity features;

[0042] Perform circuit board defect prediction on multiple training video frame images based on the training defect depth feature map and the fused training affinity features to obtain predicted defect data;

[0043] Calculate the loss value of the predicted defect data and the true defect data according to a preset target loss function;

[0044] Update the model parameter weights of the original circuit board defect detection model based on the loss value, and return to perform circuit board depth defect feature extraction on multiple training video frame images through the original circuit board defect detection model until the original circuit board defect detection model meets the preset training conditions to obtain a pre-trained circuit board defect detection model.

[0045] To achieve the above object, a second aspect of the embodiments of the present application proposes a circuit board defect detection device, and the device includes:

[0046] An image extraction module, configured to obtain a circuit board video of a target circuit board in industrial production and extract video frame images from the circuit board video;

[0047] A depth defect feature extraction module, configured to perform circuit board depth defect feature extraction on the video frame images through a pre-trained circuit board defect detection model to obtain an initial defect depth feature map;

[0048] A defect pattern feature extraction module, configured to perform circuit board defect pattern feature extraction on the initial defect depth feature map to obtain a defect pattern feature map;

[0049] A defect feature fusion module, configured to perform defect feature fusion on the defect pattern feature map and the initial defect depth feature map to obtain a target defect depth feature map;

[0050] A per-pixel affinity processing module, configured to perform per-pixel affinity processing on the video frame images to obtain per-pixel affinity features;

[0051] A per - mode affinity processing module for performing per - mode affinity processing on the defective mode feature map to obtain per - mode affinity features;

[0052] An affinity feature fusion module for performing affinity feature fusion on the per - pixel affinity features and the per - mode affinity features to obtain fused affinity features;

[0053] A circuit board defect recognition module for performing circuit board defect recognition on the video frame image based on the target defect depth feature map and the fused affinity features to obtain target defect data of the target circuit board.

[0054] To achieve the above object, a third aspect of the embodiments of the present application proposes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect above is implemented.

[0055] To achieve the above object, a fourth aspect of the embodiments of the present application proposes a computer - readable storage medium, the computer - readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0056] The circuit board defect detection method and device, electronic device, and storage medium proposed in this application obtain a circuit board video of a target circuit board in industrial production and extract video frame images from the circuit board video; extract circuit board depth defect features from the video frame images through a pre-trained circuit board defect detection model to obtain an initial defect depth feature map; extract circuit board defect pattern features from the initial defect depth feature map to obtain a defect pattern feature map; perform defect feature fusion on the defect pattern feature map and the initial defect depth feature map to obtain a target defect depth feature map; perform per-pixel affinity processing on the video frame images to obtain per-pixel affinity features; perform per-pattern affinity processing on the defect pattern feature map to obtain per-pattern affinity features; perform affinity feature fusion on the per-pixel affinity features and the per-pattern affinity features to obtain fused affinity features; and perform circuit board defect recognition on the video frame images based on the target defect depth feature map and the fused affinity features to obtain target defect data of the target circuit board. First, this application extracts defect depth features from video frame images through a circuit board defect detection model and extracts circuit board defect pattern features from the initial defect depth feature map, which can accurately extract the depth features and pattern features of the circuit board from the video frame images and generate a more comprehensive and accurate target defect depth feature map. Second, by performing per-pixel affinity and per-pattern affinity processing on the video frame images, more comprehensive fused affinity features are generated, which can capture pixel-level and pattern-level circuit board defect information and facilitate improving the accuracy of circuit board detection in the subsequent process. Finally, by performing circuit board defect recognition on the video frame images based on the target defect depth feature map and the fused affinity features, more accurate defect detection can be performed by combining depth defect features and affinity features, significantly improving the accuracy of circuit board defect detection. Description of the Drawings

[0057] Figure 1 is a flowchart of the circuit board defect detection method provided by an embodiment of this application;

[0058] Figure 2 is another flowchart of the circuit board defect detection method provided by an embodiment of this application;

[0059] Figure 3 is Figure 1 a flowchart of step S102 in

[0060] Figure 4 is Figure 1 a flowchart of step S103 in

[0061] Figure 5 is Figure 1 a flowchart of step S105 in

[0062] Figure 6 is Figure 1The flowchart of step S107 in

[0063] Figure 7 is Figure 1 The flowchart of step S108 in

[0064] Figure 8 The application example diagram of the circuit board defect detection method provided by the embodiment of the present application;

[0065] Figure 9 The structural schematic diagram of the circuit board defect detection device provided by the embodiment of the present application;

[0066] Figure 10 The hardware structural schematic diagram of the electronic device provided by the embodiment of the present application. Specific implementation manners

[0067] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0068] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the order in the flowchart. Terms such as "first" and "second" in the specification, claims and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0070] First, several nouns involved in the present application are analyzed:

[0071] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. Artificial intelligence is a branch of computer science. It attempts to understand the essence of intelligence and produce an intelligent machine that can respond in a way similar to human intelligence. The research in this field includes robots, speech recognition, image recognition, natural language processing, and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. It also refers to the theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0072] The embodiments of this application provide a method and device for detecting circuit board defects, an electronic device, and a storage medium, aiming to improve the accuracy of circuit board defect detection.

[0073] The method and device for detecting circuit board defects, the electronic device, and the storage medium provided by the embodiments of this application will be specifically described through the following embodiments. First, the method for detecting circuit board defects in the embodiments of this application will be described.

[0074] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Among them, Artificial Intelligence (AI) is the theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0075] The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0076] The circuit board defect detection method provided by the embodiments of this application relates to the field of artificial intelligence technology. The circuit board defect detection method provided by the embodiments of this application can be applied to a terminal, can also be applied to a server side, or can be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the circuit board defect detection method, etc., but is not limited to the above forms.

[0077] This application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0078] Figure 1 is an optional flowchart of the circuit board defect detection method provided by the embodiments of this application, Figure 1 The method in may include but is not limited to steps S101 to S108.

[0079] Step S101, obtain a circuit board video of a target circuit board in industrial production, and extract video frame images from the circuit board video.

[0080] Step S102, perform circuit board depth defect feature extraction on the video frame images through a pre-trained circuit board defect detection model to obtain an initial defect depth feature map.

[0081] Step S103, perform circuit board defect mode feature extraction on the initial defect depth feature map to obtain a defect mode feature map.

[0082] Step S104: Perform defect feature fusion on the defect pattern feature map and the initial defect depth feature map to obtain the target defect depth feature map.

[0083] Step S105: Perform per-pixel affinity processing on the video frame image to obtain per-pixel affinity features.

[0084] Step S106: Perform per-pattern affinity processing on the defect pattern feature map to obtain per-pattern affinity features.

[0085] Step S107: Perform affinity feature fusion on the per-pixel affinity features and the per-pattern affinity features to obtain fused affinity features.

[0086] Step S108: Based on the target defect depth feature map and the fused affinity features, perform circuit board defect recognition on the video frame image to obtain the target defect data of the target circuit board.

[0087] In the steps S101 to S108 illustrated in the embodiments of the present application, by acquiring the circuit board video of the target circuit board in industrial production and extracting the video frame image from the circuit board video; by using the pre-trained circuit board defect detection model to extract the circuit board depth defect features from the video frame image to obtain the initial defect depth feature map; performing circuit board defect pattern feature extraction on the initial defect depth feature map to obtain the defect pattern feature map; performing defect feature fusion on the defect pattern feature map and the initial defect depth feature map to obtain the target defect depth feature map; performing per-pixel affinity processing on the video frame image to obtain per-pixel affinity features; performing per-pattern affinity processing on the defect pattern feature map to obtain per-pattern affinity features; performing affinity feature fusion on the per-pixel affinity features and the per-pattern affinity features to obtain fused affinity features; based on the target defect depth feature map and the fused affinity features, performing circuit board defect recognition on the video frame image to obtain the target defect data of the target circuit board. First, the present application can accurately extract the depth features and pattern features of the circuit board from the video frame image by using the circuit board defect detection model to extract the defect depth features from the video frame image and performing circuit board defect pattern feature extraction on the initial defect depth feature map, generating a more comprehensive and accurate target defect depth feature map; secondly, by performing per-pixel affinity and per-pattern affinity processing on the video frame image, more comprehensive fused affinity features are generated, which can capture pixel-level and pattern-level circuit board defect information, facilitating the subsequent improvement of the accuracy of circuit board detection; finally, by performing circuit board defect recognition on the video frame image based on the target defect depth feature map and the fused affinity features, more accurate defect detection can be performed by combining the depth defect features and the affinity features, significantly improving the accuracy of circuit board defect detection.

[0088] In step S101 of some embodiments, specifically, the circuit board video can be captured by an industrial camera installed on the production line. The circuit board video records the dynamic manufacturing process of the circuit board on the production line.

[0089] Specifically, a video frame image refers to a static image of the circuit board at a target moment, which records the instant states such as details (e.g., solder joints, circuits, etc.), defects (e.g., point defects, line defects, and area defects, etc.), color, and texture of the circuit board.

[0090] Furthermore, video frame images can be extracted from the video at a fixed frame rate (e.g., 30 frames per second), and the extracted video frame images can be preprocessed to obtain preprocessed video frame images. Among them, the preprocessing can be video frame image cropping, using size scaling technology to scale the resolution of the video frame image to 256 or 512, and enhancing the contrast of the video frame image, etc.

[0091] Before step S102 of some embodiments, in the embodiments of the present application, by obtaining a pre-trained circuit board defect detection model, extracting the defect depth features and defect pattern features of the circuit board, more comprehensive and accurate target defect depth features can be generated, and by combining the extraction of pixel-wise affinity and pattern-wise affinity features, pixel-level and pattern-level circuit board defect information can be captured, and then circuit board defect recognition is performed based on the target defect depth features and the fused affinity features, improving the accuracy of circuit board defect detection. This process may include several sub-steps of training the circuit board defect detection model before.

[0092] Please refer to Figure 2 , in some embodiments, before step S102, the circuit board defect detection method further includes but is not limited to steps S201 to S210:

[0093] Step S201, obtain the original circuit board defect detection model and the training data set; the training data set includes multiple training video frame images and the corresponding true defect data for each training video frame image.

[0094] Step S202, extract the circuit board depth defect features from multiple training video frame images through the original circuit board defect detection model to obtain a training defect depth feature map.

[0095] Step S203, extract the circuit board defect pattern features from the training defect depth feature map to obtain a training defect pattern feature map.

[0096] Step S204, perform defect feature fusion on the training defect pattern feature map and the training defect depth feature map to obtain a target training defect depth feature map.

[0097] Step S205: Perform per-pixel affinity processing on multiple training video frame images to obtain training per-pixel affinity features.

[0098] Step S206: Perform per-pattern affinity processing on the training defect pattern feature map to obtain training per-pattern affinity features.

[0099] Step S207: Fuse the training per-pixel affinity features and the training per-pattern affinity features to obtain fused training affinity features.

[0100] Step S208: Based on the training defect depth feature map and the fused training affinity features, perform circuit board defect prediction on multiple training video frame images to obtain predicted defect data.

[0101] Step S209: Calculate the loss value of the predicted defect data and the true defect data according to a preset target loss function.

[0102] Step S210: Update the model parameter weights of the original circuit board defect detection model based on the loss value, and return to perform circuit board depth defect feature extraction on multiple training video frame images through the original circuit board defect detection model until the original circuit board defect detection model meets the preset training conditions, and obtain a pre-trained circuit board defect detection model.

[0103] In step S201 of some embodiments, specifically, the training dataset includes multiple training video frame images and the true defect data corresponding to each training video frame image. Among them, the multiple training video frame images can be video frame images extracted from a training circuit board video, and the true defect data includes true defect depth labels, true pattern feature labels, and true defect labels.

[0104] Furthermore, the true defect depth label refers to the actual depth of the defect on the circuit board surface; the true pattern feature label refers to feature labels such as the actual defect category, size, or position of the circuit board; the true defect label refers to the actual defect category and the actual defect area on the circuit board.

[0105] Specifically, the original circuit board defect detection model is a neural network model composed of a depth estimation module (Depth-Anything), a pattern decoder, an affinity module, and a defect detection module; among them, the depth estimation module is used to extract the initial defect depth feature map of the circuit board, the pattern decoder is used to extract the defect pattern features of the circuit board, and the affinity module is used to extract the per-pixel affinity features and per-pattern affinity features of the circuit board; the defect detection module is used to detect the defect category and defect area of the circuit board.

[0106] In step S202 of some embodiments, specifically, the training defect depth feature map refers to the circuit board defect depth feature map predicted by the model.

[0107] Specifically, the depth estimation function of the depth estimation module can be used to predict the defect depth of the circuit board to determine the training defect depth feature map.

[0108] In step S203 of some embodiments, specifically, the training defect pattern feature map refers to features such as the defect category, size, and location of the circuit board predicted by the model.

[0109] Specifically, the training defect depth feature map is input into the pattern decoder for pattern feature extraction to output a pattern feature tensor containing multiple channels; where each channel represents different defect pattern features, such as the category, size, and location of the defect.

[0110] Furthermore, the pattern features of all channels together form a multi-dimensional vector, and each dimension corresponds to different defect features. Through the extraction of the training defect pattern feature map, the error of the training defect depth feature map can be eliminated and the defect features of the circuit board can be accurately described.

[0111] In step S204 of some embodiments, specifically, the target training defect depth feature map refers to the training depth feature map that fuses the training defect pattern feature map and the training defect depth feature map.

[0112] Specifically, feature splicing of the training defect pattern feature map and the training defect depth feature map can achieve defect feature fusion. Combining the defect pattern features to compensate for the training defect depth features can eliminate the feature error of the training defect depth feature map and additionally extract features related to the category, location, size, etc. of the circuit board defects in the image, effectively improving the accuracy of defect depth feature extraction.

[0113] In step S205 of some embodiments, specifically, the training per-pixel affinity feature is used to characterize the pixel similarity between adjacent frame images.

[0114] Specifically, the training video frame images of all video frames are input into the affinity module for per-pixel affinity processing to identify the difference between each pixel in the two adjacent frames before and after the same pixel in the image, thereby capturing the dynamic change features between multiple frames of images.

[0115] In step S206 of some embodiments, specifically, the training per-pixel affinity feature is used to characterize the pattern feature similarity between adjacent frame images.

[0116] Specifically, input the training video frame images of all video frames into the affinity module. For the pattern features in each frame image and the corresponding pattern features of the previous frame, use the pattern feature extraction function to extract the global features of consecutive video frame images, further calculate the dot product between consecutive frames and accumulate it to obtain the per-pattern affinity feature values of consecutive frame pattern features.

[0117] In this embodiment, by performing per-pattern affinity processing on the training defect pattern feature map, the global feature changes of the video frame image can be effectively captured, and the accuracy of pattern feature extraction for consecutive video frame images is improved.

[0118] In step S207 of some embodiments, specifically, the fused training affinity feature refers to the affinity feature that fuses the training per-pixel affinity feature and the training per-pattern affinity feature.

[0119] Specifically, obtain the fused training affinity feature by performing weighted summation on the training per-pixel affinity feature and the training per-pattern affinity feature.

[0120] In this embodiment, by fusing the training per-pixel affinity feature and the training per-pattern affinity feature, the fine-grained training per-pixel affinity feature and the global training per-pattern affinity feature can be effectively captured, and the accuracy of video frame image feature extraction is further improved.

[0121] In step S208 of some embodiments, specifically, the predicted defect data includes the predicted defect category and the predicted defect location.

[0122] Specifically, input the training defect depth feature map and the fused training affinity feature into the defect detection module to classify the defect category of the circuit board through the classifier, and generate a detection box based on the classifier to determine the specific location of the circuit board defect through the detection box.

[0123] In step S209 of some embodiments, specifically, the loss value is a quantitative index that measures the difference between the predicted defect data and the true defect data. The smaller its value, the closer the defect prediction of the circuit board defect detection model is to the real situation.

[0124] Specifically, the target loss function includes a depth reconstruction loss function, a pattern supervision loss function, and a detection loss function.

[0125] Further, before calculating the loss values of the predicted defect data and the true defect data according to the preset target loss function, the circuit board defect detection method further includes: obtaining the true defect depth label of the video frame image, and determining the depth reconstruction loss function according to the true defect depth label and the training defect depth feature map; obtaining the true pattern feature label of the video frame image, and determining the pattern supervision loss function according to the true pattern feature label and the training defect pattern feature map; obtaining the true defect label of the video frame image, and determining the detection loss function according to the true defect label and the predicted defect data; performing weighted summation on the depth reconstruction loss function, the pattern supervision loss function, and the detection loss function to obtain the target loss function.

[0126] Specifically, the depth reconstruction loss function can be determined by the following formula:

[0127]

[0128] where, L depth represents the depth reconstruction loss function, N represents the total number of pixels in the video frame image, DOM′(x,y) represents the training defect depth feature map of the image pixel value (x,y), DOM gt (x,y) represents the true defect depth label of the image pixel value (x,y), x represents the pixel abscissa value in the video frame image, and y represents the pixel ordinate value in the video frame image.

[0129] Specifically, the pattern supervision loss function can be determined by the following formula:

[0130]

[0131] where, L pattern represents the pattern supervision loss function, N represents the total number of pixels in the video frame image,

[0132] P k ′(x,y) represents the training defect pattern feature map of the image pixel value (x,y) in the k-th pattern channel,

[0133] P k,gt (x,y) represents the true pattern feature label of the image pixel value (x,y), x represents the pixel abscissa value in the video frame image, and y represents the pixel ordinate value in the video frame image.

[0134] Specifically, the detection loss function can be determined by the following formula:

[0135]

[0136] where, L det represents the detection loss function, N represents the total number of pixels in the video frame image, Det gt(x, y) represents the true defect label of the image pixel value (x, y), Det(x, y) represents the predicted defect data of the image pixel value (x, y), x represents the pixel abscissa value in the video frame image, and y represents the pixel ordinate value in the video frame image.

[0137] Specifically, the target loss function can be determined by the following formula:

[0138] L total = λ1L depth + λ2L pattern + λ3L det

[0139] Wherein, L total represents the target loss function, λ1 represents the weight hyperparameter of the depth reconstruction loss function, L depth represents the depth reconstruction loss function, λ2 represents the weight hyperparameter of the pattern supervision loss function, L pattern represents the pattern supervision loss function, λ3 represents the weight hyperparameter of the detection loss function, and L det represents the detection loss function.

[0140] Specifically, by calculating the loss value of the predicted defect data and the true defect data according to the preset target loss function, the difference between the predicted defect data and the true defect data of the model can be measured, and through the calculation of the loss value, it is convenient to determine the model parameter weights subsequently, realize the update of the model parameter weights, reduce the prediction error of the circuit board defect detection, and help improve the accuracy of the circuit board defect detection.

[0141] In step S210 of some embodiments, specifically, after the loss value calculation is completed, the circuit board defect detection model enters the backpropagation stage. In this stage, the gradient of the loss function is calculated, and according to the calculated gradient, the parameter weights of the model are updated, so that the circuit board defect detection model can learn how to reduce the prediction error of the circuit board defect detection and gradually improve the accuracy of the circuit board defect detection.

[0142] Specifically, the preset training condition can be that the loss value is less than the preset loss threshold.

[0143] Specifically, through the process of updating the circuit board defect detection model parameters, the circuit board defect detection model learns how to extract comprehensive circuit board defect features from the training dataset and learns how to perform accurate circuit board defect detection according to the circuit board defect features, which helps to improve the accuracy of the circuit board defect detection model in performing circuit board defect detection.

[0144] Please refer to Figure 3 , in some embodiments, step S102 includes but is not limited to steps S301 to S302:

[0145] Step S301: Obtain the image pixel values of the video frame images.

[0146] Step S302: Perform defect depth convolution processing on the pixel values of each frame of image through a pre-trained circuit board defect detection model to obtain an initial defect depth feature map.

[0147] In step S301 of some embodiments, specifically, the image pixel values include the grayscale or color information of each pixel point in each video frame image.

[0148] In step S302 of some embodiments, specifically, the pre-trained circuit board defect detection model is a neural network model composed of a depth estimation module, a pattern decoder, an affinity module, and a defect detection module and trained to completion.

[0149] Specifically, the depth estimation module is based on a Convolutional Neural Network (CNN), and this depth estimation module is used to extract the initial defect depth feature map of the video frame image.

[0150] Specifically, the initial defect depth feature map is used to describe the depth of the circuit board defect in the video frame image, that is, the relative depth of the defect on the surface of the circuit board.

[0151] Specifically, the defect depth convolution processing can be implemented through the following formula:

[0152] DOM(x,y) = f depth (I t (x,y))

[0153] where DOM(x,y) represents the initial defect depth feature map of the image pixel value (x,y), I t (x,y) represents the pixel value (x,y) of the t-th frame of image, usually an RGB value or a grayscale value, f depth represents a depth estimation function, which is used to extract the depth defect features of the video frame image, x represents the abscissa value of the pixel in the video frame image, y represents the ordinate value of the pixel in the video frame image, and t represents the time index of the target frame.

[0154] In this embodiment, by performing defect depth convolution processing on the pixel values of each frame of image through a pre-trained circuit board defect detection model, the depth feature information related to the circuit board defect can be extracted from the video frame image, providing a key intermediate result for subsequent defect analysis and recognition.

[0155] Please refer to Figure 4 , in some embodiments, step S103 includes but is not limited to steps S401 to S402:

[0156] Step S401: Extract features of different channel dimensions from the initial defect depth feature map to obtain a channel dimension feature map.

[0157] Step S402: Identify defect pattern features from the channel dimension feature map to obtain a defect pattern feature map.

[0158] In step S401 of some embodiments, specifically, the channel dimension feature map refers to different dimensional features extracted on different channels of the feature map. Each channel may focus on capturing different aspects of the image, such as image edges, image texture, image color, circuit board defect categories, defect sizes, and defect positions.

[0159] Specifically, the channel dimension feature map can describe the characteristics of the circuit board surface from multiple angles and provide rich information for subsequent circuit board defect identification.

[0160] For example, in circuit board defect detection, image edge features can be used to identify whether there are cracks on the circuit board, and image texture features can be used to distinguish normal textures from abnormal textures caused by defects.

[0161] In step S402 of some embodiments, specifically, since the multi-channel features extracted are further transformed into features that can be directly used for defect identification, which is the defect pattern feature map, the defect pattern feature map also includes pattern features such as image edges, image texture, image color, circuit board defect categories, defect sizes, and defect positions.

[0162] Furthermore, since the initial defect depth feature map may contain error features (such as complex image backgrounds or normal textures being misjudged as defects), in order to reduce the interference caused by error features, by introducing a pattern decoder, defect-specific pattern features that eliminate error features can be extracted.

[0163] Specifically, through the pattern decoder (g pattern ) the channel dimension feature map can be output as a pattern feature tensor containing multiple channels. Among them, each channel represents different defect pattern features.

[0164] Furthermore, defect pattern feature recognition can be achieved through the following formula:

[0165] P k (x,y) = g pattern (DOM(x,y))

[0166] Where P k (x,y) represents the defect pattern feature map of the image pixel value (x,y) in the k-th pattern channel; DOM(x,y) represents the initial defect depth feature map of the image pixel value (x,y), and g pattern represents the pattern decoder.

[0167] In this embodiment, by performing defect pattern feature recognition on the channel dimension feature map, it effectively avoids mislabeling normal regions with large depth differences as defect regions, improves the accuracy of extracting defect features in the video frame image, and helps to improve the accuracy of subsequent circuit board defect detection.

[0168] In step S104 of some embodiments, the target defect depth feature map refers to a depth feature map that fuses the defect pattern feature map and the initial defect depth feature map.

[0169] Specifically, the defect feature fusion can be achieved through the following formula:

[0170] PC_DOM(x,y) = h(Concat(DOM(x,y),P k (x,y)))

[0171] where PC_DOM(x,y) represents the target defect depth feature map; h represents the fusion module, usually a convolutional layer, which is used to further process the fused features; Concat is the splicing operation, which is used to combine the defect pattern feature map and the initial defect depth feature map in the channel dimension; DOM(x,y) represents the initial defect depth feature map of the image pixel value (x,y); P k (x,y) represents the defect pattern feature map of the image pixel value (x,y) in the k-th mode channel.

[0172] In an alternative embodiment of the present application, if the video frame image is an image containing a noisy background, then when obtaining the target defect depth feature map, the fusion module h can be replaced with an enhanced fusion module h noise to generate a target defect depth feature map with significantly reduced background noise interference.

[0173] In this embodiment, by performing defect feature fusion on the defect pattern feature map and the initial defect depth feature map, it is possible to compensate the initial defect depth feature map by combining the defect pattern feature map and the initial defect depth feature map to generate a more comprehensive and accurate target defect depth feature map, overcoming the problems of complex backgrounds and mislabeling normal regions with large depth differences as defect regions in traditional methods, and further improving the accuracy of extracting circuit board defect features.

[0174] Please refer to Figure 5 , in some embodiments, step S105 includes but is not limited to steps S501 to S505:

[0175] Step S501, obtain the target frame image of the video frame image, and obtain the target frame image pixel value of the target frame image.

[0176] Step S502: Extract the adjacent frame images of the target frame image from the video frame images, and obtain the pixel values of the adjacent frame images of the adjacent frame images.

[0177] Step S503: Extract features from the target frame image pixel values to obtain the target frame image features.

[0178] Step S504: Extract features from the adjacent frame image pixel values to obtain the adjacent frame image features.

[0179] Step S505: Integrate the target frame image features and the adjacent frame image features of all video frames pixel by pixel to obtain the per-pixel affinity features.

[0180] In step S501 of some embodiments, specifically, the target frame image refers to a single frame image selected based on the target time point, and the target frame image pixel values contain the grayscale or color information of each pixel point in the target video frame image.

[0181] In step S502 of some embodiments, specifically, the adjacent frame images refer to the adjacent previous frame image or the adjacent subsequent frame image of the target frame image, and the number of adjacent frame images is determined based on the actual task of circuit board defect detection. This application does not limit the number of adjacent frame images.

[0182] Specifically, the adjacent frame images provide the state information before and after the target frame, which helps the subsequent model to capture the characteristics of the defect changing over time during circuit board defect detection.

[0183] Specifically, the adjacent frame image pixel values contain the grayscale or color information of each pixel point in the adjacent video frame image.

[0184] In step S503 of some embodiments, specifically, the target frame image features refer to the defect depth features corresponding to the target video frame image.

[0185] Specifically, the feature extraction of the target frame image pixel values can be achieved through a feature extraction function, which is usually composed of a CNN network.

[0186] In step S504 of some embodiments, specifically, the adjacent frame image features refer to the defect depth features corresponding to the adjacent video frame images.

[0187] Specifically, the feature extraction of the adjacent frame image pixel values can also be achieved through a feature extraction function.

[0188] In step S505 of some embodiments, specifically, the per-pixel affinity features are used to characterize the similarity of adjacent pixels between frames, reflect the change of each image pixel value over time, and help to identify the areas where circuit board defects change during the production process.

[0189] Specifically, the extraction of pixel-by-pixel affinity features can be achieved through the following formula:

[0190]

[0191] where I t (x, y) represents the target frame image pixel value (x, y) of the target frame image of the t-th frame, and I t-1 (x, y) represents the adjacent frame image pixel value (x, y) of the adjacent frame image of the (t - 1)-th frame. A pixel (x, y) represents the pixel-by-pixel affinity feature between the target frame image pixel value and the adjacent frame image pixel value. φ represents the feature extraction function, and T represents the total number of video frame images.

[0192] In this embodiment, by integrating the pixel features of the target frame image features and the adjacent frame image features of all video frames one by one, the dynamic change defect features at the fine-grained level of multiple-frame circuit board images can be captured, and the accuracy of pixel-by-pixel affinity feature extraction is improved.

[0193] In step S106 of some embodiments, the per-mode affinity feature is used to capture the similarity of inter-frame mode features to focus on the global feature changes of circuit board defects.

[0194] Specifically, in the affinity module, first, the defect mode feature maps in each frame image and their corresponding adjacent frame defect mode feature maps are respectively subjected to feature extraction, and the extracted features are dot-producted and accumulated to obtain the per-mode affinity feature of consecutive frames.

[0195] Furthermore, the per-mode affinity processing can be achieved through the following formula:

[0196]

[0197] where A pattern (k) represents the per-mode affinity feature of the defect mode feature map of the k-th channel, P t (k) represents the defect mode feature map of the k-th channel in the t-th frame, and P t-1 (k) represents the defect mode feature map of the k-th channel in the (t - 1)-th frame. ψ represents the mode feature extraction function, and T represents the total number of video frame images.

[0198] In this embodiment, by performing per-mode affinity processing on the defect mode feature maps, the global feature changes of circuit board defect modes in video frame images can be effectively captured, and the accuracy of mode feature extraction of consecutive video frame images is significantly improved.

[0199] Please refer to Figure 6, in some embodiments, step S107 includes but is not limited to steps S601 to S604:

[0200] Step S601, obtain the pixel feature weights of the per-pixel affinity features and obtain the pattern feature weights of the per-pattern affinity features.

[0201] Step S602, determine the enhanced per-pixel affinity features according to the pixel feature weights and the corresponding per-pixel affinity features.

[0202] Step S603, determine the enhanced per-pattern affinity features according to the pattern feature weights and the corresponding per-pattern affinity features.

[0203] Step S604, determine the fused affinity features according to the enhanced per-pixel affinity features and the enhanced per-pattern affinity features.

[0204] In step S601 of some embodiments, specifically, the pixel feature weights are used to characterize the importance of the pixel features of the per-pixel affinity features in the process of circuit board defect detection.

[0205] Specifically, the pattern feature weights are used to characterize the importance of the pattern features of the per-pattern affinity features in the process of circuit board defect detection.

[0206] For example, when detecting circuit board defects, if there are scratch defects on the circuit board, the pixel feature weights and pattern feature weights closer to the scratch area are larger because the probability of scratch defects appearing in this area is relatively high.

[0207] In step S602 of some embodiments, specifically, the enhanced per-pixel affinity features include the per-pixel affinity features and the pixel feature weights.

[0208] Specifically, the product of the pixel feature weights and the corresponding per-pixel affinity features can be determined as the enhanced per-pixel affinity features.

[0209] For example, in circuit board defect detection, the pixel feature weights reflect the contribution degree of each pixel to defect detection. The pixel weights in the areas where there may be defects such as scratches or cracks on the circuit board are relatively large, which can strengthen the key pixels for circuit board defect recognition.

[0210] In this embodiment, determining the enhanced per-pixel affinity features according to the pixel feature weights and the corresponding per-pixel affinity features not only retains the information about pixel changes in the per-pixel affinity features, but also strengthens the pixels that are more critical for circuit board defect recognition, facilitating subsequent improvement of the accuracy of circuit board defect detection.

[0211] In step S603 of some embodiments, specifically, the enhanced per-pattern affinity feature includes the per-pattern affinity feature and the pattern feature weight.

[0212] Specifically, the product of the pattern feature weight and the corresponding per-pattern affinity feature can be determined as the enhanced per-pixel affinity feature.

[0213] For example, in circuit board defect detection, the pattern feature weight reflects the contribution degree of each pattern to defect detection. For patterns that may have circuit board crack defect categories and patterns of crack defect regions, the pattern weights are relatively large, which can strengthen the key patterns for circuit board defect recognition.

[0214] In step S604 of some embodiments, specifically, the fused affinity feature includes the enhanced per-pixel affinity feature and the enhanced per-pattern affinity feature.

[0215] Specifically, by performing weighted summation on the enhanced per-pixel affinity feature and the enhanced per-pattern affinity feature, the fused affinity feature can be obtained.

[0216] Furthermore, the fused affinity feature can be determined by the following formula:

[0217] A final = α·A pixel + β·A pattern

[0218] Wherein, A final represents the fused affinity feature, A pixel represents the per-pixel affinity feature, A pattern represents the per-pattern affinity feature, α represents the pixel feature weight, which is used to control the contribution ratio of the per-pixel affinity feature in the fused affinity feature, and β represents the pattern feature weight, which is used to control the contribution ratio of the per-pattern affinity feature in the fused affinity feature.

[0219] In this embodiment, determining the fused affinity feature according to the enhanced per-pixel affinity feature and the enhanced per-pattern affinity feature can capture pixel-level and pattern-level circuit board defect information, realize the temporal dynamic information fusion between multi-frame image features, and help improve the accuracy of subsequent circuit board defect detection.

[0220] Please refer to Figure 7 , in some embodiments, step S108 includes but is not limited to steps S701 to S703:

[0221] Step S701, based on the target defect depth feature map and the fused affinity feature, perform circuit board defect category recognition on the video frame image to obtain the target defect category.

[0222] Step S702: Identify the defective positions on the circuit board in the video frame image based on the target defect category to obtain the target defective positions.

[0223] Step S703: Determine the target defect category and the target defective positions as the target defect data of the target circuit board.

[0224] In step S701 of some embodiments, specifically, the target defect data includes the target defect category and the target defective positions. Among them, the target defect category includes point defects, line defects, area defects, copper leakage, scratches, ink contamination, cracks, etc. of the circuit board.

[0225] Specifically, the target defective position refers to the specific position of the circuit board defect on the circuit board.

[0226] Specifically, input the target defect depth feature map and the fusion affinity feature into the defect detection module. Through the non-linear layer in the defect detection module, output the corresponding category probabilities for different circuit board defect categories, and select the defect category with the highest probability as the target defect category.

[0227] Furthermore, the target defect category can be determined by the following formula:

[0228] A detected (x,y) = f(PC_DOM(x,y), A final )

[0229] where A detected (x,y) represents the target defect category of the image pixel value (x,y), f represents the non-linear layer, PC_DOM(x,y) represents the target defect depth feature map of the image pixel value (x,y), and A final represents the fusion affinity feature.

[0230] In this embodiment, identifying the circuit board defect category in the video frame image based on the target defect depth feature map and the fusion affinity feature can perform more accurate circuit board defect detection by combining the depth defect feature and the fusion affinity feature, improving the accuracy of circuit board defect detection.

[0231] In step S702 of some embodiments, specifically, obtain the target defect category label of the target defect category, generate a detection box according to the target defect category label, perform defect threshold processing on the video frame image to determine the target defect area, and then segment the target defect area through the detection box to obtain the target defective positions.

[0232] Specifically, the target defective position can be determined by the following formula:

[0233] Seg(x,y) = Threshold(c(x,y))

[0234] Among them, Seg(x, y) represents the target defect position of the image pixel value (x, y), Threshold represents threshold processing, and c(x, y) represents the target defect category label of the image pixel value (x, y).

[0235] In this embodiment, by identifying the circuit board defect positions in the video frame images based on the target defect categories, it is possible to jointly use the depth defect features and the fused affinity features to identify the defect areas on the circuit board, avoiding mislabeling normal areas with large depth differences as defect areas, and significantly improving the accuracy of circuit board defect detection.

[0236] In step S703 of some embodiments, specifically, the identified target defect categories and target defect positions are combined to determine the target defect data on the circuit board. This target defect data includes not only the types of circuit board defects but also the precise positions of the defects on the circuit board, providing detailed data support for circuit board defect repair and quality control.

[0237] Please refer to Figure 8 , in some embodiments, Figure (a) shows the overall architecture of FTM-Net (Feature Fusion of Temporal and Multi-Modality Network). In this architecture, first, multiple video frame images containing t frames are used as the input of the circuit board defect detection model. The video frame images are subjected to DepthAnythingEncoder (depth encoding) operations to extract multiple frames of depth features F V , the extracted multiple frames of depth features F V are fused with image temporal features, etc., to obtain fused depth features Secondly, is subjected to DepthAnything Decoder (depth decoding) operations to extract the initial defect depth feature map DOM, and is subjected to Pattern Decoder (pattern decoding) operations to extract the defect pattern feature map Patterns. The DOM and Patterns feature maps are fused to obtain the target defect depth feature map (Pattern-Compensated DOM); finally, the target frame image x t and the target defect depth feature map are subjected to Segmentation Encoder (segmentation encoding) processing to extract the target frame image features Through the Point-wise Affinity module, the target frame image features and the multiple frames of depth features F VPerform pixel-wise feature extraction to obtain pixel-wise affinity features, and pass the target frame image features through a Pattern-wise Affinity module and multi-frame depth features F V Perform pattern-wise feature extraction to obtain pattern-wise affinity features, and then fuse the pixel-wise affinity features and the pattern-wise affinity features to obtain fused affinity features Perform a Segmentation Encoder operation for printed circuit board defect recognition to obtain predicted target defect data. Here, Frozen means that the network layer or parameters in the model are frozen during training, that is, the weights of the network layer or parameters will not be updated; Tuning means that the network layer or parameters of the model are adjusted or optimized during training, such as adjusting hyperparameters such as the model learning rate and weight initialization; Concatenate means feature concatenation; TemporalPooling means temporal pooling, which is used to extract features from time series data; Matrix Multiplication means matrix multiplication; Affinity Matrix means affinity matrix.

[0238] Furthermore, Figure (b) shows the implementation process of Point-wise Affinity. First, calculate the pixel-wise similarity between the multi-frame depth features F V and the target frame image features and generate pixel-wise affinity features using matrix multiplication This feature is used to enhance the input temporal features. Here, k represents the pattern channel, H represents the image height, W represents the image width, and C represents the image channel.

[0239] Furthermore, Figure (c) shows the implementation process of Pattern-wise Affinity. First, perform pattern feature extraction on the multi-frame depth features F V and the target frame image features and generate pattern-wise affinity features through matrix multiplication of the pattern features to further capture the inter-frame pattern changes. Here, k represents the pattern channel, H represents the image height, W represents the image width, C represents the image channel, and ψ represents the pattern feature extraction function.

[0240] In an embodiment of the present application, a circuit board video of a target circuit board in industrial production is obtained, and video frame images are extracted from the circuit board video; the pre-trained circuit board defect detection model is used to extract circuit board depth defect features from the video frame images to obtain an initial defect depth feature map; circuit board defect pattern features are extracted from the initial defect depth feature map to obtain a defect pattern feature map; defect feature fusion is performed on the defect pattern feature map and the initial defect depth feature map to obtain a target defect depth feature map; pixel-by-pixel affinity processing is performed on the video frame images to obtain pixel-by-pixel affinity features; pattern-by-pattern affinity processing is performed on the defect pattern feature map to obtain pattern-by-pattern affinity features; the pixel-by-pixel affinity features and the pattern-by-pattern affinity features are fused to obtain fused affinity features; based on the target defect depth feature map and the fused affinity features, circuit board defect recognition is performed on the video frame images to obtain target defect data of the target circuit board. First, the circuit board defect detection model is used to extract defect depth features from the video frame images, and circuit board defect pattern features are extracted from the initial defect depth feature map, which can accurately extract the depth features and pattern features of the circuit board from the video frame images and generate a more comprehensive and accurate target defect depth feature map. Second, by performing pixel-by-pixel affinity and pattern-by-pattern affinity processing on the video frame images, more comprehensive fused affinity features are generated, which can capture pixel-level and pattern-level circuit board defect information and facilitate improving the accuracy of circuit board detection in the follow-up. Finally, by performing circuit board defect recognition on the video frame images based on the target defect depth feature map and the fused affinity features, more accurate defect detection can be performed by combining depth defect features and affinity features, significantly improving the accuracy of circuit board defect detection.

[0241] Please refer to Figure 9 , an embodiment of the present application also provides a circuit board defect detection device, which can implement the above circuit board defect detection method. The device includes:

[0242] An image extraction module, configured to obtain a circuit board video of a target circuit board in industrial production and extract video frame images from the circuit board video;

[0243] A depth defect feature extraction module, configured to extract circuit board depth defect features from the video frame images through a pre-trained circuit board defect detection model to obtain an initial defect depth feature map;

[0244] A defect pattern feature extraction module, configured to extract circuit board defect pattern features from the initial defect depth feature map to obtain a defect pattern feature map;

[0245] A defect feature fusion module, configured to perform defect feature fusion on the defect pattern feature map and the initial defect depth feature map to obtain a target defect depth feature map;

[0246] A per-pixel affinity processing module for performing per-pixel affinity processing on a video frame image to obtain per-pixel affinity features;

[0247] A per-mode affinity processing module for performing per-mode affinity processing on a defect mode feature map to obtain per-mode affinity features;

[0248] An affinity feature fusion module for performing affinity feature fusion on the per-pixel affinity features and the per-mode affinity features to obtain fused affinity features;

[0249] A circuit board defect recognition module for performing circuit board defect recognition on the video frame image based on a target defect depth feature map and the fused affinity features to obtain target defect data of a target circuit board.

[0250] The specific implementation manner of this circuit board defect detection device is basically the same as the specific embodiments of the above circuit board defect detection method, and will not be elaborated here.

[0251] An embodiment of this application also provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above circuit board defect detection method is implemented. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0252] Please refer to Figure 10 , Figure 10 which schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0253] A processor 1001, which can be implemented by using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of this application;

[0254] A memory 1002, which can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1002 can store a processing system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1002, and the processor 1001 is called to execute the circuit board defect detection method of the embodiments of this application;

[0255] An input / output interface 1003 for implementing information input and output;

[0256] A communication interface 1004 for implementing communication interaction between this device and other devices, which can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0257] A bus 1005 for transmitting information between various components of the device (such as the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004);

[0258] Among them, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 achieve communication connections with each other inside the device through the bus 1005.

[0259] The embodiment of this application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned circuit board defect detection method is implemented.

[0260] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0261] The circuit board defect detection method, circuit board defect detection device, electronic device, and storage medium provided by the embodiments of the present application obtain a circuit board video of a target circuit board in industrial production and extract video frame images from the circuit board video; extract circuit board depth defect features from the video frame images through a pre-trained circuit board defect detection model to obtain an initial defect depth feature map; extract circuit board defect pattern features from the initial defect depth feature map to obtain a defect pattern feature map; perform defect feature fusion on the defect pattern feature map and the initial defect depth feature map to obtain a target defect depth feature map; perform per-pixel affinity processing on the video frame images to obtain per-pixel affinity features; perform per-pattern affinity processing on the defect pattern feature map to obtain per-pattern affinity features; perform affinity feature fusion on the per-pixel affinity features and the per-pattern affinity features to obtain a fused affinity feature; and perform circuit board defect recognition on the video frame images based on the target defect depth feature map and the fused affinity feature to obtain target defect data of the target circuit board. First, the present application extracts defect depth features from video frame images through a circuit board defect detection model and extracts circuit board defect pattern features from the initial defect depth feature map, which can accurately extract the depth features and pattern features of the circuit board from the video frame images and generate a more comprehensive and accurate target defect depth feature map. Second, by performing per-pixel affinity and per-pattern affinity processing on the video frame images, a more comprehensive fused affinity feature is generated, which can capture pixel-level and pattern-level circuit board defect information and facilitate improving the accuracy of circuit board detection in the subsequent stage. Finally, by performing circuit board defect recognition on the video frame images based on the target defect depth feature map and the fused affinity feature, more accurate defect detection can be performed by combining depth defect features and affinity features, significantly improving the accuracy of circuit board defect detection.

[0262] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0263] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0264] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0265] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in systems and devices, can be implemented as software, firmware, hardware, or a suitable combination thereof.

[0266] As used in the specification of this application and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0267] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or a similar expression means any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0268] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above-mentioned unit division is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0269] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0270] In addition, each functional unit in various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0271] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present application. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs and other various media that can store programs.

[0272] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings. However, this does not limit the scope of the rights of the embodiments of the present application. Any modification, equivalent replacement, and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A circuit board defect detection method, characterized in that: The method comprises: Acquire a circuit board video of a target circuit board in industrial production, and extract a video frame image from the circuit board video; Extracting circuit board depth defect features from the video frame image using a pre-trained circuit board defect detection model to obtain an initial defect depth feature map; Extracting circuit board defect mode features from the initial defect depth feature map to obtain a defect mode feature map; Performing defect feature fusion on the defect mode feature map and the initial defect depth feature map to obtain a target defect depth feature map; Performing pixel-by-pixel affinity processing on the video frame image to obtain pixel-by-pixel affinity features; Performing mode-by-mode affinity processing on the defect mode feature graph to obtain mode-by-mode affinity features; Performing affinity feature fusion on the pixel-by-pixel affinity feature and the pattern-by-pattern affinity feature to obtain a fused affinity feature; The circuit board defect recognition is performed on the video frame image based on the target defect depth feature map and the fused affinity feature to obtain target defect data of the target circuit board.

2. The method according to claim 1, characterized in that The step of extracting circuit board defect mode features from the initial defect depth feature map to obtain a defect mode feature map includes: Performing feature extraction of different channel dimensions on the initial defect depth feature map to obtain a channel dimension feature map; Defect mode feature recognition is performed on the channel dimension feature map to obtain the defect mode feature map.

3. The method according to claim 1, characterized in that The performing pixel-by-pixel affinity processing on the video frame image to obtain a pixel-by-pixel affinity feature includes: Acquire a target frame image of the video frame image, and acquire a target frame image pixel value of the target frame image; Extracting adjacent frame images of the target frame image from the video frame image, and obtaining adjacent frame image pixel values ​​of the adjacent frame images; Extracting features from pixel values ​​of the target frame image to obtain features of the target frame image; Extracting features from pixel values ​​of adjacent frame images to obtain features of adjacent frame images; The target frame image features and the adjacent frame image features of all video frames are pixel-by-pixel integrated to obtain the pixel-by-pixel affinity features.

4. The method according to claim 1, characterized in that The step of fusing the pixel-by-pixel affinity feature and the pattern-by-pattern affinity feature to obtain a fused affinity feature includes: Obtaining a pixel feature weight of the pixel-by-pixel affinity feature, and obtaining a pattern feature weight of the pattern-by-pattern affinity feature; Determining an enhanced pixel-by-pixel affinity feature according to the pixel feature weight and the corresponding pixel-by-pixel affinity feature; Determining an enhanced mode-by-mode affinity feature according to the mode feature weight and the corresponding mode-by-mode affinity feature; The fused affinity feature is determined according to the enhanced pixel-by-pixel affinity feature and the enhanced pattern-by-pattern affinity feature.

5. The method according to claim 1, characterized in that The target defect data includes a target defect category and a target defect position, and the circuit board defect recognition is performed on the video frame image based on the target defect depth feature map and the fusion affinity feature to obtain the target defect data of the target circuit board, including: Based on the target defect depth feature map and the fused affinity feature, the video frame image is subjected to circuit board defect category identification to obtain the target defect category; Based on the target defect category, the circuit board defect position is identified on the video frame image to obtain the target defect position; The target defect category and the target defect position are determined as the target defect data of the target circuit board.

6. The method according to any one of claims 1 to 5, characterized in that: The circuit board defect detection model pre-trained to extract circuit board depth defect features from the video frame image to obtain an initial defect depth feature map includes: Obtaining image pixel values ​​of the video frame image; The pre-trained circuit board defect detection model is used to perform defect depth convolution processing on the image pixel values ​​of each frame to obtain the initial defect depth feature map.

7. The method according to any one of claims 1 to 5, characterized in that: Before extracting circuit board depth defect features from the video frame image using the pre-trained circuit board defect detection model to obtain an initial defect depth feature map, the method further includes: Obtaining the original circuit board defect detection model and training data set; the training data set includes a plurality of training video frame images and real defect data corresponding to each of the training video frame images; Extracting circuit board depth defect features from a plurality of training video frame images using the original circuit board defect detection model to obtain a training defect depth feature map; Extracting circuit board defect pattern features from the training defect depth feature map to obtain a training defect pattern feature map; Performing defect feature fusion on the training defect mode feature map and the training defect depth feature map to obtain a target training defect depth feature map; Performing pixel-by-pixel affinity processing on a plurality of the training video frame images to obtain training pixel-by-pixel affinity features; Performing pattern-by-pattern affinity processing on the training defect pattern feature graph to obtain training pattern-by-pattern affinity features; Performing affinity feature fusion on the training pixel-by-pixel affinity feature and the training pattern-by-pattern affinity feature to obtain a fused training affinity feature; Based on the training defect depth feature map and the fused training affinity feature, circuit board defect prediction is performed on the plurality of training video frame images to obtain predicted defect data; Calculate the loss value of the predicted defect data and the actual defect data according to a preset target loss function; The model parameter weights of the original circuit board defect detection model are updated based on the loss value, and the circuit board depth defect feature extraction of the multiple training video frame images is performed back through the original circuit board defect detection model until the original circuit board defect detection model meets the preset training conditions, thereby obtaining the pre-trained circuit board defect detection model.

8. A circuit board defect detection device, characterized in that: The device comprises: An image extraction module is used to obtain a circuit board video of a target circuit board in industrial production and extract a video frame image from the circuit board video; A depth defect feature extraction module is used to extract circuit board depth defect features from the video frame image through a pre-trained circuit board defect detection model to obtain an initial defect depth feature map; A defect mode feature extraction module, used to extract circuit board defect mode features from the initial defect depth feature map to obtain a defect mode feature map; A defect feature fusion module, used to perform defect feature fusion on the defect mode feature map and the initial defect depth feature map to obtain a target defect depth feature map; A pixel-by-pixel affinity processing module, used to perform pixel-by-pixel affinity processing on the video frame image to obtain pixel-by-pixel affinity features; A mode-by-mode affinity processing module, used for performing mode-by-mode affinity processing on the defect mode feature map to obtain a mode-by-mode affinity feature; An affinity feature fusion module, used for performing affinity feature fusion on the pixel-by-pixel affinity feature and the pattern-by-pattern affinity feature to obtain a fused affinity feature; A circuit board defect recognition module is used to perform circuit board defect recognition on the video frame image based on the target defect depth feature map and the fused affinity feature to obtain target defect data of the target circuit board.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the circuit board defect detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the circuit board defect detection method according to any one of claims 1 to 7 is implemented.

Images