Image processing method, electronic device, and computer-readable storage medium

Abstract

The application discloses an image processing method, an electronic device and a computer readable storage medium. The image processing method is implemented through an image processing model and comprises the following steps: obtaining an initial mask image of a circuit board and a first image of a product to be detected processed based on the initial mask image; inputting the first image into an encoder of a semantic segmentation network of the image processing model for feature extraction to obtain a feature map; inputting the feature map and the initial mask image into a decoder of the semantic segmentation network for feature fusion to obtain a semantic segmentation image; and inputting the semantic segmentation image and the first image into an image generation model for fusion rendering to obtain a defect-free reconstruction image. Through the image processing model, the semantic features and texture information of the first image are fused in the initial mask image. Not only can the image processing method realize automatic operation in defect detection and repair, but also can accurately repair possible defects in the first image, thereby effectively improving the defect detection efficiency and repair precision of the product.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to image processing methods, electronic devices, and computer-readable storage media. Background Technology

[0002] In industrial manufacturing, quality inspection is a key part of the production process. For example, in the field of PCB (Printed Circuit Board) manufacturing, inspecting the surface condition of the PCB is an important means of controlling product quality, determining whether there are defects or flaws in the product, and taking appropriate actions based on the inspection results. This is crucial for improving the yield of the production line.

[0003] In existing technologies, PCB inspection and repair are mainly carried out by combining machine inspection and manual inspection. First, image inspection is used to inspect the appearance of products on the production line and automatically report the location of suspected defects. Then, manual inspection is carried out. After taking pictures using existing imaging systems, quality inspectors observe the pictures to determine the specific location of the defects. Finally, the defect images are repaired manually.

[0004] However, the detection principle of existing imaging systems is insufficient. The master image of the circuit board used for feature comparison is the surface image of the first product in each batch of work orders. Since the master image itself may have defects, it is difficult to guarantee the quality of subsequent products by directly comparing the master image. On the other hand, using manual inspection and repair of the master image also has the problems of low quality inspection efficiency and low accuracy, which is not conducive to real-time defect detection and repair of products, thus affecting the final defect detection efficiency and repair accuracy. Summary of the Invention

[0005] The main technical problem addressed by this application is to provide an image processing method, electronic device, and computer-readable storage medium that can solve the problems of low efficiency in product defect detection and repair accuracy in the prior art.

[0006] To address the aforementioned technical problems, the first technical solution adopted in this application is to provide an image processing method. This method is implemented using a trained image processing model. The image processing model includes a cascaded semantic segmentation network and an image generation model. The semantic segmentation network includes a cascaded encoder and decoder. The image processing method includes: acquiring an initial mask image of a circuit board and a first image of the product to be inspected, processed based on the initial mask image; inputting the first image into the semantic segmentation network of the image processing model, and extracting features from the first image using the encoder to obtain a feature map of the first image; inputting the feature map and the initial mask image into the decoder, and fusing the features of the feature map and the initial mask image using the decoder to obtain a semantically segmented image; and inputting the semantically segmented image and the first image into the image generation model, and fusing and rendering the first image and the semantically segmented image using the image generation model to obtain a defect-free reconstructed image of the first image.

[0007] The steps of inputting the first image into the semantic segmentation network of the image processing model and extracting features from the first image through the encoder to obtain the feature map of the first image include: using the encoder to perform downsampling and multiple convolution operations on the first image to obtain a low-level feature map and a high-level feature map of the first image; upsampling the high-level feature map to make the resolution of the upsampled high-level feature map consistent with that of the low-level feature map; and stacking the upsampled high-level feature map and the low-level feature map to obtain a feature map.

[0008] The step of inputting the feature map and the initial mask image into the decoder, and then fusing the features of the feature map and the initial mask image through the decoder to obtain a semantic segmentation image, includes: downsampling the initial mask image to obtain a first mask image; wherein the resolution of the first mask image is consistent with the resolution of the feature map; the step of inputting the feature map and the initial mask image into the decoder, and then fusing the features of the feature map and the initial mask image through the decoder to obtain a semantic segmentation image, includes: inputting the feature map and the first mask image into the decoder, and then using a spatial adaptive normalization mechanism to fuse the features of the feature map and the first mask image to obtain a semantic segmentation image.

[0009] The steps of inputting the feature map and the first mask image into the decoder and fusing the feature map and the first mask image using a spatial adaptive normalization mechanism to obtain a semantic segmentation image include: adjusting the size of the feature map and the first mask image to the same set size; performing batch normalization on the feature map; performing convolution on the first mask image to obtain a second mask image with a first preset number of channels; performing nonlinear activation and convolution on the second mask image to obtain a third mask image with a second preset number of channels; multiplying the third mask image and the normalized feature map pixel by pixel to obtain an updated feature map, and then adding the updated feature map and the third mask image pixel by pixel to fill the feature map in multiple regions of the third mask image to obtain a first output feature map after the first feature fusion; adjusting the size of the first output feature map and the first mask image to the same set size again, and repeating the above steps of batch normalization, nonlinear activation, convolution, pixel-by-pixel multiplication and pixel-by-pixel addition until an output feature map after a preset number of feature fusions is obtained; and performing convolution and nonlinear activation on the output feature map to obtain a semantic segmentation image.

[0010] The step of performing convolution and nonlinear activation on the output feature map to obtain a semantic segmentation image includes: performing convolution and nonlinear activation on the output feature map to obtain an initial semantic segmentation image; performing an upsampling operation on the initial semantic segmentation image to obtain a semantic segmentation image; wherein the semantic segmentation image has the same resolution as the first image.

[0011] The steps of inputting the semantic segmentation image and the first image into an image generation model, and fusing and rendering the first image and the semantic segmentation image through the image generation model to obtain a defect-free reconstructed image of the first image include: concatenating the first image and the semantic segmentation image using the image generation model to obtain a concatenated image; performing multiple convolutions and nonlinear activations on the concatenated image to fuse the texture information in the first image with the semantic segmentation image to obtain a defect-free reconstructed image of the first image.

[0012] The semantic segmentation network's encoder includes multiple sets of serial feature extraction units; the decoder includes multiple sets of serial spatial adaptive normalization residual units; and the spatial adaptive normalization residual units include spatial adaptive normalization units and convolutional units.

[0013] The image processing model is trained using labeled images. The training method includes: acquiring multiple initial mask images, multiple first images, and multiple second images repaired based on the first images to form a dataset; dividing the multiple initial mask images, multiple first images, and multiple second images in the dataset into a training set, a test set, and a validation set according to a set ratio; inputting the training data in the training set into a preset deep learning model for training to obtain a first model; inputting the test data in the test set into the first model for prediction, and calculating the total loss function based on the prediction results, and using the total loss function to back-update the model parameters of the first model to obtain a second model; inputting the validation data in the validation set into the second model for prediction, and evaluating the prediction results of the second model based on the prediction results to construct the image processing model.

[0014] To solve the above-mentioned technical problems, the second technical solution adopted in this application is to provide an electronic device, including: a memory for storing program data, which, when executed, implements the steps in the image processing method described above; and a processor for executing the program data stored in the memory to implement the steps in the image processing method described above.

[0015] To solve the above-mentioned technical problems, the third technical solution adopted in this application is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the image processing method described above.

[0016] The beneficial effects of this application are as follows: Unlike existing technologies, this application provides an image processing method, electronic device, and computer-readable storage medium. A trained image processing model is used to detect defects in a first image of the product to be inspected. This image processing model includes a semantic segmentation network, which can extract feature maps from the first image using the encoder of the semantic segmentation network, and fuse the extracted feature maps with an initial mask image using the decoder. This fully integrates the semantic features of the first image at different locations in the initial mask image. Furthermore, by using an image generation model to fuse and render the first image and the semantic segmentation image, the texture information in the first image can be fully integrated into the semantic segmentation image, thereby obtaining a defect-free reconstructed image of the first image. This application processes the first image using an image processing model and fuses the semantic features and texture information of the first image in the initial mask image. This not only automates each step of defect detection and repair but also accurately repairs potential defects in the first image, thereby effectively improving the efficiency of product defect detection and repair accuracy. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an architecture diagram of one embodiment of the image processing model of this application;

[0019] Figure 2 yes Figure 1 Architecture diagram of a specific implementation of the encoder of a semantic segmentation network;

[0020] Figure 3 yes Figure 1 Architecture diagram of a specific implementation of the decoder for a semantic segmentation network;

[0021] Figure 4 yes Figure 3 Architecture diagram of one implementation of the mid-space adaptive normalized residual unit;

[0022] Figure 5 yes Figure 4 Architecture diagram of one implementation of the space adaptive normalization unit;

[0023] Figure 6 yes Figure 1 Architecture diagram of a specific implementation of the image generation model;

[0024] Figure 7 This is a flowchart illustrating one implementation method of the image processing model training method of this application;

[0025] Figure 8 This is a flowchart illustrating one embodiment of the image processing method of this application;

[0026] Figure 9 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0027] Figure 10 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0029] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless otherwise clearly indicated above. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0030] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0031] It should be understood that the terms "comprising," "including," or any other variations used herein are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0032] In existing technologies, PCB inspection and repair primarily employ a combination of machine and manual inspection. First, image detection is used to inspect the surface of products on the production line and automatically identify suspected defects. Then, manual inspection is performed, using existing imaging systems to take pictures, which quality inspectors then observe to pinpoint the exact location of the defects. Finally, the defective images are manually repaired. However, the detection principles of existing imaging systems have shortcomings. The master image of the PCB used for feature comparison is the surface image of the first product in each batch of work orders. Since the master image itself may contain defects, directly comparing it with subsequent products cannot guarantee quality. Furthermore, using manual inspection and repair of the master image also suffers from low inspection efficiency and accuracy, hindering real-time defect detection and repair, thus affecting the final defect detection efficiency and repair accuracy.

[0033] Based on the above, this application provides an image processing method, an electronic device, and a computer-readable storage medium, which can solve the problems of low product defect detection efficiency and low repair accuracy in the prior art.

[0034] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] This application first provides an image processing model.

[0036] Specifically, please refer to Figure 1 , Figure 1 This is an architecture diagram of one embodiment of the image processing model of this application. In this embodiment, the image processing model 10 includes a cascaded semantic segmentation network 20 and an image generation model 30.

[0037] In this embodiment, the semantic segmentation network 20 includes a cascaded encoder 21 and decoder 22. The encoder 21 is used for image feature extraction, and the decoder 22 is used to restore the feature image size and output the final semantic segmentation image. The image generation model 30 is used to generate the final fused image based on the semantic segmentation image.

[0038] In this embodiment, the master circuit board image a is input into the semantic segmentation network 20 of the image processing model 10. The encoder 21 extracts the image features of the master circuit board image b, and then inputs the image features and the initial mask image a into the decoder 22. The decoder 22 fuses the image features with the initial mask image a, so that the spatial adaptive normalization unit 221 fully fuses the semantic features of the master circuit board image at different positions of the initial mask image a, thereby generating a semantic segmentation image c. Then, the master circuit board image b and the semantic segmentation image c are input into the image generation model 30. The image generation model 30 extracts the texture information of the master circuit board image b, and fuses and renders the texture information with the semantic segmentation image c, so that the texture information at different positions in the master circuit board image b is filled into the corresponding areas of the semantic segmentation image c, thereby generating a defect-free reconstructed image d of the master circuit board image.

[0039] Here, the initial mask image 'a' refers to the theoretical design drawing (Label image) of the circuit board, which marks the mask information of each area on the circuit board, i.e., the theoretical parameters of the design drawing in the circuit board. Before processing the board, it is necessary to first write the theoretical design drawing of the circuit board so that actual processing can be carried out based on the theoretical design drawing.

[0040] Understandably, by using the semantic segmentation network 20 to extract image features from the circuit board master image b and fusing them with the initial mask image a, Gaussian features rich in semantic information can be filled into each region of the initial mask image a, thereby achieving spatially adaptive normalized semantic image synthesis. Furthermore, by using the image generation model 30 to extract texture information from the circuit board master image b and fusing the texture information with the semantic segmentation image c, the real texture information from the actual product can be integrated into the theoretical design graphic. This ensures that the obtained image contains both the theoretical parameters of the circuit board and the rich semantic and texture information from the master image, thus obtaining a defect-free reconstructed image d of the actual processed product.

[0041] Understandably, since the above steps are all implemented through the image processing model 10, not only can each step of defect detection and defect repair be automated, but defects that may exist in the first image can also be accurately repaired, thereby effectively improving the defect detection efficiency and repair accuracy of the product.

[0042] In this embodiment, the encoder 21 of the semantic segmentation network 20 includes multiple sets of serial feature extraction units 210. The feature extraction units 210 are used to extract image features.

[0043] Specifically, please refer to Figure 2 , Figure 2 yes Figure 1 Architecture diagram of a specific implementation of the encoder of a semantic segmentation network.

[0044] In this embodiment, the encoder 21 includes six sets of serial feature extraction units 210.

[0045] Each feature extraction unit 210 has a 3×3 convolution kernel. Specifically, the first group of feature extraction units 210 has 64 convolution kernels, the second group has 128 convolution kernels, the third group has 256 convolution kernels, and the fourth, fifth, and sixth groups of feature extraction units 210 each have 512 convolution kernels.

[0046] In this process, after each feature extraction unit 210 extracts image features, it performs instance normalization (InsNorm) and nonlinear activation on the network layer, and the activation function used is the rectified linear unit (ReLU).

[0047] Specifically, the master circuit board image is input into encoder 21. First, the master circuit board image is downsampled to reduce the image resolution. Then, the downsampled master circuit board image is input into the first feature extraction unit 210 for feature extraction, resulting in a first feature map with 64 channels. Next, the first feature map is input into the second feature extraction unit 210 for feature extraction, resulting in a second feature map with 128 channels. Then, the second feature map is input into the third feature extraction unit 210 for feature extraction, resulting in a third feature map with 256 channels. Then, the third feature map is input into the fourth feature extraction unit 210 for feature extraction, resulting in a fourth feature map with 256 channels. Next, the fourth feature map is input into the fifth feature extraction unit 210 for feature extraction, resulting in a fifth feature map with 256 channels. Finally, the fifth feature map is input into the sixth feature extraction unit 210 for feature extraction, resulting in a sixth feature map with 256 channels. The sixth feature map is then subjected to global pooling, resulting in an output feature map of dimension 8192×1×1. Here, 8192 refers to the number of channels (C), the first '1' represents the height (H), and the second '1' represents the width (W). The output feature map is then processed into two linear Gaussian vectors of length 256, representing the Gaussian mean and variance, respectively. These two Gaussian vectors are output to represent the multi-scale Gaussian features in the circuit board master image.

[0048] The Gaussian mean can be viewed as a low-level feature map, encompassing relatively comprehensive local features in the master circuit board image. The Gaussian variance can be viewed as a high-level feature map, including deeper semantic features in the master circuit board image.

[0049] In this embodiment, the decoder 22 of the semantic segmentation network 20 includes multiple sets of serial Spatial Adaptive Normalization Residual Blocks (SPADE resblocks) 220. Each Spatial Adaptive Normalization Residual Block 220 includes both Spatial Adaptive Normalization (SPADE) units and convolutional units.

[0050] Specifically, please refer to Figure 3 , Figure 4 and Figure 5 , Figure 3 yes Figure 1 Architecture diagram of a specific implementation of the decoder in a semantic segmentation network. Figure 4 yes Figure 3Architecture diagram of one implementation of the mid-space adaptive normalized residual unit. Figure 5 yes Figure 4 Architecture diagram of one implementation of the space adaptive normalization unit.

[0051] In this embodiment, the decoder 22 includes seven sets of serial spatial adaptive normalized residual units 220. Each set of spatial adaptive normalized residual units 220 includes left and right branches. The left branch includes two sets of serial SPADE and convolution units, and the right branch includes one set of SPADE and convolution units.

[0052] Specifically, the Gaussian variance is first upsampled to match the resolution of the Gaussian mean, and then the two are stacked to obtain a feature map with dimensions of 256×16384, where 256 refers to the length of the feature map and 16384 refers to the width. Next, the feature map is resized to 1024×4×4, where 1024 refers to the length, the first 4 refers to the width, and the second 4 refers to the number of feature channels. Finally, the initial mask image is downsampled to match the resolution of the feature map.

[0053] The feature map with consistent resolution and the initial mask image are then input into the first set of spatially adaptive normalized residual units 220. Simultaneously, the initial mask image and the feature map are fed into the left and right branches. The SPADE in both branches first adjusts the size of the initial mask image to the same set size as the feature map, and then convolves the initial mask image to obtain a mask image with 128 channels. The convolution kernel size is 3×3, and the number of convolution kernels is 128. Next, nonlinear activation (ReLU) and convolution (3×3×k convolution, 3×3×k Conv) are performed on the mask image to obtain a mask image with k channels. The value of k can be 1024. In other embodiments, the value of k can be other values, which are not limited in this application. The SPADE in both branches is also used to perform Sync Batch Normalization (Sync BN) on the input feature map. Then, the mask image with k channels is multiplied pixel by pixel with the normalized feature map to obtain the updated feature map. The updated feature map is then added pixel by pixel with the mask image with k channels to fill the corresponding Gaussian vectors in multiple regions of the mask image, resulting in the output feature map after feature fusion.

[0054] Furthermore, in the left branch, after applying ReLU nonlinear activation to the feature map output by the first SPADE, a convolutional unit (3×3×k convolution, 3×3×k Conv) is used to convolve the output feature map. This convolved feature map is then used as the input feature map for the first SPADE, while the initial mask image is used as the input image for the second SPADE. The second SPADE then performs feature fusion again between the input feature map and the initial mask image, resulting in a more fully fused output feature map. The output feature map obtained from the left branch is then pixel-wise added to the output feature map obtained from the right branch to obtain the first output feature map.

[0055] Further, the first output feature map and the initial mask image are input into the second set of spatial adaptive normalization residual units 220. The sizes of the first output feature map and the initial mask image are adjusted to the same set size again, and the above steps of synchronous batch normalization, nonlinear activation, convolution, pixel-wise multiplication, and pixel-wise addition are repeated to obtain the second output feature map. The second output feature map and the initial mask image are then input into the third set of spatial adaptive normalization residual units 220. The above steps are repeated until the output feature map is obtained through the seventh set of spatial adaptive normalization residual units 220. Then, the output feature map is convolved using a convolution unit (3×3×1Conv), and the convolved output feature map is nonlinearly activated using an activation function of hyperbolic tangent (TanH) to obtain the initial semantic segmentation image. Finally, the initial semantic segmentation image is upsampled to obtain the final semantic segmentation image. The resolution of the semantic segmentation image is consistent with that of the circuit board master image.

[0056] Understandably, by setting left and right branches in the spatial adaptive normalization residual unit 220 and using SPADE in the two branches to fuse the initial mask image and the input feature map, the Gaussian features can be guided to learn more important and complete object regions, thereby realizing spatial adaptive normalization semantic image synthesis.

[0057] Furthermore, by fusing the input feature map with the initial mask image through multiple sets of serial spatial adaptive normalization residual units 220, Gaussian features with rich semantic information can be filled in each region of the initial mask image to improve the accuracy of spatial adaptive normalization semantic image synthesis.

[0058] In this embodiment, the image generation model 30 includes multiple convolutional units 301.

[0059] Specifically, please refer to Figure 6 , Figure 6 yes Figure 1Architecture diagram of a specific implementation of the image generation model.

[0060] In this embodiment, the image generation model 30 includes six sets of serial convolutional units 301.

[0061] The first group of convolutional units 301 has a 3×3 kernel size, 32 kernels, and a padding parameter of 3. The second group of convolutional units 301 has a 5×5 kernel size, 64 kernels, and a padding parameter of 5. The third group of convolutional units 301 has a 7×7 kernel size, 128 kernels, and a padding parameter of 7. The fourth group of convolutional units 301 has a 3×3 kernel size, 32 kernels, and a padding parameter of 3. The fifth group of convolutional units 301 has a 3×3 kernel size, 1 kernel, and a padding parameter of 3.

[0062] In each group of convolutional units 301, after convolving the image, the image is non-linearly activated using an activation layer, and the activation function used is TanH.

[0063] Specifically, the circuit board master image and the semantic segmentation image are input into the image generation model 30. First, the circuit board master image is concatenated using the image generation model 30 to obtain a concatenated image. Then, six sets of serial convolutional units 301 and multiple activation layers are used to perform multiple convolutions and nonlinear activations on the concatenated image to fuse and render the texture information in the circuit board master image with the semantic segmentation image, thereby obtaining a defect-free reconstructed image of the circuit board master image.

[0064] In this embodiment, the image processing model 10 is trained using labeled images. For details, please refer to... Figure 7 , Figure 7 This is a flowchart illustrating one embodiment of the training method for the image processing model of this application. In this embodiment, the training method includes:

[0065] S71: Obtain multiple initial mask images, multiple first images, and multiple second images repaired based on the first images to form a dataset.

[0066] In this embodiment, the first image is the surface image of the first product in each batch of work orders, which is also the master image of the circuit board.

[0067] The second image is a defect-free version of the first image that has been manually repaired.

[0068] Each initial mask image, along with its corresponding first and second images, constitutes associated training data.

[0069] S72: Divide the multiple initial mask images, multiple first images, and multiple second images in the dataset into a training set, a test set, and a validation set according to a set ratio.

[0070] In this embodiment, the training set is used to train the preset deep learning model, the test set is used to evaluate and adjust the model, and the validation set is used to perform the final evaluation of the model.

[0071] In this embodiment, the multiple initial mask images, multiple first images, and multiple second images in the dataset are divided into a training set, a test set, and a validation set in a ratio of 70%:20%:10%.

[0072] In other embodiments, the set ratio may be other ratios, and this application does not limit it.

[0073] S73: Input the training data in the training set into the preset deep learning model for training to obtain the first model.

[0074] In this embodiment, when training using training data in the training set, the Gaussian feature loss function between the image features output by the encoder of the semantic segmentation network and the mean of the overall Gaussian feature vector in the training set, the segmentation loss function between the semantic segmentation image output by the decoder of the semantic segmentation network and the initial mask image, the derivation loss function between the semantic segmentation image output by the decoder and the first image, and the defect removal loss function between the defect-free reconstructed image output by the image generation model and the labeled second image are calculated.

[0075] In calculating the segmentation loss function between the semantic segmentation image and the initial mask image, the semantic segmentation image and the initial mask image are concatenated, and a multi-layer convolutional network and activation layer are used to perform convolution and non-linear activation on the concatenated image to obtain the segmentation loss function.

[0076] In a specific implementation scenario, a multi-layer convolutional network with five sets of 4×4 kernels and 64, 128, 256, 512 and 1 kernels can be used to convolve the cascaded image, and ReLU can be used to activate the convolved image.

[0077] In this embodiment, the total loss function of the preset deep learning model is calculated based on the Gaussian feature loss function, the segmentation loss function, the derived loss function, and the defect removal loss function. The model parameters of the preset deep learning model are then updated in reverse using the total loss function to obtain the first model.

[0078] Understandably, the total loss function in the training set represents the training direction of the pre-defined deep learning model.

[0079] S74: Input the test data from the test set into the first model for prediction, and calculate the total loss function based on the prediction results. Then, update the model parameters of the first model in reverse based on the total loss function to obtain the second model.

[0080] For details on how to calculate the loss function, please refer to the description in S73, which will not be repeated here.

[0081] Understandably, the total loss function in the test set serves as the basis for adjusting the parameters of the first model.

[0082] S75: Input the validation data from the validation set into the second model for prediction, and evaluate the prediction results of the second model based on the prediction results to construct an image processing model.

[0083] For details on how to calculate the loss function, please refer to the description in S73, which will not be repeated here.

[0084] Understandably, the total loss function in the validation set serves as the evaluation criterion for the second model.

[0085] In this embodiment, the model parameters of the preset deep learning model are back-trained using the total loss function to calculate the gradient values ​​of all model parameters and back-update the parameter values ​​of the preset segmentation model to achieve the purpose of optimizing the model and thus obtaining the trained image processing model.

[0086] Please see Figure 8 , Figure 8 This is a flowchart illustrating one embodiment of the image processing method of this application. In this embodiment, the image processing method is implemented through the aforementioned image processing model, which includes a cascaded semantic segmentation network and an image generation model. The semantic segmentation network includes a cascaded encoder and decoder.

[0087] The image processing method includes:

[0088] S81: Obtain the initial mask image of the circuit board and the first image of the product to be inspected processed based on the initial mask image.

[0089] In this embodiment, the initial mask image refers to the theoretical design drawing (Label image) of the circuit board, which marks the mask information of each area on the circuit board, i.e., the theoretical parameters of the design drawing in the circuit board. Before processing the board, it is necessary to first write the theoretical design drawing of the circuit board so that actual processing can be carried out based on the theoretical design drawing.

[0090] In this embodiment, the product to be tested is the first product processed based on the initial mask image in each batch of work orders. The first image is the surface image of the first product obtained by taking a picture, that is, the master image of the circuit board.

[0091] S82: Input the first image into the semantic segmentation network of the image processing model, and extract features from the first image through the encoder to obtain the feature map of the first image.

[0092] In this embodiment, the encoder is used to downsample the first image and perform multiple convolution operations to obtain the low-level feature map and high-level feature map of the first image.

[0093] Among them, the low-level feature map is the Gaussian mean, and the high-level feature map is the Gaussian variance, both of which are Gaussian vectors of length 256.

[0094] S83: Input the feature map and the initial mask image into the decoder. The decoder performs feature fusion on the feature map and the initial mask image to obtain the semantic segmentation image.

[0095] In this embodiment, the high-level feature map is first upsampled using a decoder to ensure that the resolution of the upsampled high-level feature map matches that of the low-level feature map. The upsampled high-level feature map and the low-level feature map are then stacked to obtain the feature map of the first image. Next, the initial mask image is downsampled to obtain the first mask image. The resolution of the first mask image is consistent with the resolution of the feature map.

[0096] Furthermore, the feature map and the first mask image are input into multiple sets of serial spatial adaptive normalization residual units (SPADE resblock) of the decoder. The feature map and the first mask image are fused using the spatial adaptive normalization mechanism to obtain the semantic segmentation image.

[0097] Specifically, the first set of Spatial Adaptive Normalization Units (SPADE) in the SPADE resblock is used to adjust the size of both the feature map and the first mask image to the same set size, and then the feature map is synchronously batch normalized. Next, the first mask image is convolved to obtain a second mask image with a first preset number of channels. The second mask image is then non-linearly activated and convolved using an activation function and a convolutional network to obtain a third mask image with a second preset number of channels. The third mask image is then multiplied pixel-by-pixel with the normalized feature map to obtain an updated feature map. Finally, the updated feature map is added pixel-by-pixel to the third mask image to fill multiple regions of the third mask image, resulting in the output feature map.

[0098] After applying ReLU activation to the feature map output by the first SPADE using the left branch of the SPADE resblock, a convolutional unit is used to convolve the output feature map. This convolutional feature map is then used as the input feature map for the first SPADE. Simultaneously, the initial mask image is used as the input image for the second SPADE. The second SPADE then fuses the input feature map with the initial mask image again, resulting in a more fully fused output feature map. The output feature map obtained from the left branch is then pixel-by-pixel added to the output feature map obtained from the SPADE in the right branch, yielding the first output feature map after the first feature fusion.

[0099] Further, the first output feature map and the initial mask image are input into the second set of spatial adaptive normalization residual units. The sizes of the first output feature map and the initial mask image are adjusted back to the same set size, and the above steps of synchronous batch normalization, nonlinear activation, convolution, pixel-wise multiplication, and pixel-wise addition are repeated to obtain the second output feature map. The second output feature map and the initial mask image are then input into the third set of spatial adaptive normalization residual units. These steps are repeated until a preset number of spatial adaptive normalization residual units are used to obtain the output feature map after a preset number of feature fusion iterations. Then, convolution and nonlinear activation are performed on the output feature map to obtain the initial semantic segmentation image, and an upsampling operation is performed on the initial semantic segmentation image to obtain the semantic segmentation image. The semantic segmentation image has the same resolution as the first image.

[0100] S84: Input the semantic segmentation image and the first image into the image generation model, and use the image generation model to fuse and render the first image and the semantic segmentation image to obtain a defect-free reconstructed image of the first image.

[0101] In this embodiment, an image generation model is used to concatenate the first image and the semantic segmentation image to obtain a concatenated image. Then, multiple convolutions and nonlinear activations are performed on the concatenated image to fuse the texture information in the first image with the semantic segmentation image for rendering, resulting in a defect-free reconstructed image of the first image.

[0102] Furthermore, the defect-free reconstructed image is used as a new circuit board master pattern to supply the remaining processes of the machine.

[0103] Understandably, defect-free reconstructed images conform to the actual situation of the production line and are standardized. Using them as the master image for feature comparison can reduce the false alarm rate of image detection, thereby improving detection efficiency.

[0104] Unlike existing technologies, this embodiment uses a trained image processing model to perform defect detection on a first image of the product to be inspected. This model includes a semantic segmentation network, which uses an encoder to extract feature maps from the first image and a decoder to fuse these extracted feature maps with an initial mask image. This fully integrates the semantic features of the first image at different locations within the initial mask image. Furthermore, an image generation model is used to fuse and render the first image and the semantically segmented image, fully integrating the texture information from the first image into the semantically segmented image, thereby obtaining a defect-free reconstructed image of the first image. This application processes the first image using an image processing model and fuses its semantic features and texture information into the initial mask image. This not only automates each step of defect detection and repair but also accurately repairs potential defects in the first image, effectively improving the efficiency and accuracy of product defect detection and repair.

[0105] Correspondingly, this application provides an electronic device.

[0106] Please see Figure 9 , Figure 9 This is a schematic diagram of one embodiment of the electronic device of this application. For example... Figure 9 As shown, the electronic device 90 includes a memory 91 and a processor 92.

[0107] In this embodiment, the memory 91 is used to store program data, and when the program data is executed, it implements the steps in the image processing method described above; the processor 92 is used to execute the program instructions stored in the memory 91 to implement the steps in the image processing method described above.

[0108] Specifically, processor 92 controls itself and memory 91 to implement the steps in the image processing method described above. Processor 92 can also be referred to as a CPU (Central Processing Unit). Processor 92 may be an integrated circuit chip with signal processing capabilities. Processor 92 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 92 can be implemented using multiple integrated circuit chips.

[0109] Unlike existing technologies, this embodiment uses a processor 92 to perform defect detection on a first image of the product to be inspected. The processor 92 employs an image processing model including a semantic segmentation network, which extracts feature maps from the first image using the encoder and fuses these extracted feature maps with an initial mask image using the decoder. This fully integrates the semantic features of the first image at different locations within the initial mask image. Furthermore, by using an image generation model to fuse and render the first image and the semantic segmentation image, the texture information in the first image is fully integrated into the semantic segmentation image, resulting in a defect-free reconstructed image of the first image. This application processes the first image using an image processing model and fuses its semantic features and texture information into the initial mask image. This not only automates each step of defect detection and repair but also accurately repairs potential defects in the first image, effectively improving the efficiency and accuracy of product defect detection and repair.

[0110] Correspondingly, this application provides a computer-readable storage medium.

[0111] Please see Figure 10 , Figure 10 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.

[0112] The computer-readable storage medium 100 includes a computer program 1001 stored on it. When executed by the processor, the computer program 1001 implements the steps of the image processing method described above. Specifically, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 100. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium 100 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 100 includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image processing method, characterized in that, The image processing method is implemented through a trained image processing model; the image processing model includes a cascaded semantic segmentation network and an image generation model, and the semantic segmentation network includes a cascaded encoder and decoder; The image processing method includes: Obtain the initial mask image of the circuit board and the first image of the product to be inspected processed based on the initial mask image; The first image is input into the semantic segmentation network of the image processing model, and the encoder extracts features from the first image to obtain the feature map of the first image. The feature map and the initial mask image are input into the decoder, and the decoder performs feature fusion on the feature map and the initial mask image to obtain a semantic segmentation image; The semantic segmentation image and the first image are input into the image generation model, and the image generation model is used to fuse and render the first image and the semantic segmentation image to obtain a defect-free reconstructed image of the first image. Before the step of inputting the feature map and the initial mask image into the decoder, and performing feature fusion on the feature map and the initial mask image by the decoder to obtain a semantic segmentation image, the following steps are included: The initial mask image is downsampled to obtain a first mask image; wherein the resolution of the first mask image is the same as the resolution of the feature map; The step of inputting the feature map and the initial mask image into the decoder, and performing feature fusion on the feature map and the initial mask image by the decoder to obtain a semantic segmentation image includes: The feature map and the first mask image are input into the decoder, and a spatial adaptive normalization mechanism is used to fuse the features of the feature map and the first mask image to obtain the semantic segmentation image. The step of inputting the feature map and the first mask image into the decoder, and using a spatial adaptive normalization mechanism to fuse the features of the feature map and the first mask image to obtain the semantic segmentation image includes: Adjust the size of the feature map and the first mask image to the same set size; Perform batch normalization on the feature maps; Convolve the first mask image to obtain a second mask image with a first preset number of channels; The second mask image is subjected to nonlinear activation and convolution to obtain a third mask image with a second preset number of channels; The third mask image is multiplied pixel by pixel with the normalized feature map to obtain the updated feature map. Then the updated feature map is added pixel by pixel to the third mask image to fill the feature map in multiple regions of the third mask image, resulting in the first output feature map after the first feature fusion. The sizes of the first output feature map and the first mask image are adjusted to the same set size again, and the above steps of batch normalization, non-linear activation, convolution, pixel-by-pixel multiplication and pixel-by-pixel addition are repeated until the output feature map after feature fusion after a preset number of times is obtained. The output feature map is convolved and nonlinearly activated to obtain the semantic segmentation image.

2. The image processing method according to claim 1, characterized in that, The step of inputting the first image into the semantic segmentation network of the image processing model, and extracting features from the first image through the encoder to obtain the feature map of the first image includes: The encoder is used to downsample and perform multiple convolution operations on the first image to obtain the low-level feature map and high-level feature map of the first image; The high-level feature map is upsampled so that the resolution of the upsampled high-level feature map is consistent with that of the low-level feature map; The upsampled high-level feature map is stacked with the low-level feature map to obtain the feature map.

3. The image processing method according to claim 2, characterized in that, The step of performing convolution and nonlinear activation on the output feature map to obtain the semantic segmentation image includes: The output feature map is convolved and nonlinearly activated to obtain an initial semantic segmentation image; An upsampling operation is performed on the initial semantic segmentation image to obtain the semantic segmentation image; wherein the semantic segmentation image has the same resolution as the first image.

4. The image processing method according to claim 3, characterized in that, The step of inputting the semantic segmentation image and the first image into the image generation model, and fusing and rendering the first image and the semantic segmentation image through the image generation model to obtain a defect-free reconstructed image of the first image includes: The image generation model is used to concatenate the first image and the semantic segmentation image to obtain a concatenated image. The cascaded image is subjected to multiple convolutions and nonlinear activations to fuse the texture information in the first image with the semantic segmentation image, thereby obtaining the defect-free reconstructed image of the first image.

5. The image processing method according to claim 1, characterized in that, The encoder of the semantic segmentation network includes multiple sets of serial feature extraction units; The decoder includes multiple sets of serial spatial adaptive normalized residual units; wherein, the spatial adaptive normalized residual unit includes a spatial adaptive normalized unit and a convolutional unit.

6. The image processing method according to claim 5, characterized in that, The image processing model is trained using labeled images, and the training method includes: Multiple initial mask images, multiple first images, and multiple second images repaired based on the first images are obtained to form a dataset; The dataset contains multiple initial mask images, multiple first images, and multiple second images, which are divided into a training set, a test set, and a validation set according to a set ratio. The training data in the training set is input into a preset deep learning model for training to obtain the first model; The test data in the test set is input into the first model for prediction, and the total loss function is calculated based on the prediction results. The model parameters of the first model are then updated in reverse based on the total loss function to obtain the second model. The validation data in the validation set is input into the second model for prediction, and the prediction results of the second model are evaluated based on the prediction results to construct the image processing model.

7. An electronic device, characterized in that, include: A memory for storing program data, which, when executed, implements the steps of the image processing method as described in any one of claims 1 to 6; A processor for executing the program data stored in the memory to implement the steps of the image processing method as claimed in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 6.

Images