Industrial visual inspection method based on deep convolutional neural network and related device

Abstract

The application relates to the technical field of image processing, and provides an industrial visual inspection method based on a deep convolutional neural network and related devices. The method comprises the following steps: acquiring an original image of an industrial product to be inspected, and performing environment interference suppression processing on the original image to obtain a pretreated image; performing multi-scale feature extraction on the pretreated image based on a preset convolutional neural network to obtain multi-scale local feature information; performing global analysis on the multi-scale local feature information based on a preset Transformer encoder to obtain global feature information; performing weighted fusion on the multi-scale local feature information based on a preset attention gate mechanism, taking the global feature information as a guide, to obtain a multi-scale attention-enhanced feature pyramid; and performing analysis on the multi-scale attention-enhanced feature pyramid based on a preset parallel prediction subnetwork to generate category and position information of defects in the original image. The method is helpful to improve the detection accuracy of micro defects.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an industrial vision inspection method and related apparatus based on deep convolutional neural networks. Background Technology

[0002] In modern industrial automated production, vision-based defect detection is a crucial step in ensuring product quality and production safety. Currently, industrial vision inspection typically relies on traditional image processing algorithms or single-architecture deep convolutional neural network models to identify various defects on product surfaces.

[0003] However, since tiny defects occupy a limited pixel area in an image and have weak features, traditional vision systems or single-architecture deep convolutional neural network models struggle to achieve stable and accurate identification of tiny defects, affecting the accuracy and reliability of defect detection. Summary of the Invention

[0004] The main objective of this application is to provide an industrial vision inspection method and related apparatus based on deep convolutional neural networks, aiming to solve the problems mentioned in the background art.

[0005] In a first aspect, this application provides an industrial vision inspection method based on a deep convolutional neural network, the method comprising the following steps: The original image of the industrial product to be inspected is acquired, and environmental interference suppression processing is performed on the original image to obtain a preprocessed image; Multi-scale feature extraction is performed on the preprocessed image based on a preset convolutional neural network to obtain multi-scale local feature information; The multi-scale local feature information is analyzed globally based on the preset Transformer encoder to obtain global feature information; Guided by the global feature information, the multi-scale local feature information is weighted and fused based on a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid; the multi-scale attention-enhanced feature pyramid is analyzed based on a preset parallel prediction sub-network to generate the category and location information of defects in the original image.

[0006] Secondly, this application also provides an industrial vision inspection system based on a deep convolutional neural network, the industrial vision inspection system based on a deep convolutional neural network comprising: The acquisition module is used to acquire the original image of the industrial product to be inspected, and to perform environmental interference suppression processing on the original image to obtain a preprocessed image; The feature extraction module is used to perform multi-scale feature extraction on the preprocessed image based on a preset convolutional neural network to obtain multi-scale local feature information; The first analysis module is used to perform global analysis on the multi-scale local feature information based on a preset Transformer encoder to obtain global feature information; The feature fusion module is used to perform weighted fusion of the multi-scale local feature information based on the global feature information and a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid; the second analysis module is used to analyze the multi-scale attention-enhanced feature pyramid based on a preset parallel prediction sub-network to generate the category and location information of defects in the original image.

[0007] Thirdly, this application also provides a terminal device, the terminal device including a processor, a memory and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the industrial vision inspection method based on deep convolutional neural networks as described in any of the preceding claims.

[0008] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the industrial vision inspection method based on a deep convolutional neural network as described in any of the preceding claims.

[0009] This embodiment provides an industrial vision inspection method and related apparatus based on a deep convolutional neural network. The method first acquires the original image of the industrial product to be inspected and performs environmental interference suppression processing on the original image to obtain a preprocessed image. This suppresses lighting changes, noise, and blur interference in the industrial environment, providing stable input for subsequent analysis. Then, based on a preset convolutional neural network, multi-scale feature extraction is performed on the preprocessed image to obtain multi-scale local feature information, capturing complete local visual cues from micro-texture to macro-structure. Next, based on a preset Transformer encoder, global analysis is performed on the multi-scale local feature information to obtain global features. Information establishes contextual semantic relationships between local features, enabling an understanding of the overall composition of the image. Furthermore, guided by global feature information, a pre-defined attention gating mechanism is used to weightedly fuse multi-scale local feature information, resulting in a multi-scale attention-enhanced feature pyramid. This achieves adaptive complementary fusion of local details and global semantics, enhancing the ability to express minute and weak features. Finally, a pre-defined parallel prediction sub-network is used to analyze the multi-scale attention-enhanced feature pyramid to generate the category and location information of defects in the original image, completing the end-to-end mapping from features to detection results and solving the problem of insufficient sensitivity of traditional methods to the identification of minute defects. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating an industrial vision inspection method based on a deep convolutional neural network, provided in an embodiment of this application; Figure 2 A schematic block diagram of the structure of an industrial vision inspection system based on a deep convolutional neural network, provided in an embodiment of this application; Figure 3 This is a schematic block diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0014] This application provides an industrial vision inspection method and related apparatus based on a deep convolutional neural network.

[0015] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0016] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an industrial vision inspection method based on a deep convolutional neural network, provided as an embodiment of this application. This deep convolutional neural network-based industrial vision inspection method can be used in a server, which can be a standalone server, a server cluster, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0017] like Figure 1 As shown, the industrial vision inspection method based on deep convolutional neural networks includes steps S1 to S5.

[0018] Step S1: Obtain the original image of the industrial product to be inspected, and perform environmental interference suppression processing on the original image to obtain a preprocessed image.

[0019] The environmental interference suppression processing refers to a series of digital image processing operations aimed at eliminating or reducing interference such as noise, blur, and uneven illumination introduced by physical environmental factors during image acquisition. Specifically, a two-dimensional digital matrix containing the complete field of view of the industrial product to be inspected is acquired through an image sensor; this matrix is ​​the original image. Subsequently, the original image is input into a processing pipeline composed of multiple algorithms. The pipeline sequentially performs correction for uneven illumination, filtering for random noise, and restoration for motion blur, outputting an image with improved quality; this image is the preprocessed image.

[0020] Step S2: Based on a preset convolutional neural network, perform multi-scale feature extraction on the preprocessed image to obtain multi-scale local feature information.

[0021] The convolutional neural network (CNN) refers to a feedforward neural network model composed of multiple convolutional layers, activation layers, and downsampling layers stacked together. Multi-scale feature extraction refers to the operation of extracting feature maps with different spatial resolutions and semantic abstraction levels from different depths of the CNN. Multi-scale local feature information refers to the set of these feature maps at different scales. Specifically, the preprocessed image output from step S1 is input into a pre-trained and structurally defined CNN model. During the forward propagation of this model, multiple feature maps with successively decreasing spatial dimensions but successively increasing feature channel numbers are simultaneously extracted from multiple output layers at specified depths (e.g., after the second, fourth, and sixth downsampling layers). All extracted feature maps are organized into an ordered set according to resolution from high to low or from low to high; this set is the multi-scale local feature information.

[0022] Step S3: Perform global analysis on the multi-scale local feature information based on the preset Transformer encoder to obtain global feature information.

[0023] The Transformer encoder refers to a neural network module based on a self-attention mechanism used to model long-range dependencies within sequential data. The global feature information refers to the updated feature representation that incorporates the overall semantic context of the image. Specifically, from the multi-scale local feature information obtained in step S2, a feature map (usually an intermediate resolution layer) that strikes a balance between semantic representativeness and computational efficiency is selected. This feature map is then expanded spatially, and the feature vector at each spatial location is treated as an independent token, forming a token sequence. This sequence is input into a pre-defined Transformer encoder containing multi-head self-attention layers and feedforward network layers. Inside the encoder, the correlation between all token pairs is calculated using a self-attention mechanism, ensuring that the features of each token are integrated with information from all other tokens. After processing, the updated token sequence is reassembled into a two-dimensional feature map structure, which is the global feature information.

[0024] Step S4: Guided by the global feature information, the multi-scale local feature information is weighted and fused based on a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid. The attention gating mechanism refers to a learnable computational module that dynamically generates spatial weight masks based on guiding information to control the fusion ratio of different feature sources. The multi-scale attention-enhanced feature pyramid is a structure composed of multi-level feature maps, where each layer is formed by fusing the original local features of the corresponding layer with high-level semantic features according to dynamic weights. Specifically, firstly, the global feature information output from step S3 is transformed through a convolutional layer to generate an initial fusion feature with the same size as the lowest resolution (highest semantic layer) feature map in the multi-scale local feature information. Then, a top-down iterative fusion process is performed: for the current processing level, the fusion feature generated in the previous level is upsampled to the same resolution as the local feature map of the current level through bilinear interpolation. At the same time, the global feature information is used as input and passed through an attention gating module composed of a convolutional layer and a sigmoid activation function to calculate and generate a spatial weight map with the same resolution as the current feature map and each element value between 0 and 1. Using this spatial weight map, the upsampled fusion feature and the original local feature information of the current level are summed element-wise with weights. The formula can be expressed as: Fusion result = Weight map × Upsampled feature × (1 - Weight map) × Local features; Starting from the highest semantic layer, iterate this process layer by layer downwards until the highest resolution local feature layer is processed; Finally, reorganize the fusion results of all levels generated during the iteration process in scale order. This structure is the multi-scale attention-enhanced feature pyramid.

[0025] Step S5: Analyze the multi-scale attention-enhanced feature pyramid based on a preset parallel prediction sub-network to generate the category and location information of defects in the original image.

[0026] The parallel prediction sub-network refers to a lightweight network module containing two parallel convolutional sub-network branches: one branch for classification and the other for bounding box regression. Specifically, each feature map layer in the multi-scale attention-enhanced feature pyramid generated in step S4 is input into the same parallel prediction sub-network. In this sub-network, the classification branch, through several convolutional layers, outputs a probability distribution vector representing the various defect categories and background at each preset anchor point or pixel location in the feature map. The regression branch, through several convolutional layers, outputs a geometric offset vector at the same location, used to finely adjust the coordinates and size of the predefined anchor box at that location. The preliminary predicted category probabilities and adjusted bounding boxes at all scales and locations are collected to form a preliminary prediction result set. A non-maximum suppression algorithm is applied to this set to remove redundant prediction boxes with excessive overlap. Finally, a structured list is output, where each entry contains the category label, confidence score, and bounding box coordinates of a confirmed defect in the original image, thereby generating the defect category and location information.

[0027] The method provided in this embodiment first acquires the original image of the industrial product to be inspected and performs environmental interference suppression processing on the original image to obtain a preprocessed image, which suppresses the light changes, noise, and blur interference in the industrial environment, providing a stable input for subsequent analysis. Then, based on a preset convolutional neural network, multi-scale feature extraction is performed on the preprocessed image to obtain multi-scale local feature information, capturing complete local visual cues from micro-texture to macro-structure. Second, based on a preset Transformer encoder, global analysis is performed on the multi-scale local feature information to obtain global feature information, establishing contextual semantic relationships between local features and understanding the overall composition of the image. Further, guided by the global feature information, a preset attention gating mechanism is used to perform weighted fusion of the multi-scale local feature information to obtain a multi-scale attention-enhanced feature pyramid, realizing adaptive complementary fusion of local details and global semantics, and enhancing the ability to express small and weak features. Finally, based on a preset parallel prediction sub-network, the multi-scale attention-enhanced feature pyramid is analyzed to generate the category and location information of defects in the original image, completing the end-to-end mapping from features to detection results, and solving the problem of insufficient sensitivity of traditional methods for identifying small defects.

[0028] In some embodiments, acquiring the original image of the industrial product to be inspected and performing environmental interference suppression processing on the original image to obtain a preprocessed image includes: Step S11: Perform adaptive brightness and contrast equalization processing on the original image to obtain a first intermediate image. The adaptive brightness and contrast equalization processing is an algorithm that dynamically adjusts the pixel grayscale value distribution based on the image's own statistical characteristics to improve visual perception or standardize lighting conditions. Specifically, an algorithm based on Retinex theory is used to process the original image; this algorithm decomposes the image into reflection and illumination components, estimates and compensates for non-uniform illumination components, and performs non-linear transformation on the pixel values ​​of the original image, making the overall brightness distribution of the output image more uniform, while enhancing dark details and suppressing overexposed areas. The processed image is then output as the first intermediate image.

[0029] Step S12: Perform noise suppression processing on the first intermediate image based on a preset hybrid noise suppression algorithm to obtain a second intermediate image. The hybrid noise suppression algorithm refers to an algorithm that combines multiple filtering principles (such as nonlocal similarity and edge preservation) to maximize the preservation of image details while removing noise. Specifically, the first intermediate image is input into a hybrid filtering module; this module may first apply a fast nonlocal mean filter to smooth Gaussian noise using the similarity of nonlocal regions in the image; subsequently, a bilateral filter is applied, which considers both spatial proximity and pixel value similarity, further reducing noise while protecting the sharpness of edges and textures; the image after these two steps is output as the second intermediate image.

[0030] Step S13: Estimate the point spread function of the second intermediate image and apply an image restoration algorithm to generate the preprocessed image.

[0031] The point spread function (PSF) is a mathematical model describing image blurring caused by factors such as optical diffraction, defocusing, or motion in an imaging system. The image restoration algorithm uses the estimated PSF to reverse-process a blurred image to restore a clear image. Specifically, a region containing clear edges or high-frequency textures is selected from the second intermediate image. By analyzing the gradient information of this region or using a blind deconvolution method, a PSF kernel describing the degree of blurring in the image is estimated. Then, the second intermediate image and the estimated PSF kernel are input into an image restoration algorithm, such as Wiener filtering or the Richardson-Lucy iterative algorithm. This algorithm performs deconvolution operations in the frequency or spatial domain to compensate for information loss caused by blurring, outputting a clearer image, which is the final preprocessed image.

[0032] The method provided in this embodiment first performs adaptive brightness and contrast equalization processing on the original image to obtain a first intermediate image, which corrects non-uniform illumination and lays the foundation for standardized subsequent processing. Then, based on a preset hybrid noise suppression algorithm, noise suppression processing is performed on the first intermediate image to obtain a second intermediate image, which removes random noise while protecting key structural details. Finally, point spread function estimation is performed on the second intermediate image and an image restoration algorithm is applied to generate a preprocessed image, which effectively corrects motion or defocus blur and improves image clarity and usability.

[0033] In some embodiments, the multi-scale feature extraction of the preprocessed image based on a preset convolutional neural network to obtain multi-scale local feature information includes: Step S21: Input the preprocessed image into the convolutional neural network for forward propagation.

[0034] Specifically, the preprocessed image generated in step S1 is loaded into memory as a three-dimensional tensor (height (H) × width (W) × number of channels (N)); convolution calculation, nonlinear activation and spatial downsampling operations are performed sequentially according to the preset layer order of the convolutional neural network until the data flows through the entire network.

[0035] Step S22: Simultaneously extract feature maps with decreasing resolution and increasing semantic information from multiple specified convolutional layers of different depths in the convolutional neural network; wherein, the set of extracted feature maps constitutes the multi-scale local feature information.

[0036] Here, the multiple specified convolutional layers of different depths refer to the pre-selected output positions of convolutional layers in the network, located after different downsampling stages. Specifically, during the forward propagation of the network, "hooks" are set at multiple pre-defined specific network layers (e.g., the output ends of convolutional layers located after the first, second, and third downsampling layers, respectively) or their output tensors are directly read; these output tensors are feature maps, which have different spatial dimensions (e.g., 1 / 4, 1 / 8, 1 / 16 of the original image) and different channel dimensions; these feature maps captured at different scales are saved simultaneously, and the set they constitute is defined as multi-scale local feature information.

[0037] The method provided in this embodiment first inputs the preprocessed image into a convolutional neural network for forward propagation, activating each layer of the network to progressively abstract the image information; then, from multiple specified convolutional layers of different depths in the convolutional neural network, feature maps with decreasing resolution and increasing semantic information are extracted simultaneously, resulting in a structured multi-scale feature representation set covering everything from fine local details to high-level semantic concepts.

[0038] In some embodiments, the global analysis of the multi-scale local feature information based on a preset Transformer encoder to obtain global feature information includes: Step S31: Select a feature map of a certain level from the multi-scale local feature information.

[0039] Specifically, based on a predefined strategy (such as selecting a feature map with a moderate number of channels and a resolution at an intermediate level), a specific feature map is selected from the multi-scale local feature information set output in step S22 as the input source of the Transformer encoder.

[0040] Step S32: Serialize the spatial location feature vectors of the selected feature map and input them into the Transformer encoder. Specifically, the selected two-dimensional feature map is flattened in the spatial dimension, that is, its shape is reshaped from [H, W, C] to [N, C], where N = H × W and C is the number of channels; at this time, each vector of length C corresponds to a spatial location in the original feature map, and N vectors constitute a sequence; this sequence is used as input and fed into the preset Transformer encoder.

[0041] Step S33: Utilize the multi-head self-attention mechanism in the Transformer encoder to calculate the correlation weight between any two spatial location feature vectors in the spatial location feature vector sequence, so as to generate the global feature information.

[0042] Specifically, within the Transformer encoder, a multi-head self-attention mechanism performs multiple self-attention computations in parallel. For each computation, a learnable linear transformation converts the feature vector at each position into a query vector, a key vector, and a value vector. By calculating the dot product of all query vectors and all key vectors and applying Softmax normalization, an N×N attention weight matrix is ​​obtained, where each element reflects the correlation strength between two positions. This weight matrix is ​​used to perform a weighted summation of all value vectors, thereby updating the feature representation at each position and incorporating information from all positions in the sequence. After processing through multiple such attention layers and feedforward network layers, the output sequence is reshaped back to the original two-dimensional spatial structure [H, W, C], and this updated feature map is the global feature information.

[0043] The method provided in this embodiment first selects a feature map of a certain level from multi-scale local feature information to determine the input object for global analysis; then, the spatial location feature vector of the selected feature map is serialized and input into the Transformer encoder to transform the spatial structure of the image into sequence data for processing; secondly, the multi-head self-attention mechanism in the Transformer encoder is used to calculate the correlation weight between any two spatial location feature vectors in the spatial location feature vector sequence to generate global feature information, enabling the features of any two points in the image to interact directly and modeling the global contextual dependency relationship.

[0044] In some embodiments, the step of using the global feature information as a guide and performing weighted fusion of the multi-scale local feature information based on a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid includes: Step S41: Transform the global feature information to generate an initial fusion feature that matches the size of the highest semantic layer feature map in the multi-scale local feature information. The highest semantic layer feature map refers to the feature map with the lowest spatial resolution, the most channels, and the highest semantic abstraction in the multi-scale local feature information set. Specifically, the global feature information generated in step S33 is processed through a 1×1 convolutional layer; the number of output channels of this convolutional layer is set to be the same as the number of channels of the highest semantic layer feature map, and by setting an appropriate stride or pooling operation, its output spatial height and width are also made to match those of the highest semantic layer feature map. Figure 1 Therefore, the output of this convolutional layer is the initial fused feature.

[0045] Step S42: For each level of the multi-scale local feature information, the fused features generated at the previous level are upsampled to the resolution of the current level. A spatial weight map is generated using the attention gating mechanism, guided by the global feature information. The spatial weight map is then used to perform a weighted summation of the upsampled fused features and the local feature information of the current level. The spatial weight map is a two-dimensional matrix where each value is between 0 and 1, indicating whether, at the corresponding spatial location, fusion should prioritize higher-level fused features or current-level local features. Specifically, following the order from low resolution (high-level semantics) to high resolution (low-level details), the multi-scale local feature information is traversed at each level. For the current level i, the fused feature generated in step S41 (for the first layer) or the previous iteration is upsampled using a bilinear interpolation algorithm to make its spatial size the same as the local feature map of the current level. Simultaneously, the global feature information (or a representation derived from it and reduced in dimensionality through convolution) is concatenated or element-wise added to the upsampled fused feature. The concatenated or added result is input into an attention gating module consisting of a convolutional layer and a sigmoid activation function. This module outputs a single-channel feature map of the same size as the feature map of the current level, and maps the value of each pixel to the [0,1] interval using the sigmoid function to generate a spatial weight map. Subsequently, a weighted summation is performed: Current layer fusion result = Spatial weight map ⊙ Upsampled fused feature + (1 - Spatial weight map) ⊙ The local feature information of the current layer, where ⊙ represents element-wise multiplication; this result is the fused feature of the current layer, and is used for iteration of the next layer (higher resolution layer).

[0046] Step S43: Determine the set of feature maps at each level generated in the final iteration as the multi-scale attention-enhanced feature pyramid.

[0047] Specifically, after the iterative process of step S42 has processed all levels of the multi-scale local feature information (usually starting from the highest semantic layer with the lowest resolution and ending at the lowest semantic layer with the highest resolution), the fused feature map generated in each iteration is saved. These feature maps are arranged in the order of their corresponding original scales and together form a new set of multi-scale feature maps, which is the multi-scale attention-enhanced feature pyramid.

[0048] The method provided in this embodiment first transforms the global feature information to generate initial fusion features that match the size of the highest semantic layer feature map in the multi-scale local feature information, providing a starting point for the top-down fusion process. Then, for each level of the multi-scale local feature information, the fusion features generated at the previous level are upsampled to the resolution of the current level. A spatial weight map is generated through an attention gating mechanism guided by the global feature information. The spatial weight map is used to perform a weighted summation of the upsampled fusion features and the local feature information of the current level, realizing the dynamic adjustment of the fusion ratio between local details and high-level semantics according to the global semantics. Finally, the set of feature maps generated in the final iteration is determined as a multi-scale attention-enhanced feature pyramid, constructing a unified pyramid structure in which features at each scale are optimized and enhanced by global context information.

[0049] In some embodiments, the analysis of the multi-scale attention-enhanced feature pyramid based on a preset parallel prediction sub-network to generate category and location information of defects in the original image includes: step S51, inputting each feature map of the multi-scale attention-enhanced feature pyramid into the parallel prediction sub-network. Specifically, each layer of feature maps in the multi-scale attention-enhanced feature pyramid generated in step S43 is input sequentially (e.g., from low resolution to high resolution) or in parallel into the input interface of the same parallel prediction sub-network; this sub-network is designed to accept inputs of different spatial sizes but with a fixed number of channels.

[0050] Step S52: The classification subnetwork in the parallel prediction subnetwork outputs a probability distribution vector of the defect category at each location of the feature map. The probability distribution vector is a vector of length (K+1), where K is the number of defect categories, the extra dimension represents the background, and the sum of all elements is 1, representing the probability of belonging to each category. Specifically, for each input feature map layer, the classification subnetwork (usually composed of several convolutional layers) performs convolution calculations on it; in the last convolutional layer, a (K+1)-dimensional real vector is output for each spatial location of the feature map (or each preset anchor center); then, this real vector is normalized using the Softmax function, converting it into a standard probability distribution vector, which is the category prediction result at that location.

[0051] Step S53: The regression subnetwork in the parallel prediction subnetwork outputs geometric offset vectors at various locations on the feature map to adjust the preset bounding boxes. These geometric offset vectors contain several real values ​​(typically four: dx, dy, dw, dh) used to offset the center point and scale the predefined anchor boxes at those locations. Specifically, in parallel with the classification subnetwork, the regression subnetwork (also composed of several convolutional layers) processes the same input feature map; in the last convolutional layer, it outputs a geometric offset vector for each spatial location corresponding to the classification prediction; this vector is associated with one or more anchor boxes pre-set at that location, and the offset is applied to the anchor boxes using formulas (e.g., prediction box center x = anchor box center x + dx × anchor box width, prediction box width = anchor box width × exp(dw)) to obtain the adjusted predicted bounding box coordinates.

[0052] Step S54: Perform non-maximum suppression processing on the preliminary prediction results at all output positions to generate the category and location information of defects in the original image.

[0053] The preliminary prediction results refer to all unfiltered predicted bounding boxes and their corresponding class probabilities generated by steps S52 and S53. Specifically, predicted bounding boxes (with coordinates adjusted using geometric offset vectors) generated at all scales and locations are collected, along with their corresponding highest class confidence scores. This set is used as input to a non-maximum suppression algorithm. The algorithm first discards predictions with confidence scores below a preset threshold. Then, the remaining predictions are grouped by class, and the predicted bounding boxes for each class are sorted in descending order of confidence score. The bounding box with the highest confidence score is selected as the retained result, and its intersection-union ratio (IU) with all other bounding boxes in the same class is calculated. Boxes with IU ratios exceeding a preset overlap threshold are deleted. This process is iterated until all predictions are processed. Finally, the output list of retained bounding boxes, along with their class and confidence score, constitutes the final class and location information of defects in the original image.

[0054] The method provided in this embodiment first inputs the feature maps of the multi-scale attention-enhanced feature pyramid into a parallel prediction sub-network, mapping the optimized features to the detection task space. Then, through the classification sub-network in the parallel prediction sub-network, the probability distribution vector of the defect category is output at each position of the feature map, completing the discrimination of defect existence and category. Second, through the regression sub-network in the parallel prediction sub-network, the geometric offset vector used to adjust the preset bounding box is output at each position of the feature map, completing the fine localization of the defect location. Finally, non-maximum suppression processing is applied to the preliminary prediction results at all output positions to generate the category and location information of the defect in the original image, eliminating redundant detection boxes and outputting a clear and unique list of detection results.

[0055] In some embodiments, performing non-maximum suppression processing on the preliminary prediction results at all output locations to generate the category and location information of defects in the original image includes: Step S541: Based on a preset confidence threshold, select candidate prediction boxes with confidence scores higher than the confidence threshold from all preliminary prediction results. The confidence threshold is a real number between 0 and 1, used to filter low-quality predictions; the candidate prediction boxes refer to the prediction boxes retained after passing the threshold. Specifically, iterate through all preliminary prediction results (each result contains the coordinates of a prediction box and its corresponding highest category confidence score); compare the confidence score of each prediction box with a preset confidence threshold (e.g., 0.05); only retain prediction boxes with confidence scores greater than the threshold, and the set of these retained prediction boxes is marked as candidate prediction boxes.

[0056] Step S542: Group the candidate prediction boxes according to defect categories, and sort the candidate prediction boxes of each category in descending order of confidence score. Specifically, based on the category label associated with each candidate prediction box (i.e., the category corresponding to the highest probability in the probability distribution vector), all candidate prediction boxes are divided into different groups, so that prediction boxes in the same group belong to the same defect category; then, for each category group, all candidate prediction boxes in the group are sorted in descending order of their confidence scores to generate an ordered list.

[0057] Step S543: Select the candidate predicted box with the highest current confidence from the sorted list as the retained result, and calculate the intersection-union ratio (IUR) of the retained result with the remaining candidate predicted boxes in the list. The IUR is the ratio of the intersection area to the union area of ​​two rectangular boxes. Specifically, from the currently processed category sorting list, take the candidate predicted box at the top of the list (i.e., the box with the highest current confidence), remove it from the list, and add it to the final retained result set; calculate the IUR of this retained box with each of the remaining candidate predicted boxes in the current list.

[0058] Step S544: Remove candidate predicted boxes whose intersection-union ratio (IU) exceeds a preset overlap threshold from the list. The preset overlap threshold is a threshold (e.g., 0.5) used to determine whether two boxes are too close to point to the same object. Specifically, each IU calculated in step S543 is compared with the preset overlap threshold; for any candidate predicted box with an IU greater than this threshold, it is removed from the currently processed sorting list.

[0059] Step S545: Iteratively execute the selection, calculation, and deletion steps until all candidate prediction boxes in the list have been processed to generate the category and location information of defects in the original image.

[0060] Specifically, steps S543 and S544 are repeated: after deleting a batch of highly overlapping predicted boxes, if the current sorting list is not empty, the box with the highest confidence is taken from the first position of the new list as the retained result, and its intersection-union ratio with the remaining boxes in the list is calculated, and highly overlapping boxes are continued to be deleted; this iterative process continues until the sorting list of the current category becomes empty; after processing one category, the above process is repeated for the grouping list of the next defect category; when all candidate predicted boxes of all categories have been processed, all predicted boxes and their categories and confidence scores contained in the final retained result set constitute the final category and location information of the defects in the original image.

[0061] The method provided in this embodiment firstly filters out candidate prediction boxes with confidence levels higher than a preset confidence threshold from all preliminary prediction results, eliminating a large number of unreliable predictions and reducing subsequent computation. Then, the candidate prediction boxes are grouped according to defect categories, and each category is sorted in descending order of confidence level, preparing data for independent screening of each defect category. Secondly, the candidate prediction box with the highest current confidence level is selected from the sorted list as the retained result, and the intersection-union ratio (IUR) between the retained result and the remaining candidate prediction boxes in the list is calculated to determine the overlap between the optimal prediction box and its neighboring boxes. Further, candidate prediction boxes with IUR exceeding a preset overlap threshold are deleted from the list, suppressing redundant detection of the same real defect. Finally, the selection, calculation, and deletion steps are iteratively executed until all candidate prediction boxes in the list have been processed, generating the defect category and location information in the original image and outputting a set of non-redundant, high-confidence final detection results.

[0062] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of an industrial vision inspection system based on a deep convolutional neural network, provided as an embodiment of this application. Figure 2 As shown, the industrial vision inspection system 100 based on a deep convolutional neural network includes: The acquisition module 110 is used to acquire the original image of the industrial product to be inspected and to perform environmental interference suppression processing on the original image to obtain a preprocessed image.

[0063] The feature extraction module 120 is used to perform multi-scale feature extraction on the preprocessed image based on a preset convolutional neural network to obtain multi-scale local feature information.

[0064] The first analysis module 130 is used to perform global analysis on the multi-scale local feature information based on a preset Transformer encoder to obtain global feature information.

[0065] The feature fusion module 140 is used to perform weighted fusion of the multi-scale local feature information based on the global feature information and a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid. The second analysis module 150 is used to analyze the multi-scale attention-enhanced feature pyramid based on a preset parallel prediction sub-network to generate the category and location information of defects in the original image.

[0066] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the system and its modules described above can be referred to the corresponding processes in the aforementioned embodiments of the industrial vision inspection method based on deep convolutional neural networks, and will not be repeated here.

[0067] The industrial vision inspection system 100 based on deep convolutional neural networks provided in the above embodiments can be implemented as a computer program, which can be used in various ways, such as... Figure 3 The terminal device 200 shown is running on it.

[0068] Please see Figure 3 , Figure 3 The following is a schematic block diagram of the structure of a terminal device 200 provided in an embodiment of this application. The terminal device 200 includes a processor 201 and a memory 202, which are connected through a system bus 203. The memory 202 may include a non-volatile storage medium and internal memory.

[0069] The non-volatile storage medium can store a computer program. The computer program includes program instructions that, when executed by the processor 201, cause the processor 201 to perform any of the aforementioned industrial vision inspection methods based on deep convolutional neural networks.

[0070] The processor 201 provides computing and control capabilities to support the operation of the entire terminal device 200.

[0071] The internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor 201, the processor 201 can execute any of the above-mentioned industrial vision inspection methods based on deep convolutional neural networks.

[0072] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal device 200 involved in the present application. The specific terminal device 200 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0073] It should be understood that processor 201 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, the general-purpose processor can be a microprocessor or any conventional processor.

[0074] In some embodiments, the processor 201 is configured to run a computer program stored in memory to perform the following steps: The original image of the industrial product to be inspected is acquired, and environmental interference suppression processing is performed on the original image to obtain a preprocessed image; Multi-scale feature extraction is performed on the preprocessed image based on a preset convolutional neural network to obtain multi-scale local feature information; The multi-scale local feature information is analyzed globally based on the preset Transformer encoder to obtain global feature information; Guided by the global feature information, the multi-scale local feature information is weighted and fused based on a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid; the multi-scale attention-enhanced feature pyramid is analyzed based on a preset parallel prediction sub-network to generate the category and location information of defects in the original image.

[0075] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the terminal device 200 described above can be referred to the corresponding process of the aforementioned industrial vision inspection method based on deep convolutional neural networks, and will not be repeated here.

[0076] This application also provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, causes the one or more processors to implement the industrial vision inspection method based on a deep convolutional neural network as provided in this application.

[0077] The computer-readable storage medium can be an internal storage unit of the terminal device 200 in the aforementioned embodiments, such as a hard disk or memory of the terminal device 200. The computer-readable storage medium can also be an external storage device of the terminal device 200, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided with the terminal device 200.

[0078] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0079] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0080] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An industrial visual inspection method based on a deep convolutional neural network, characterized in that, include: The original image of the industrial product to be inspected is acquired, and environmental interference suppression processing is performed on the original image to obtain a preprocessed image; Multi-scale feature extraction is performed on the preprocessed image based on a preset convolutional neural network to obtain multi-scale local feature information; The multi-scale local feature information is analyzed globally based on the preset Transformer encoder to obtain global feature information; Guided by the global feature information, the multi-scale local feature information is weighted and fused based on a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid; the multi-scale attention-enhanced feature pyramid is analyzed based on a preset parallel prediction sub-network to generate the category and location information of defects in the original image.

2. The industrial visual inspection method based on deep convolutional neural networks according to claim 1, characterized in that, The process of acquiring the original image of the industrial product to be inspected and performing environmental interference suppression processing on the original image to obtain a preprocessed image includes: The original image is subjected to adaptive brightness and contrast equalization processing to obtain a first intermediate image; the first intermediate image is subjected to noise suppression processing based on a preset hybrid noise suppression algorithm to obtain a second intermediate image; the second intermediate image is subjected to point spread function estimation and an image restoration algorithm is applied to generate the preprocessed image.

3. The industrial visual inspection method based on deep convolutional neural networks according to claim 1, characterized in that, The preprocessed image is subjected to multi-scale feature extraction based on a preset convolutional neural network to obtain multi-scale local feature information, including: The preprocessed image is input into the convolutional neural network for forward propagation; Feature maps with decreasing resolution and increasing semantic information are extracted synchronously from multiple specified convolutional layers of different depths in the convolutional neural network; wherein the set of extracted feature maps constitutes the multi-scale local feature information.

4. The industrial visual inspection method based on a deep convolutional neural network according to claim 1, characterized in that, The preset Transformer encoder performs global analysis on the multi-scale local feature information to obtain global feature information, including: Select a feature map of one level from the multi-scale local feature information; The spatial location feature vectors of the selected feature map are serialized and input into the Transformer encoder; the multi-head self-attention mechanism in the Transformer encoder is used to calculate the correlation weight between any two spatial location feature vectors in the spatial location feature vector sequence to generate the global feature information.

5. The industrial visual inspection method based on a deep convolutional neural network according to claim 1, characterized in that, Guided by the global feature information, the multi-scale local feature information is weighted and fused based on a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid, including: The global feature information is transformed to generate an initial fusion feature that matches the size of the highest semantic layer feature map in the multi-scale local feature information. For each level of the multi-scale local feature information, the fusion feature generated at the previous level is upsampled to the resolution of the current level, and a spatial weight map is generated through the attention gating mechanism guided by the global feature information. The upsampled fusion feature and the local feature information of the current level are weighted and summed using the spatial weight map. The set of feature maps generated in the final iteration is determined as the multi-scale attention-enhanced feature pyramid.

6. The industrial visual inspection method based on a deep convolutional neural network according to claim 1, characterized in that, The method of analyzing the multi-scale attention-enhanced feature pyramid based on a preset parallel prediction sub-network to generate the category and location information of defects in the original image includes: inputting each feature map of the multi-scale attention-enhanced feature pyramid into the parallel prediction sub-network; outputting the probability distribution vector of the defect category at each position of the feature map through the classification sub-network in the parallel prediction sub-network; outputting the geometric offset vector for adjusting the preset bounding box at each position of the feature map through the regression sub-network in the parallel prediction sub-network; and performing non-maximum suppression processing on the preliminary prediction results at all output positions to generate the category and location information of defects in the original image.

7. The industrial visual inspection method based on a deep convolutional neural network according to claim 6, characterized in that, The step of performing non-maximum suppression processing on the preliminary prediction results at all output locations to generate the category and location information of defects in the original image includes: Based on a preset confidence threshold, candidate prediction boxes with confidence levels higher than the threshold are selected from all preliminary prediction results. These candidate prediction boxes are then grouped by defect category, and sorted in descending order of confidence level for each category. The candidate prediction box with the highest current confidence level is selected from the sorted list as the retained result, and the intersection-union ratio (IUR) between the retained result and the remaining candidate prediction boxes in the list is calculated. Candidate prediction boxes with IUR exceeding a preset overlap threshold are deleted from the list. The selection, calculation, and deletion steps are iteratively executed until all candidate prediction boxes in the list have been processed to generate the defect category and location information in the original image.

8. An industrial vision inspection system based on a deep convolutional neural network, characterized in that, include: The acquisition module is used to acquire the original image of the industrial product to be inspected, and to perform environmental interference suppression processing on the original image to obtain a preprocessed image; The feature extraction module is used to perform multi-scale feature extraction on the preprocessed image based on a preset convolutional neural network to obtain multi-scale local feature information; The first analysis module is used to perform global analysis on the multi-scale local feature information based on a preset Transformer encoder to obtain global feature information; The feature fusion module is used to perform weighted fusion of the multi-scale local feature information based on the global feature information and a preset attention gating mechanism to obtain a multi-scale attention-enhanced feature pyramid. The second analysis module is used to analyze the multi-scale attention-enhanced feature pyramid based on a preset parallel prediction sub-network to generate the category and location information of defects in the original image.

9. A terminal device, characterized in that, The terminal device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the industrial vision inspection method based on a deep convolutional neural network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the industrial vision inspection method based on a deep convolutional neural network as described in any one of claims 1 to 7.

Images