Carton die-cutting indentation breakage defect visual detection method based on deep learning

Abstract

The present application belongs to the technical field of quality detection, and relates to a paper box die-cutting indentation damage defect visual detection method based on deep learning. A gray-scale image of a die-cutting indentation area under a preset illumination condition is input into a deep convolution network to extract a hierarchical feature map, a spatial weight map is generated, the spatial weight map is adjusted to the size corresponding to each hierarchical feature map through upsampling, and each hierarchical feature map after enhancement is obtained by element-by-element multiplication, resolution upsampling is performed to obtain a fusion feature map, a deformable pooling operation is performed on the area covered by each anchor frame to obtain a fixed-length feature vector, and the fixed-length feature vector is input into a defect category classification branch and a position regression branch respectively, and defect category prediction and position offset prediction are output. The present application effectively improves the detection accuracy and positioning accuracy of die-cutting indentation damage, fracture and insufficient depth and the like, reduces missed detection and false positives caused by complex illumination and material differences, and significantly enhances the automation level and reliability of paper box quality detection in an industrial field.

Description

Technical Field

[0001] This invention belongs to the field of quality inspection technology and relates to a deep learning-based visual inspection method for paper box die-cutting creasing and damage defects. Background Technology

[0002] In the production of cardboard packaging, the die-cutting and creasing process is a crucial step in determining the quality of the formed cardboard box. Defects such as damage, breakage, or insufficient creasing depth in the die-cutting and creasing area directly affect the folding accuracy of the cardboard box and the structural strength of the final product. Traditional detection of die-cutting and creasing defects mainly relies on manual visual inspection or automated optical inspection systems based on traditional image processing algorithms. Manual inspection is inefficient and easily affected by subjective factors, while traditional image processing algorithms are usually based on fixed edge detection operators, threshold segmentation, or morphological operations. These methods lack adaptability to changes in lighting, differences in cardboard materials, and the diversity of creasing patterns. When creasing defects manifest as subtle grayscale changes or irregular geometric deformations, traditional algorithms often fail to accurately identify them, leading to high false negative and false positive rates.

[0003] In recent years, deep learning technology, especially convolutional neural networks, has received widespread attention in the field of industrial defect detection. Some methods input the entire cardboard box image into a deep network for end-to-end defect classification or target detection.

[0004] However, most existing deep learning-based detection schemes directly apply general object detection network structures without fully considering the unique characteristics of die-cutting indentation defects. Die-cutting indentation regions appear as elongated, low-contrast linear structures in images, with a wide spatial scale, including long strip-shaped areas extending along the crease direction and small clustered areas at corners. General feature pyramid or multi-scale feature fusion strategies often employ fixed upsampling or interpolation methods, lacking adaptive differentiation of the spatial importance of the indentation region. This leads to the loss of detailed information in low-resolution feature maps and insufficient semantic information in high-resolution feature maps, making it difficult to accurately locate and classify indentation defects. Furthermore, existing methods typically use regular grids in anchor frame setting and feature sampling, ignoring the geometric deformation and positional shifts that may occur in the image, making it difficult for the anchor frame to accurately cover the defect area and affecting the final detection accuracy. Summary of the Invention

[0005] To address the aforementioned issues, a visual inspection method specifically designed for die-cutting and creasing defects in paper boxes is needed. This method should adaptively highlight discriminative spatial locations within the creasing area, employ a reasonable multi-scale feature fusion strategy that balances global semantic information with local detail features, utilize a deformable sampling mechanism to adapt to the geometric deformation of creasing defects, and perform effective confidence screening and overlap suppression on the detection results during post-processing. This will enable stable and accurate detection of various die-cutting and creasing defects under varying lighting and material conditions in industrial settings, thereby improving the automation level and reliability of paper box production quality control.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] A deep learning-based visual detection method for die-cutting creasing and tear defects in cardboard boxes includes the following steps:

[0008] S1, Obtain a grayscale image of the die-cutting and crease area of ​​the cardboard box to be inspected under preset lighting conditions;

[0009] S2, the grayscale image is input into a deep convolutional network, which includes multiple convolutional modules connected in sequence. Each convolutional module outputs a hierarchical feature map of a corresponding resolution, thereby obtaining at least three hierarchical feature maps of different resolutions.

[0010] S3, Select the feature map of the lowest resolution level, perform spatial attention extraction operation on the feature map of the lowest resolution level to obtain a spatial weight map, the spatial size of the spatial weight map is the same as the spatial size of the feature map of the lowest resolution level;

[0011] S4, the spatial weight map is adjusted to the spatial size of the feature maps of the other levels through an upsampling operation, and the adjusted spatial weight map is multiplied element by element with the corresponding level feature map to obtain the feature maps of each enhancement level.

[0012] S5. Perform resolution upsampling operation on the feature maps of each enhancement level to make all feature maps of enhancement levels have the same spatial size. Then, add the feature maps of each enhancement level with the same spatial size element by element to obtain the fused feature map.

[0013] S6, set multiple anchor boxes on the fused feature map, perform deformable pooling operation on the fused feature map region covered by each anchor box to obtain a fixed-length feature vector corresponding to each anchor box, input the fixed-length feature vector into the defect category classification branch and the location regression branch respectively, and output the defect category prediction value and location offset prediction value corresponding to each anchor box.

[0014] Furthermore, the deep convolutional network is a residual network, and the at least three hierarchical feature maps of different resolutions are respectively the feature map output by the third-to-last residual stage, the feature map output by the second-to-last residual stage, and the feature map output by the last residual stage in the residual network.

[0015] Furthermore, the spatial attention extraction operation includes:

[0016] Global average pooling is performed on the feature map of the lowest resolution layer to obtain a first pooling descriptor, and global max pooling is performed on the feature map of the lowest resolution layer to obtain a second pooling descriptor.

[0017] The first pooling descriptor and the second pooling descriptor are concatenated along the channel dimension to obtain the concatenated descriptor;

[0018] The concatenated descriptor is input into a convolutional layer. The kernel size of the convolutional layer is a preset size, and the number of output channels is a single channel. The single-channel feature map output by the convolutional layer is processed by a non-linear activation function and used as the spatial weight map.

[0019] Furthermore, the process of performing deformable pooling on the fused feature map region covered by each anchor frame specifically includes:

[0020] For each anchor frame, based on the coordinates of the top-left vertex, the width, and the height of the anchor frame on the fused feature map, an initial sampling grid is constructed according to a preset number of sampling rows and columns. The x-coordinate of each sampling point in the initial sampling grid is the x-coordinate of the top-left vertex plus the product of the column where the sampling point is located and the column spacing.

[0021] The fused feature map sub-region covered by the anchor frame is input into the offset prediction network, which includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence.

[0022] Add the x-coordinate of each sampling point in the initial sampling grid to the corresponding x-coordinate offset to obtain the adjusted x-coordinate; add the y-coordinate of each sampling point to the corresponding y-coordinate offset to obtain the adjusted y-coordinate.

[0023] For each adjusted sample point coordinate, determine the positions of the four adjacent integer coordinates on the fused feature map;

[0024] Read the four feature values ​​corresponding to the four integer coordinate positions respectively. Determine the horizontal weight based on the horizontal distance between the adjusted sampling point's horizontal coordinate and the horizontal coordinates of each integer coordinate. Determine the vertical weight based on the vertical distance between the adjusted sampling point's vertical coordinate and the vertical coordinates of each integer coordinate. Multiply the four feature values ​​by their corresponding horizontal and vertical weights respectively and sum them to obtain the interpolation feature value of the adjusted sampling point.

[0025] Perform global average pooling on the interpolated feature values ​​of all adjusted sampling points to obtain the fixed-length feature vector.

[0026] Furthermore, the defect category classification branch includes a first fully connected layer and a classification output layer. The first fully connected layer maps the fixed-length feature vector to intermediate classification features. The classification output layer uses a preset activation function to convert the intermediate classification features into confidence probabilities of each defect category, which are used as the predicted values ​​of the defect categories.

[0027] The location regression branch includes a second fully connected layer and a regression output layer. The second fully connected layer maps the fixed-length feature vector to intermediate regression features, and the regression output layer outputs a coordinate offset of a preset dimension as the predicted location offset value.

[0028] Furthermore, after outputting the defect category prediction value and location offset prediction value corresponding to each anchor box, the following operations are also performed:

[0029] Based on the initial top-left vertex coordinates, anchor width, anchor height, and corresponding position offset prediction values ​​of each anchor frame, calculate the top-left and bottom-right vertex coordinates of the predicted bounding box corresponding to each anchor frame.

[0030] Sort all the predicted bounding boxes corresponding to the anchor boxes from highest to lowest confidence level in their defect category prediction values ​​to obtain the sorted sequence of predicted bounding boxes.

[0031] Select the highest-confidence predicted bounding box from the sorted predicted bounding box sequence as the retained bounding box, add the retained bounding box to the final result set, and calculate the intersection-union ratio (IUR) between the retained bounding box and the remaining predicted bounding boxes in the sorted predicted bounding box sequence. Remove the remaining predicted bounding boxes with IUR values ​​greater than a preset IUR threshold from the sequence.

[0032] Repeat the process of selecting the predicted bounding box with the highest confidence as the retained bounding box and removing overlapping predicted bounding boxes until the sorted sequence of predicted bounding boxes is empty; output the defect category and location corresponding to each retained bounding box in the final result set as the final detection result.

[0033] Furthermore, the step of performing resolution upsampling on the feature maps of each enhancement level to ensure that all feature maps of enhancement levels have the same spatial size specifically includes:

[0034] Traverse all enhancement level feature maps, read the current height value and current width value of each enhancement level feature map, select the maximum height value from all current height values ​​as the target height value, select the maximum width value from all current width values ​​as the target width value, and determine the target height value and target width value as the target spatial size;

[0035] For each enhancement level feature map, the ratio of the target height value to the current height value of the enhancement level feature map is calculated as the height upsampling factor, and the ratio of the target width value to the current width value of the enhancement level feature map is calculated as the width upsampling factor.

[0036] Perform a transposed convolution operation in the height direction on the enhanced layer feature map. The number of input channels of the transposed convolution operation in the height direction is equal to the current number of channels of the enhanced layer feature map, and the number of output channels is a preset number of intermediate channels.

[0037] A transpose convolution operation in the width direction is performed on the feature map after transpose convolution in the height direction. The number of input channels of the transpose convolution operation in the width direction is equal to the preset number of intermediate channels, and the number of output channels is the preset number of uniform channels.

[0038] After performing transpose convolution operations in the height and width directions, an edge cropping operation is performed on the feature map. The edge cropping operation removes the additional pixels on the edges caused by the transpose convolution filling, so that the spatial size of the feature map is exactly equal to the target height and target width values.

[0039] For each enhanced layer feature map, the above height-direction transposed convolution, width-direction transposed convolution, and edge clipping operations are performed to obtain each upsampled feature map. The spatial dimensions of each upsampled feature map are the target height value and the target width value, and the number of channels is the preset unified number of channels.

[0040] The sampled feature maps are added element by element in the spatial dimension to obtain the added feature map. The added feature map is then processed by a preset nonlinear activation function to obtain the fused feature map.

[0041] Furthermore, the preset size and preset aspect ratio of the multiple anchor frames are set according to the geometry of the die-cutting and creasing. The preset size includes multiple different size levels, each size level corresponding to a different anchor frame area range. The preset aspect ratio includes the elongated ratio corresponding to the straight line segment of the die-cutting and creasing and the square ratio corresponding to the corner of the die-cutting and creasing. The setting position of each anchor frame on the fused feature map is centered on each pixel of the fused feature map, and multiple anchor frames of different shapes are generated according to the preset size and preset aspect ratio.

[0042] Furthermore, the preset lighting conditions include using a coaxial light source to provide vertical illumination and using a ring light source to provide oblique illumination. The coaxial light source and the ring light source are turned on simultaneously, and the ratio between the light intensity of the coaxial light source and the light intensity of the ring light source is set to a preset ratio. When acquiring the grayscale image, the camera optical axis is perpendicular to the plane where the die-cutting and creasing area of ​​the cardboard box to be inspected is located, the camera exposure time is set to a preset exposure time, and the camera gain is set to a preset gain value.

[0043] This invention belongs to the field of quality inspection technology and relates to a deep learning-based visual detection method for die-cutting creases and damage defects in cardboard boxes. The method involves acquiring a grayscale image of the die-cutting crease area under preset lighting conditions and inputting it into a deep convolutional network to extract hierarchical feature maps, generating spatial weight maps. The spatial weight maps are then upsampled to the corresponding dimensions of each hierarchical feature map and multiplied element-wise to obtain enhanced hierarchical feature maps. Resolution upsampling is then performed to obtain a fused feature map. Deformable pooling is performed on the area covered by each anchor frame to obtain a fixed-length feature vector, which is then input into the defect category classification branch and the location regression branch, respectively, outputting defect category prediction and location offset prediction. This invention effectively improves the detection accuracy and positioning accuracy of defects such as die-cutting creases, breaks, and insufficient depth, while reducing missed detections and false alarms caused by complex lighting and material differences, significantly enhancing the automation level and reliability of cardboard box quality inspection in industrial settings. Attached Figure Description

[0044] Figure 1 A flowchart illustrating the workflow of a deep learning-based visual detection method for die-cutting creasing and damage defects in paper boxes, as claimed in an embodiment of the present invention.

[0045] Figure 2 The second flowchart is shown for a deep learning-based visual detection method for die-cutting creasing and damage defects in paper boxes, as claimed in an embodiment of the present invention.

[0046] Figure 3 The third flowchart is a visual inspection method for paper box die-cutting creasing and damage defects based on deep learning, as claimed in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0048] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0049] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0050] According to the first embodiment of the present invention, referring to Figure 1 This invention claims protection for a deep learning-based visual detection method for die-cutting creases and damage defects in cardboard boxes, comprising the following steps:

[0051] S1, Obtain a grayscale image of the die-cutting and crease area of ​​the cardboard box to be inspected under preset lighting conditions;

[0052] S2, the grayscale image is input into a deep convolutional network, which includes multiple convolutional modules connected in sequence. Each convolutional module outputs a hierarchical feature map of a corresponding resolution, thereby obtaining at least three hierarchical feature maps of different resolutions.

[0053] S3, Select the feature map of the lowest resolution level, perform spatial attention extraction operation on the feature map of the lowest resolution level to obtain a spatial weight map, the spatial size of the spatial weight map is the same as the spatial size of the feature map of the lowest resolution level;

[0054] S4, the spatial weight map is adjusted to the spatial size of the feature maps of the other levels through an upsampling operation, and the adjusted spatial weight map is multiplied element by element with the corresponding level feature map to obtain the feature maps of each enhancement level.

[0055] S5. Perform resolution upsampling operation on the feature maps of each enhancement level to make all feature maps of enhancement levels have the same spatial size. Then, add the feature maps of each enhancement level with the same spatial size element by element to obtain the fused feature map.

[0056] S6, set multiple anchor boxes on the fused feature map, perform deformable pooling operation on the fused feature map region covered by each anchor box to obtain a fixed-length feature vector corresponding to each anchor box, input the fixed-length feature vector into the defect category classification branch and the location regression branch respectively, and output the defect category prediction value and location offset prediction value corresponding to each anchor box.

[0057] In this embodiment, a batch of corrugated cardboard boxes to be inspected is used as the inspection object. The die-cutting and crease areas of this batch of cardboard boxes are located at the crease lines of the cardboard boxes, and the crease lines are approximately 2 mm wide. The inspection system includes an industrial area scan camera with a resolution of 2048 pixels × 2048 pixels, two sets of LED light sources, and an industrial control computer equipped with a graphics processor. The camera is installed directly above the cardboard box conveyor line, with the camera's optical axis perpendicular to the surface of the cardboard box. The two sets of LED light sources are respectively positioned on both sides of the camera, illuminating the surface of the cardboard box at a 45-degree angle, creating a clear contrast between light and dark in the die-cutting and crease areas.

[0058] Before the inspection begins, the conveyor line transports the cardboard box to the area below the camera. After triggering the position sensor, the camera captures an 8-bit grayscale image with a resolution of 1024 pixels × 1024 pixels. In this grayscale image, the die-cutting and crease lines appear as a continuous, dark line, while the background cardboard area appears as a uniform light gray. The grayscale value of each pixel in the image ranges from 0 to 255. If the crease lines have defects such as damage, breakage, or insufficient crease depth, the grayscale value at the corresponding location will differ from the grayscale value of the normal crease area.

[0059] After acquiring the grayscale image, the computer inputs it into a pre-constructed deep convolutional network. This deep convolutional network employs a residual network structure, with its main body containing five residual stages, each consisting of multiple stacked residual modules. The grayscale image first passes through a convolutional layer with a kernel size of 7×7 and a stride of 2, followed by a 3×3 max-pooling layer, and then sequentially through the five residual stages. The resolution of the feature map output by each residual stage decreases progressively, while the number of channels increases progressively. In the fifth, fourth, and third residual stages... After the residual stage, three hierarchical feature maps are extracted respectively; the hierarchical feature map output by the fifth residual stage has a resolution of 32 pixels × 32 pixels and 2048 channels. This feature map contains the highest level of semantic information and can reflect the overall shape and positional distribution of the indentation area; the hierarchical feature map output by the fourth residual stage has a resolution of 64 pixels × 64 pixels and 1024 channels; the hierarchical feature map output by the third residual stage has a resolution of 128 pixels × 128 pixels and 512 channels; thus, three hierarchical feature maps with different resolutions are obtained.

[0060] The lowest-resolution feature map is selected, specifically a 32×32 pixel feature map with 2048 channels. Spatial attention extraction is then performed on this feature map. Specifically, the 32×32×2048 feature map is input into a global average pooling layer and a global max pooling layer. The global average pooling layer performs an averaging operation on the feature map in the spatial dimension; that is, for each channel, it calculates the arithmetic mean of the feature values ​​at 32×32 spatial locations, resulting in a pooling descriptor composed of 2048 averages. The global max pooling layer performs a maximizing operation on the feature map in the spatial dimension; that is, for each channel, it finds the maximum feature value among the 32×32 spatial locations, resulting in a pooling descriptor composed of 2048 maximum values. The 2048-dimensional descriptor output by global average pooling and the 2048-dimensional descriptor output by global max pooling are concatenated along the channel dimension to obtain a concatenated 4096-dimensional descriptor. This 4096-dimensional descriptor is then input into a convolutional layer with a kernel size of 1×1 and an output channel of 1. The convolutional layer performs a convolution operation on the concatenated descriptor and outputs a 32×32×1 spatial weight map. Each spatial position in this spatial weight map corresponds to a spatial position on the original 32×32 feature map, and its value represents the importance of that position in subsequent processing.

[0061] The aforementioned spatial weight map is adjusted to the spatial size of the feature maps of other levels through an upsampling operation. Specifically, bilinear interpolation upsampling is performed on the 32×32 spatial weight map to enlarge it to a spatial size of 64 pixels × 64 pixels, resulting in the first adjusted weight map. Then, bilinear interpolation upsampling is performed on this spatial weight map to enlarge it to a spatial size of 128 pixels × 128 pixels, resulting in the second adjusted weight map. Finally, the first adjusted weight map is multiplied element-wise with the 64×64 resolution hierarchical feature map, resulting in the first adjusted weight map. The weight value at coordinate (i,j) is multiplied by the feature values ​​of all channels at the same coordinate (i,j) in the 64×64 hierarchical feature map to obtain the enhanced 64×64 hierarchical feature map. Similarly, the second adjusted weight map is multiplied element-wise by the hierarchical feature map with a resolution of 128×128 to obtain the enhanced 128×128 hierarchical feature map. For the lowest resolution 32×32 hierarchical feature map, it is directly multiplied element-wise by the original 32×32 spatial weight map to obtain the enhanced 32×32 hierarchical feature map. Thus, three enhanced hierarchical feature maps are obtained.

[0062] The three enhanced layer feature maps are upsampled to ensure that all enhanced layer feature maps have the same spatial size, i.e., 128 pixels × 128 pixels. For a 32×32 enhancement layer feature map, its height and width need to be enlarged by a factor of 4, and upsampling is achieved using transposed convolution with a kernel size of 4×4 and a stride of 4. For a 64×64 enhancement layer feature map, its height and width need to be enlarged by a factor of 2, and upsampling is achieved using transposed convolution with a kernel size of 2×2 and a stride of 2. For a 128×128 enhancement layer feature map, its resolution is already the target size, and no upsampling is required. After upsampling, the spatial size of the three enhancement layer feature maps is 128 pixels × 128 pixels, and the number of channels is adjusted to a uniform 256 channels through transposed convolution. These three 128×128×256 feature maps are then added element-wise in the spatial dimension. That is, for each spatial location and each channel, the three feature values ​​of the corresponding position and corresponding channel in the three feature maps are added together to obtain a 128×128×256 fused feature map.

[0063] Multiple anchor boxes are set on the fused feature map. Three different sizes and three different aspect ratios are set centered on each pixel of the fused feature map; the three sizes correspond to 1, 2, and 4 times the width of the indentation line, respectively, and the three aspect ratios correspond to 1:1, 1:2, and 2:1, respectively; for each spatial location on the fused feature map, 9 anchor boxes are generated, resulting in a total of 147,456 anchor boxes generated on the entire 128×128 fused feature map. For each anchor box, determine the rectangular region it covers on the fused feature map. For each region covered by the anchor box, perform deformable pooling. Set the number of sampling rows and columns to 7, meaning 49 points are sampled for each anchor box region. Calculate the initial positions of the 49 sampling points based on the coordinates of the top-left vertex, width, and height of the anchor box. Then, use an offset prediction network to predict the horizontal and vertical offsets for each sampling point. Add the offsets to the initial positions to obtain the adjusted sampling positions. For each adjusted sampling position, use bilinear interpolation to obtain the feature values. Then, perform global average pooling on the 49 interpolated feature values ​​to obtain a fixed-length feature vector. This fixed-length feature vector contains the local feature information of the region covered by the anchor box.

[0064] The fixed-length feature vector is input into the defect category classification branch and the location regression branch, respectively. The defect category classification branch contains a fully connected layer that maps the fixed-length feature vector into a score vector containing three types of defects: indentation damage, indentation fracture, and insufficient indentation depth. The location regression branch contains another fully connected layer that maps the fixed-length feature vector into four coordinate offsets, corresponding to the horizontal offset of the top left corner of the anchor frame, the vertical offset of the top left corner, the width offset, and the height offset, respectively. The defect category classification branch outputs the confidence probability values ​​of the three categories for each anchor frame, and the location regression branch outputs the predicted values ​​of the four location offsets for each anchor frame.

[0065] Furthermore, the deep convolutional network is a residual network, and the at least three hierarchical feature maps of different resolutions are respectively the feature map output by the third-to-last residual stage, the feature map output by the second-to-last residual stage, and the feature map output by the last residual stage in the residual network.

[0066] In this embodiment, the deep convolutional network specifically adopts the ResNet-50 residual network structure. The ResNet-50 network includes an initial convolutional layer, a max pooling layer, and four residual stages, namely conv2_x, conv3_x, conv4_x, and conv5_x. Each residual stage contains multiple bottleneck residual modules, each of which is composed of a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer connected sequentially. Each residual module has skip connections, directly adding the module's input to the module's output. After the grayscale image is input, it passes through an initial 7×7 convolutional layer and a 3×3 max pooling layer, reducing the feature map resolution to 256 pixels × 256 pixels. Subsequently, it passes through the conv2_x stage (containing 3 residual modules), the conv3_x stage (containing 4 residual modules), the conv4_x stage (containing 6 residual modules), and the conv5_x stage (containing...). (Contains 3 residual modules); After the conv5_x stage, the feature map resolution is reduced to 32 pixels × 32 pixels, and the number of channels increases to 2048; The feature map output from the conv5_x stage is selected as the lowest resolution layer feature map, with a resolution of 32 × 32 and a number of channels of 2048; The feature map output from the conv4_x stage is selected as the intermediate resolution layer feature map, with a resolution of 64 × 64 and a number of channels of 1024; The feature map output from the conv3_x stage is selected as the highest resolution layer feature map, with a resolution of 128 × 128 and a number of channels of 512; These three layer feature maps correspond to the outputs of the last residual stage (conv5_x), the second to last residual stage (conv4_x), and the third to last residual stage (conv3_x) in the residual network, respectively.

[0067] Furthermore, the spatial attention extraction operation includes:

[0068] Global average pooling is performed on the feature map of the lowest resolution layer to obtain a first pooling descriptor, and global max pooling is performed on the feature map of the lowest resolution layer to obtain a second pooling descriptor.

[0069] The first pooling descriptor and the second pooling descriptor are concatenated along the channel dimension to obtain the concatenated descriptor;

[0070] The concatenated descriptor is input into a convolutional layer. The kernel size of the convolutional layer is a preset size, and the number of output channels is a single channel. The single-channel feature map output by the convolutional layer is processed by a non-linear activation function and used as the spatial weight map.

[0071] In this embodiment, the lowest resolution layer feature map is 32 pixels × 32 pixels with 2048 channels. First, global average pooling is performed on this feature map. The global average pooling operation is as follows: for each of the 2048 channels of the feature map, the feature values ​​at 32 rows × 32 columns (a total of 1024 positions) are summed, then divided by 1024 to obtain an average value. After calculating the average value for each of the 2048 channels, 2048 average values ​​are obtained. These average values ​​are arranged in channel order to form the first pooling descriptor, which has a dimension of 2048. Global max pooling is performed on the feature map. The operation is as follows: for each of the 2048 channels in the feature map, the feature values ​​at 32 rows × 32 columns (1024 positions) are traversed, and the maximum value is found. After finding the maximum value for each of the 2048 channels, 2048 maximum values ​​are obtained. These maximum values ​​are arranged in channel order to form the second pooling descriptor, which also has a dimension of 2048. The first and second pooling descriptors are then concatenated along the channel dimension. The specific concatenation method is: the 2048 values ​​of the first pooling descriptor are used as... The first 2048 channels are used as the last 2048 channels, and the 2048 values ​​of the second pooling descriptor are merged to form a 4096-dimensional concatenated descriptor. This 4096-dimensional concatenated descriptor is then input into a convolutional layer. This convolutional layer has a 1×1 kernel size, 4096 input channels, and 1 output channel. The convolutional layer performs a 1×1 convolution operation on the concatenated descriptor: for each value in the 4096 channels of the concatenated descriptor, it multiplies by the corresponding convolutional kernel weight, and then adds the 4096 products to obtain an output value. Since the output channel is 1... The convolutional layer outputs a 32×32 single-channel feature map, which is the original form of the spatial weight map. This single-channel feature map is then input to the Sigmoid non-linear activation function. The Sigmoid function maps each input value to a value between 0 and 1. Specifically, for an input value x, the output value is 1 divided by (1 plus e to the power of negative x). After processing by the Sigmoid function, the final spatial weight map is obtained, which still has a spatial size of 32×32 and 1 channel. The weight value of each spatial location is between 0 and 1, and the closer the weight value is to 1, the more important the location is.

[0072] Furthermore, referring to Figure 2 The process of performing deformable pooling on the fused feature map region covered by each anchor frame specifically includes:

[0073] For each anchor box, based on the coordinates of the top-left vertex, the width, and the height of the anchor box on the fused feature map, an initial sampling grid is constructed according to the preset number of sampling rows and columns. The x-coordinate of each sampling point in the initial sampling grid is the x-coordinate of the top-left vertex plus the product of the column where the sampling point is located and the column spacing.

[0074] The fused feature map sub-region covered by the anchor box is input into the offset prediction network, which includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence.

[0075] Add the x-coordinate of each sampling point in the initial sampling grid to the corresponding x-coordinate offset to obtain the adjusted x-coordinate; add the y-coordinate of each sampling point to the corresponding y-coordinate offset to obtain the adjusted y-coordinate.

[0076] For each adjusted sample point coordinate, determine the positions of the four adjacent integer coordinates on the fused feature map;

[0077] Read the four feature values ​​corresponding to the four integer coordinate positions respectively. Determine the horizontal weight based on the horizontal distance between the adjusted sampling point's horizontal coordinate and the horizontal coordinates of each integer coordinate. Determine the vertical weight based on the vertical distance between the adjusted sampling point's vertical coordinate and the vertical coordinates of each integer coordinate. Multiply the four feature values ​​by their corresponding horizontal and vertical weights respectively and sum them to obtain the interpolation feature value of the adjusted sampling point.

[0078] Perform global average pooling on the interpolated feature values ​​of all adjusted sampling points to obtain a fixed-length feature vector.

[0079] In this embodiment, deformable pooling is performed on the fused feature map region covered by each anchor frame. Taking a specific anchor frame on the fused feature map as an example, it is assumed that the top-left corner coordinates of the anchor frame on the fused feature map are (200, 150), the anchor frame width is 64 pixels, and the anchor frame height is 32 pixels. The preset number of sampling rows is set to 7, and the preset number of sampling columns is set to 7, meaning a total of 49 points are sampled. First, an initial sampling grid is constructed. For the sampling point in the i-th row and j-th column (where i ranges from 0 to 6, and j ranges from 0 to 6), its horizontal coordinate is calculated as: the horizontal coordinate of the top-left corner vertex (200) plus j multiplied by the column spacing. The column spacing is calculated as the anchor frame width (64) divided by (the preset number of rows and columns). The sample number is 7 minus 1, which is 64 divided by 6, approximately equal to 10.67. Therefore, the x-coordinates of the sampling points in the j-th column are 200, 210.67, 221.33, 232, 242.67, 253.33, and 264. The y-coordinate is calculated as follows: the y-coordinate of the top left corner vertex is 150 plus i multiplied by the row spacing. The row spacing is calculated as the anchor frame height 32 divided by (the preset number of sampling rows is 7 minus 1), which is 32 divided by 6, approximately equal to 5.33. Therefore, the y-coordinates of the sampling points in the i-th row are 150, 155.33, 160.67, 166, 171.33, 176.67, and 182. This forms an initial sampling grid of 7 rows and 7 columns, with a total of 49 sampling points.

[0080] The sub-region of the fused feature map covered by the anchor frame is input into the offset prediction network. This sub-region refers to all pixels and their corresponding feature values ​​within the range of x-coordinate 200 to 264 and y-coordinate 150 to 182 on the fused feature map. The offset prediction network consists of a first convolutional layer, a first ReLU activation layer, a second convolutional layer, and a second ReLU activation layer connected sequentially. The first convolutional layer has a 3×3 kernel size, 256 input channels (the number of channels in the fused feature map), and 128 output channels. The first convolutional layer performs a convolution operation on the input sub-region of the fused feature map to extract local features. The first ReLU activation layer performs non-linear activation on the output of the first convolutional layer, setting all negative values ​​to 0 and leaving positive values ​​unchanged. The second convolutional layer... The kernel size of the first convolutional layer is 3×3, with 128 input channels and 98 output channels. The second convolutional layer performs a convolution operation on the output of the first ReLU activation layer, outputting a feature map with 98 channels. The second ReLU activation layer performs non-linear activation on the output of the second convolutional layer. The 98 channels output by the second convolutional layer are arranged in 49 groups according to channel order. Each group contains 2 channels, corresponding to the horizontal and vertical coordinate offsets of the 49 sampling points, respectively. For the k-th sampling point (k from 0 to 48), the value of the (2k)-th channel is used as the horizontal coordinate offset of the sampling point, and the value of the (2k+1)-th channel is used as the vertical coordinate offset of the sampling point. These offsets can be positive or negative. Positive values ​​indicate an offset in the positive direction of the coordinate axis, and negative values ​​indicate an offset in the negative direction of the coordinate axis.

[0081] Add the x-coordinate of each sampling point in the initial sampling grid to its corresponding x-coordinate offset to obtain the adjusted x-coordinate. For example, if the x-coordinate of the sampling point in row 0, column 0 of the initial sampling grid is 200, and the corresponding x-coordinate offset is +3.2, then the adjusted x-coordinate is 203.2. Add the y-coordinate of each sampling point to its corresponding y-coordinate offset to obtain the adjusted y-coordinate. For example, if the y-coordinate of the sampling point in row 0, column 0 of the initial sampling grid is 150, and the corresponding y-coordinate offset is -1.5, then the adjusted y-coordinate is 148.5. Perform the same operation on all 49 sampling points to obtain 49 adjusted sampling point coordinates. Each adjusted sampling point coordinate may be a non-integer.

[0082] For each adjusted sample point coordinate, determine the four adjacent integer coordinate positions on the fused feature map. For example, for an adjusted sample point coordinate of (203.2, 148.5), its four adjacent integer coordinate positions are (203, 148), (204, 148), (203, 149), and (204, 149). Read the corresponding feature values ​​on the fused feature map for each of these four integer coordinate positions. Each feature value is a 256-dimensional vector. Then, based on the relationship between the adjusted sample point's x-coordinate 203.2 and the x-coordinates of each integer coordinate... The horizontal distance determines the horizontal weight: For the integer x-coordinate 203, the horizontal distance is |203.2 - 203| = 0.2, and the horizontal weight is 1 - 0.2 = 0.8; for the integer x-coordinate 204, the horizontal distance is |203.2 - 204| = 0.8, and the horizontal weight is 1 - 0.8 = 0.2; The vertical weight is determined based on the vertical distance between the adjusted sampling point y-coordinate 148.5 and the y-coordinates of each integer coordinate: For the integer y-coordinate 148, the vertical distance is |148.5 - 148| = 0.5, and the vertical weight is 1 - 0.5 = 0.5; for the integer y-coordinate... The vertical distance is |148.5-149|=0.5, and the vertical weight is 1-0.5=0.5. The feature values ​​at the four integer coordinate positions are multiplied by their corresponding horizontal and vertical weights and then summed. Specifically: the feature value at position (203,148) is multiplied by 0.8 and then by 0.5; the feature value at position (204,148) is multiplied by 0.2 and then by 0.5; the feature value at position (203,149) is multiplied by 0.8 and then by 0.5; the feature value at position (204,149) is multiplied by 0.2 and then by 0.5. These four products are then added together to obtain the result. The interpolated feature value of the adjusted sampling point is a 256-dimensional vector. The same bilinear interpolation operation is performed on all 49 adjusted sampling points to obtain 49 256-dimensional interpolated feature vectors. These 49 256-dimensional interpolated feature vectors are then subjected to global average pooling in the spatial dimensions. That is, for each of the 256 dimensions, the 49 values ​​of the 49 interpolated feature vectors in that dimension are added together and then divided by 49 to obtain 256 average values. These 256 average values ​​are arranged in order to form the fixed-length feature vector corresponding to the anchor frame, which has a dimension of 256.

[0083] Furthermore, the defect category classification branch includes a first fully connected layer and a classification output layer. The first fully connected layer maps the fixed-length feature vector to intermediate classification features. The classification output layer uses a preset activation function to convert the intermediate classification features into confidence probabilities of each defect category, which are used as the predicted values ​​of the defect categories.

[0084] The location regression branch includes a second fully connected layer and a regression output layer. The second fully connected layer maps the fixed-length feature vector to intermediate regression features, and the regression output layer outputs a coordinate offset of a preset dimension as the predicted location offset value.

[0085] In this embodiment, a fixed-length feature vector is input into the defect category classification branch and the location regression branch, respectively. The fixed-length feature vector is a 256-dimensional vector, denoted as V. The defect category classification branch includes a first fully connected layer and a classification output layer. The first fully connected layer maps the 256-dimensional fixed-length feature vector into 3-dimensional intermediate classification features. The weight matrix of the first fully connected layer is 3 rows and 256 columns, and the bias vector is 3-dimensional. The mapping method is as follows: the 256-dimensional input vector is multiplied element-wise by the 256 column weights, summed, and then the corresponding bias value is added to obtain three intermediate classification feature values, which correspond to the original scores of the three types of defects, respectively. The classification output... The layer uses the Softmax activation function to convert the three intermediate classification feature values ​​into confidence probabilities for three types of defects. The specific conversion method of the Softmax function is as follows: for the score s_c of the c-th defect, its confidence probability is e raised to the power of s_c divided by the sum of (e raised to the power of s_1, e raised to the power of s_2, and e raised to the power of s_3), where e is the base of the natural logarithm. After Softmax processing, the sum of the confidence probabilities of the three categories is 1, and each probability value is between 0 and 1. These three probability values ​​are used as the defect category prediction values ​​corresponding to the anchor frame, representing the probability that the anchor frame area belongs to indentation damage, indentation fracture, and insufficient indentation depth, respectively.

[0086] The location regression branch consists of a second fully connected layer and a regression output layer. The second fully connected layer maps a 256-dimensional fixed-length feature vector to 4-dimensional intermediate regression features. The weight matrix of the second fully connected layer is 4 rows and 256 columns, and the bias vector is 4-dimensional. The mapping is performed by multiplying the 256-dimensional input vector element-wise with the 256 column weights, summing the results, and adding the corresponding bias values ​​to obtain four intermediate regression feature values. The regression output layer directly outputs these four intermediate regression feature values ​​as coordinate offsets, corresponding to the horizontal offset of the top left corner, the vertical offset of the top left corner, the width offset, and the height offset of the anchor frame, respectively. These four offsets serve as the predicted position offset values ​​for the anchor frame.

[0087] Furthermore, after outputting the defect category prediction value and location offset prediction value corresponding to each anchor box, the following operations are also performed:

[0088] Based on the initial top-left vertex coordinates, anchor width, anchor height, and corresponding position offset prediction values ​​of each anchor frame, calculate the top-left and bottom-right vertex coordinates of the predicted bounding box corresponding to each anchor frame.

[0089] Sort all the predicted bounding boxes corresponding to the anchor boxes from highest to lowest confidence level in their defect category prediction values ​​to obtain the sorted sequence of predicted bounding boxes.

[0090] Select the highest-confidence predicted bounding box from the sorted predicted bounding box sequence as the retained bounding box, add the retained bounding box to the final result set, and calculate the intersection-union ratio (IUR) between the retained bounding box and the remaining predicted bounding boxes in the sorted predicted bounding box sequence. Remove the remaining predicted bounding boxes with IUR values ​​greater than a preset IUR threshold from the sequence.

[0091] Repeat the process of selecting the predicted bounding box with the highest confidence as the retained bounding box and removing overlapping predicted bounding boxes until the sorted sequence of predicted bounding boxes is empty; output the defect category and location corresponding to each retained bounding box in the final result set as the final detection result.

[0092] In this embodiment, after outputting the defect category prediction value and position offset prediction value corresponding to each anchor frame, the following post-processing operations are performed: First, for each anchor frame, based on the initial top-left corner vertex coordinates (x0, y0), anchor frame width w0, anchor frame height h0, and the corresponding position offset prediction value (Δx, Δy, Δw, Δh), the top-left corner vertex coordinates and bottom-right corner vertex coordinates of the predicted bounding box corresponding to the anchor frame are calculated; specifically, the x-coordinate of the top-left corner vertex of the predicted bounding box is x0 plus Δx, the y-coordinate of the top-left corner vertex is y0 plus Δy, and the width of the predicted bounding box is w0 multiplied by e raised to the power of Δw. The height of the bounding box is h0 multiplied by e raised to the power of Δh; then the coordinates of the lower right corner are calculated based on the coordinates of the upper left corner and the width and height; for example, if the initial anchor box has the coordinates of the upper left corner (200, 150), a width of 64, and a height of 32, and the predicted position offset is (+5, -3, +0.1, -0.05), then the coordinates of the upper left corner of the predicted bounding box are (205, 147), the width is 64 multiplied by e raised to the power of 0.1, which is approximately 70.7, the height is 32 multiplied by e raised to the power of -0.05, which is approximately 30.4, and the coordinates of the lower right corner are (205+70.7, 147+30.4), which is (275.7, 177.4).

[0093] After calculating the predicted bounding boxes for all anchor boxes, a large number of predicted bounding boxes are obtained. Each predicted bounding box corresponds to a defect category prediction value (the confidence probabilities of the three categories) and a location. All predicted bounding boxes are sorted from highest to lowest according to the highest confidence among their defect category prediction values. For example, if the confidence of a certain predicted bounding box is (0.85, 0.10, 0.05) for the three categories, its highest confidence is 0.85. All predicted bounding boxes are sorted in descending order of the highest confidence to obtain the sorted sequence of predicted bounding boxes.

[0094] Select the highest-confidence predicted bounding box from the sorted sequence of predicted bounding boxes as the retained bounding box; add this retained bounding box to the final result set; then calculate the intersection-union ratio (IUR) between this retained bounding box and the remaining predicted bounding boxes in the sorted sequence; the IUR is calculated as the intersection area of ​​two predicted bounding boxes divided by the union area; the intersection area is the area of ​​the overlapping portion of the two rectangles, and the union area is the sum of the areas of the two rectangles minus the intersection area; remove the remaining predicted bounding boxes from the sequence whose IUR is greater than a preset IU threshold (e.g., set to 0.5). Remove; repeat the above operation: select the predicted bounding box with the highest confidence from the remaining sequence as the new retained bounding box, add it to the final result set, calculate its intersection-union ratio (IU) with other predicted bounding boxes in the sequence, and remove predicted bounding boxes with IU greater than the threshold; repeat this process until the sorted sequence of predicted bounding boxes is empty; each retained bounding box in the final result set is the finally detected defect location, and its corresponding defect category is the category corresponding to the highest confidence; output the defect category and location corresponding to these retained bounding boxes as the final detection result for subsequent quality assessment.

[0095] Furthermore, referring to Figure 3 The step of performing resolution upsampling on the feature maps of each enhancement level to ensure that all feature maps of enhancement levels have the same spatial size specifically includes:

[0096] Traverse all enhancement level feature maps, read the current height and current width values ​​of each enhancement level feature map, select the maximum height value from all current height values ​​as the target height value, select the maximum width value from all current width values ​​as the target width value, and determine the target height and target width values ​​as the target spatial dimensions;

[0097] For each enhancement level feature map, calculate the ratio of the target height value to the current height value of the enhancement level feature map, as the height upsampling factor; calculate the ratio of the target width value to the current width value of the enhancement level feature map, as the width upsampling factor.

[0098] Perform a transpose convolution operation in the height direction on the enhanced layer feature map. The number of input channels of the transpose convolution operation in the height direction is equal to the current number of channels of the enhanced layer feature map, and the number of output channels is the preset number of intermediate channels.

[0099] Perform a transpose convolution operation in the width direction on the feature map after transpose convolution in the height direction. The number of input channels for the transpose convolution operation in the width direction is equal to the preset number of intermediate channels, and the number of output channels is the preset number of uniform channels.

[0100] After performing transpose convolution operations in the height and width directions, an edge cropping operation is performed on the feature map. The edge cropping operation removes the additional pixels on the edges caused by the transpose convolution filling, so that the spatial size of the feature map is exactly equal to the target height and target width values.

[0101] For each enhanced layer feature map, the above height-direction transposed convolution, width-direction transposed convolution, and edge clipping operations are performed to obtain each upsampled feature map. The spatial dimensions of each upsampled feature map are the target height value and the target width value, and the number of channels is a preset uniform number of channels.

[0102] Each upsampled feature map is added element-wise along the spatial dimension to obtain a summed feature map. A preset nonlinear activation function is then applied to the summed feature map to obtain a fused feature map.

[0103] In this embodiment, resolution upsampling is performed on each enhancement level feature map to ensure that all enhancement level feature maps have the same spatial size. The specific process is as follows: First, the three enhancement level feature maps are traversed, and the current height and width values ​​of each enhancement level feature map are read. The current height and width of the first enhancement level feature map (derived from the lowest resolution level feature map) are 32 pixels. The current height and width of the second enhancement level feature map (derived from the intermediate resolution level feature map) are 64 pixels. The current height and width of the third enhancement level feature map (derived from the highest resolution level feature map) are 128 pixels. The maximum height value is selected from all current height values ​​as the target height value, i.e., 128 pixels. The maximum width value is selected from all current width values ​​as the target width value, i.e., 128 pixels. The target height value of 128 and the target width value of 128 are determined as the target spatial size.

[0104] Then, for each enhancement layer feature map, its respective upsampling factor is calculated. For the first enhancement layer feature map, the target height value of 128 divided by the current height value of 32 yields a height upsampling factor of 4; the target width value of 128 divided by the current width value of 32 yields a width upsampling factor of 4. For the second enhancement layer feature map, the target height value of 128 divided by the current height value of 64 yields a height upsampling factor of 2; the target width value of 128 divided by the current width value of 64 yields a width upsampling factor of 2. For the third enhancement layer feature map, the target height value of 128 divided by the current height value of 128 yields a height upsampling factor of 1; the target width value of 128 divided by the current width value of 128 yields a width upsampling factor of 1.

[0105] Next, a transposed convolution operation in the height direction is performed on the first enhanced layer feature map. The current number of channels in this enhanced layer feature map is 2048, and the preset number of intermediate channels is set to 512. The input number of the transposed convolution operation in the height direction is equal to 2048, and the output number of channels is 512. The kernel size is the preset height kernel size, set to 4×1 (i.e., 4 in the height direction and 1 in the width direction). The stride is equal to the height upsampling factor of 4. The padding method is the preset padding method, set to pad in the height direction so that the height of the output feature map is equal to the input height multiplied by the stride. After the height-direction transposed convolution... Afterwards, the height of the feature map changes from 32 to 128, while the width remains unchanged at 32, and the number of channels becomes 512. Then, a transposed convolution operation in the width direction is performed on the feature map after the height-direction transposed convolution. The number of input channels for the width-direction transposed convolution operation is equal to 512, and the number of output channels is a preset uniform number of 256. The convolution kernel size is a preset width convolution kernel size, set to 1×4 (i.e., 1 in the height direction and 4 in the width direction). The stride is equal to the width upsampling factor of 4. After the width-direction transposed convolution, the width of the feature map changes from 32 to 128, while the height remains unchanged at 128, and the number of channels becomes 256.

[0106] After performing transposed convolution operations in the height and width directions, edge cropping is performed on the feature map. Since the padding operation of transposed convolution will generate additional pixels at the edges of the feature map, these additional pixels need to be removed so that the spatial size of the feature map is exactly equal to the target height value of 128 and the target width value of 128. The specific cropping method is as follows: in the height direction, remove several rows of pixels generated by padding at the top and bottom of the feature map; in the width direction, remove several columns of pixels generated by padding on the left and right sides of the feature map. After cropping, the spatial size of the feature map is exactly 128×128, and the number of channels is 256.

[0107] A similar operation is performed on the second enhanced layer feature map; its current height and width values ​​are both 64, and the height upsampling factor and width upsampling factor are both 2; the kernel size of the height-direction transposed convolution is 2×1, the stride is 2, and the number of output channels is 512; the kernel size of the width-direction transposed convolution is 1×2, the stride is 2, and the number of output channels is 256; after transposed convolution and edge clipping, the spatial size of the feature map becomes 128×128, and the number of channels is 256.

[0108] For the third enhanced layer feature map, its current height and width values ​​are already equal to the target height and width values, and the height upsampling factor and width upsampling factor are both 1. This feature map directly adjusts the number of channels from 512 to 256 through a 1×1 convolutional layer, without the need for transposed convolutional upsampling and edge clipping. After the above processing, the spatial size of the three feature maps is 128×128, and the number of channels is 256. These three 128×128×256 feature maps are added element-wise in the spatial dimension, that is, for each spatial position (i,j) and each channel c, the three values ​​located at (i,j,c) in the three feature maps are added to obtain a sum. After all positions and all channels are added, a summed feature map is obtained, whose size is still 128×128×256. The ReLU nonlinear activation function is applied to the summed feature map, that is, all negative values ​​are set to 0, and positive values ​​remain unchanged, to obtain the final fused feature map.

[0109] Furthermore, the preset size and preset aspect ratio of the multiple anchor frames are set according to the geometry of the die-cutting and creasing. The preset size includes multiple different size levels, each size level corresponding to a different anchor frame area range. The preset aspect ratio includes the elongated ratio corresponding to the straight line segment of the die-cutting and creasing and the square ratio corresponding to the corner of the die-cutting and creasing. The setting position of each anchor frame on the fused feature map is centered on each pixel of the fused feature map, and multiple anchor frames of different shapes are generated according to the preset size and preset aspect ratio.

[0110] In this embodiment, the preset size and aspect ratio of multiple anchor frames are set according to the geometry of the die-cutting crease. The die-cutting crease line appears as a thin, elongated line in the image, with a typical width of about 2 to 4 pixels and a typical length of tens to hundreds of pixels. The preset size includes three different size levels: the small size level corresponds to an anchor frame area of ​​16 square pixels (i.e., the area corresponding to a square with a side length of 4 pixels), the medium size level corresponds to an anchor frame area of ​​64 square pixels (i.e., the area corresponding to a square with a side length of 8 pixels), and the large size level corresponds to an anchor frame area of ​​256 square pixels (i.e., the area corresponding to a square with a side length of 16 pixels). Each size level corresponds to a different range of anchor frame areas: the small size level covers a side length range of 4 to 6 pixels, the medium size level covers a side length range of 7 to 10 pixels, and the large size level covers a side length range of 11 to 20 pixels.

[0111] The preset aspect ratios include the elongated ratio corresponding to the straight line segment of the die-cutting crease and the square ratio corresponding to the corner of the die-cutting crease. The elongated ratio corresponding to the straight line segment of the die-cutting crease includes 1:2 (width to height) and 2:1 (height to width). These two ratios can better match the slender shape of the crease line extending horizontally or vertically in the image. The square ratio corresponding to the corner of the die-cutting crease is 1:1. This ratio can better match the local area of ​​the approximately square formed by the crease line at the corner.

[0112] Each anchor box is positioned on the fused feature map by generating multiple anchor boxes of different shapes centered on each pixel of the fused feature map, according to the aforementioned preset size and aspect ratio. Specifically, for each pixel position (x, y) on the fused feature map, nine anchor boxes are generated centered on that pixel. The first anchor box is a small-sized square with an aspect ratio of 1:1, i.e., a square with a width of 4 pixels and a height of 4 pixels. The second anchor box is a small-sized rectangle with an aspect ratio of 1:2, i.e., a rectangle with a width of 4 pixels and a height of 8 pixels. The third anchor box is a small-sized rectangle with an aspect ratio of 2:1, i.e., a rectangle with a width of 8 pixels and a height of 4 pixels. The fourth anchor box is a medium-sized rectangle with an aspect ratio of 1:1, i.e., a square with a width of 8 pixels and a height of 8 pixels. The fifth anchor box is a medium-sized rectangle with an aspect ratio of 1:2 (8 pixels wide and 16 pixels high); the sixth anchor box is a medium-sized rectangle with an aspect ratio of 2:1 (16 pixels wide and 8 pixels high); the seventh anchor box is a large rectangle with an aspect ratio of 1:1 (16 pixels wide and 16 pixels high); the eighth anchor box is a large rectangle with an aspect ratio of 1:2 (16 pixels wide and 32 pixels high); and the ninth anchor box is a large rectangle with an aspect ratio of 2:1 (32 pixels wide and 16 pixels high). For the 16,384 pixels on the 128×128 fused feature map, 9 anchor boxes are generated for each pixel, for a total of 147,456 anchor boxes.

[0113] Furthermore, the preset lighting conditions include using a coaxial light source to provide vertical illumination and using a ring light source to provide oblique illumination. The coaxial light source and the ring light source are turned on simultaneously, and the ratio between the light intensity of the coaxial light source and the light intensity of the ring light source is set to a preset ratio. When acquiring the grayscale image, the camera optical axis is perpendicular to the plane where the die-cutting and creasing area of ​​the cardboard box to be inspected is located, the camera exposure time is set to a preset exposure time, and the camera gain is set to a preset gain value.

[0114] In this embodiment, the preset lighting conditions include using a coaxial light source to provide vertical illumination and using a ring light source to provide oblique illumination. The coaxial light source is positioned between the camera lens and the cardboard box, and a 45-degree tilted beam splitter directs light vertically downwards onto the cardboard box surface. The light reflected from the cardboard box surface passes vertically upwards through the beam splitter and enters the camera lens. The ring light source is positioned around the camera lens, consisting of multiple LED beads arranged in a ring, illuminating the cardboard box surface at an angle of 45 to 60 degrees relative to the normal direction of the cardboard box surface. The coaxial light source and the ring light source are turned on simultaneously. The ratio between the light intensity of the coaxial light source and the light intensity of the ring light source is set to a preset ratio of 1:2, meaning the light intensity of the ring light source is twice that of the coaxial light source. The vertical illumination provided by the coaxial light source highlights the depth information of the indented areas on the cardboard box surface, making the indented areas appear as uniform dark lines in the image. The oblique illumination provided by the ring light source enhances the contrast of the cardboard box surface texture and edges, making the damage and breakage of the indented edges more clearly visible.

[0115] When acquiring grayscale images, the camera's optical axis is perpendicular to the plane containing the die-cutting and crease area of ​​the cardboard box to be inspected. The camera is a global shutter type industrial area array camera, whose optical axis coincides with the normal direction of the cardboard box surface, ensuring that the captured image does not suffer from perspective distortion. The camera's exposure time is set to a preset exposure time of 500 microseconds. The camera's gain is set to a preset gain value of 1.0. During the exposure time, the light reflected from the cardboard box surface accumulates charge on the camera's image sensor, which is then converted from analog to digital to generate a digital grayscale image. The combined illumination of the coaxial light source and the ring light source gives the die-cutting and crease area a stable grayscale characteristic in the image. The grayscale value is lower at the crease indentation (approximately 30 to 60), while the grayscale value is higher in the background cardboard area (approximately 180 to 220). The grayscale value at the edge of the crease is between the two or exhibits irregular grayscale abrupt changes.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0118] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A deep learning-based visual detection method for die-cutting creases and damage defects in paper boxes, characterized in that, Includes the following steps: S1, Obtain a grayscale image of the die-cutting and crease area of ​​the cardboard box to be inspected under preset lighting conditions; S2, the grayscale image is input into a deep convolutional network, which includes multiple convolutional modules connected in sequence. Each convolutional module outputs a hierarchical feature map of a corresponding resolution, thereby obtaining at least three hierarchical feature maps of different resolutions. S3, Select the feature map of the lowest resolution level, perform spatial attention extraction operation on the feature map of the lowest resolution level to obtain a spatial weight map, the spatial size of the spatial weight map is the same as the spatial size of the feature map of the lowest resolution level; S4, the spatial weight map is adjusted to the spatial size of the feature maps of the other levels through an upsampling operation, and the adjusted spatial weight map is multiplied element by element with the corresponding level feature map to obtain the feature maps of each enhancement level. S5. Perform resolution upsampling operation on the feature maps of each enhancement level to make all feature maps of enhancement levels have the same spatial size. Then, add the feature maps of each enhancement level with the same spatial size element by element to obtain the fused feature map. S6, set multiple anchor boxes on the fused feature map, perform deformable pooling operation on the fused feature map region covered by each anchor box to obtain a fixed-length feature vector corresponding to each anchor box, input the fixed-length feature vector into the defect category classification branch and the location regression branch respectively, and output the defect category prediction value and location offset prediction value corresponding to each anchor box.

2. The method according to claim 1, characterized in that, The deep convolutional network is a residual network, and the at least three hierarchical feature maps of different resolutions are the feature maps output by the third-to-last residual stage, the second-to-last residual stage, and the last residual stage, respectively.

3. The method according to claim 1, characterized in that, The spatial attention extraction operation includes: Global average pooling is performed on the feature map of the lowest resolution layer to obtain a first pooling descriptor, and global max pooling is performed on the feature map of the lowest resolution layer to obtain a second pooling descriptor. The first pooling descriptor and the second pooling descriptor are concatenated along the channel dimension to obtain the concatenated descriptor; The concatenated descriptor is input into a convolutional layer. The kernel size of the convolutional layer is a preset size, and the number of output channels is a single channel. The single-channel feature map output by the convolutional layer is processed by a non-linear activation function and used as the spatial weight map.

4. The method according to claim 1, characterized in that, The process of performing deformable pooling on the fused feature map region covered by each anchor frame specifically includes: For each anchor frame, based on the coordinates of the top-left vertex, the width, and the height of the anchor frame on the fused feature map, an initial sampling grid is constructed according to a preset number of sampling rows and columns. The x-coordinate of each sampling point in the initial sampling grid is the x-coordinate of the top-left vertex plus the product of the column where the sampling point is located and the column spacing. The fused feature map sub-region covered by the anchor frame is input into the offset prediction network, which includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence. Add the x-coordinate of each sampling point in the initial sampling grid to the corresponding x-coordinate offset to obtain the adjusted x-coordinate; add the y-coordinate of each sampling point to the corresponding y-coordinate offset to obtain the adjusted y-coordinate. For each adjusted sample point coordinate, determine the positions of the four adjacent integer coordinates on the fused feature map; Read the four feature values ​​corresponding to the four integer coordinate positions respectively. Determine the horizontal weight based on the horizontal distance between the adjusted sampling point's horizontal coordinate and the horizontal coordinates of each integer coordinate. Determine the vertical weight based on the vertical distance between the adjusted sampling point's vertical coordinate and the vertical coordinates of each integer coordinate. Multiply the four feature values ​​by their corresponding horizontal and vertical weights respectively and sum them to obtain the interpolation feature value of the adjusted sampling point. Perform global average pooling on the interpolated feature values ​​of all adjusted sampling points to obtain the fixed-length feature vector.

5. The method according to claim 1, characterized in that, The defect category classification branch includes a first fully connected layer and a classification output layer. The first fully connected layer maps the fixed-length feature vector to intermediate classification features. The classification output layer uses a preset activation function to convert the intermediate classification features into confidence probabilities of each defect category, which are used as the predicted values ​​of the defect categories. The location regression branch includes a second fully connected layer and a regression output layer. The second fully connected layer maps the fixed-length feature vector to intermediate regression features, and the regression output layer outputs a coordinate offset of a preset dimension as the predicted location offset value.

6. The method according to claim 1, characterized in that, After outputting the defect category prediction value and location offset prediction value for each anchor box, the following operations are also performed: Based on the initial top-left vertex coordinates, anchor width, anchor height, and corresponding position offset prediction values ​​of each anchor frame, calculate the top-left and bottom-right vertex coordinates of the predicted bounding box corresponding to each anchor frame. Sort all the predicted bounding boxes corresponding to the anchor boxes from highest to lowest confidence level in their defect category prediction values ​​to obtain the sorted sequence of predicted bounding boxes. Select the highest-confidence predicted bounding box from the sorted predicted bounding box sequence as the retained bounding box, add the retained bounding box to the final result set, and calculate the intersection-union ratio (IUR) between the retained bounding box and the remaining predicted bounding boxes in the sorted predicted bounding box sequence. Remove the remaining predicted bounding boxes with IUR values ​​greater than a preset IUR threshold from the sequence. Repeat the process of selecting the predicted bounding box with the highest confidence as the retained bounding box and removing overlapping predicted bounding boxes until the sorted sequence of predicted bounding boxes is empty; output the defect category and location corresponding to each retained bounding box in the final result set as the final detection result.

7. The method according to claim 1, characterized in that, The step of performing resolution upsampling on the feature maps of each enhancement level to ensure that all feature maps of enhancement levels have the same spatial size specifically includes: Traverse all enhancement level feature maps, read the current height value and current width value of each enhancement level feature map, select the maximum height value from all current height values ​​as the target height value, select the maximum width value from all current width values ​​as the target width value, and determine the target height value and target width value as the target spatial size; For each enhancement level feature map, the ratio of the target height value to the current height value of the enhancement level feature map is calculated as the height upsampling factor, and the ratio of the target width value to the current width value of the enhancement level feature map is calculated as the width upsampling factor. Perform a transposed convolution operation in the height direction on the enhanced layer feature map. The number of input channels of the transposed convolution operation in the height direction is equal to the current number of channels of the enhanced layer feature map, and the number of output channels is a preset number of intermediate channels. A transpose convolution operation in the width direction is performed on the feature map after transpose convolution in the height direction. The number of input channels of the transpose convolution operation in the width direction is equal to the preset number of intermediate channels, and the number of output channels is the preset number of uniform channels. After performing transpose convolution operations in the height and width directions, an edge cropping operation is performed on the feature map. The edge cropping operation removes the additional pixels on the edges caused by the transpose convolution filling, so that the spatial size of the feature map is exactly equal to the target height and target width values. For each enhanced layer feature map, the above height-direction transposed convolution, width-direction transposed convolution, and edge clipping operations are performed to obtain each upsampled feature map. The spatial dimensions of each upsampled feature map are the target height value and the target width value, and the number of channels is the preset unified number of channels. The sampled feature maps are added element by element in the spatial dimension to obtain the added feature map. The added feature map is then processed by a preset nonlinear activation function to obtain the fused feature map.

8. The method according to claim 1, characterized in that, The preset size and preset aspect ratio of the multiple anchor frames are set according to the geometry of the die-cutting and creasing. The preset size includes multiple different size levels, each size level corresponding to a different anchor frame area range. The preset aspect ratio includes the ratio of the long strip corresponding to the straight line segment of the die-cutting and creasing and the ratio of the square corresponding to the corner of the die-cutting and creasing. The setting position of each anchor frame on the fused feature map is centered on each pixel of the fused feature map, and multiple anchor frames of different shapes are generated according to the preset size and preset aspect ratio.

9. The method according to claim 1, characterized in that, The preset lighting conditions include using a coaxial light source to provide vertical illumination and using a ring light source to provide oblique illumination. The coaxial light source and the ring light source are turned on simultaneously, and the ratio between the light intensity of the coaxial light source and the light intensity of the ring light source is set to a preset ratio. When acquiring the grayscale image, the camera's optical axis is perpendicular to the plane containing the die-cutting and creasing area of ​​the cardboard box to be inspected, the camera's exposure time is set to a preset exposure time, and the camera's gain is set to a preset gain value.

Images