A wire port size extraction method based on deep learning

Abstract

The application relates to the technical field of machine vision and image processing, and particularly discloses a wire port size extraction method based on deep learning, which receives an original image, constructs an image pyramid, and cuts out a port detection area; extracts the direction gradient histogram features, the gray level co-occurrence matrix texture features and the Hu invariant rectangular shape features of the area, fuses the features after dimension reduction through principal component analysis, inputs a layered support vector machine to distinguish mixed materials and identify the port posture, inputs the classified image into a transform network, generates an edge enhancement feature map through Laplace convolution, erases a high-confidence area by using residual learning to focus on a weak edge, and outputs a binary segmentation mask; performs bilateral Gaussian filtering to preserve edges and remove noise, extracts a Canny edge, removes redundant points through curve approximation, obtains a sub-pixel level pixel height through convex hull analysis and minimum circumscribed rectangle fitting, calibrates the actual physical size, compares the actual physical size with a preset process threshold, and outputs a good product or a defective product determination result.

Description

Technical Field

[0001] This invention relates to the field of machine vision and image processing technology, specifically to a method for extracting wire port dimensions based on deep learning. Background Technology

[0002] In the modern electronics and information industry, the manufacturing processes for mainstream cable ports such as Type-C are numerous, and the requirements for size and shape precision are extremely high. During the actual acquisition process of cable port images, due to the high reflectivity of the metal pin surface, overexposure or halos are easily generated, resulting in weak and blurry edges in the image.

[0003] Traditional image processing and measurement methods often fail to accurately extract effective detection areas when faced with such reflective interference, resulting in slow positioning and large measurement errors. To address the weakness and difficulty in continuous edge extraction caused by the high reflectivity of metal pin surfaces, this paper employs an edge extraction module within a transform network to generate an edge enhancement feature map using Laplacian convolution on the input features. Then, residual learning is used to erase high-confidence regions, forcing the network to abandon accurately segmented and easily identifiable areas and focus on the weak boundaries affected by reflective interference, thus outputting a seamless binary segmentation mask. After bilateral filtering for edge preservation and noise reduction, and curve approximation to remove redundant points, this mask can be fitted with a minimum bounding rectangle to obtain stable sub-pixel-level pixel heights, which are then converted into physical dimensions. This solves the problem of edge breakage and inability to continuously measure under reflective interference in traditional algorithms. Summary of the Invention

[0004] The purpose of this invention is to provide a method for extracting wire port dimensions based on deep learning, so as to solve the problems mentioned above.

[0005] The objective of this invention can be achieved through the following technical solutions: A method for extracting wire port dimensions based on deep learning includes the following steps: S1 receives the original digital image containing the cable port, constructs an image pyramid of the source image and the template image, matches the extreme coordinates layer by layer from the top of the pyramid downwards, and quickly locates and crops out the port detection area. S2, extract the directional gradient histogram features, gray-level co-occurrence matrix texture features and Hu's invariant rectangular features from the port detection area respectively. After performing principal component analysis to reduce the dimensionality of the directional gradient histogram features, it is concatenated and fused with the gray-level co-occurrence matrix texture features and Hu's invariant rectangular features to obtain a comprehensive feature vector. The comprehensive feature vector is input into a hierarchical support vector machine. The first layer determines whether there is material mixing, the second layer identifies the port posture, and outputs the classification label. S3: The image carrying the classification label is input into the pre-trained transformation network. The transformation network generates an edge enhancement feature map by convolving the input features. Residual learning guides the transformation network to focus on weak edge regions and outputs a high-resolution binarized segmentation mask. S4. Apply bilateral Gaussian filtering to the binarized segmentation mask to preserve edges and denoise, extract continuous edges, apply curve approximation algorithm to the extracted edge contours to remove redundant points, combine convex hull analysis to obtain the overall contour, and calculate the sub-pixel level pixel height by fitting the minimum bounding rectangle. S5 multiplies the pixel height by the system-calibrated pixel equivalent coefficient, maps it to the actual physical size, compares it with the preset process threshold, and outputs the judgment result of good or bad product.

[0006] As a further aspect of the present invention: the rapid positioning and trimming of the port detection area specifically includes: Construct an image pyramid between the source image and the template image, calculate the zero-mean normalized cross-correlation value starting from the top of the pyramid, and determine the center coordinates with the maximum value as candidate regions; Map the coordinates of the candidate region to the next layer, traverse the extreme points only within the mapped region, and iterate layer by layer to the bottom of the pyramid. The port detection area is clipped out using the coordinates output from the underlying layer.

[0007] As a further aspect of the present invention: the calculation of the zero-mean normalized cross-correlation value starting from the top of the pyramid specifically includes: Select the source image region covered by the current sliding window, and calculate the grayscale average and standard deviation of all pixels within the source image region; Calculate the mean and standard deviation of gray levels for all pixels within the template image; The cross-correlation metric of the window position is obtained by subtracting the average gray value of the source image region from each pixel within the window region and dividing it by its standard deviation, and by subtracting the average gray value of the template from each pixel within the template and dividing it by the template standard deviation. The products of the corresponding positions are summed and divided by the total number of pixels within the window. Iterate through all sliding window positions at the top of the pyramid, select the center coordinates of the window with the largest cross-correlation value as the candidate extreme point, and pass the coordinates of the candidate extreme point down to the next layer.

[0008] As a further aspect of the present invention: the output classification label specifically includes: Principal component analysis was performed on the directional gradient histogram features to reduce the dimensionality to 64 dimensions, and then concatenated with the 6-dimensional gray-level co-occurrence matrix texture features and the 7-dimensional Hu's invariant rectangular features to form a one-dimensional comprehensive feature vector. The one-dimensional comprehensive feature vector is input into the first layer of the hierarchical support vector machine, and a linear kernel function is used to determine whether it is the target product category. The feature vectors identified as the target product in the first layer are fed into the front and back orientation ports of the second layer to output classification labels.

[0009] As a further aspect of the present invention: the step of using a linear kernel function to determine whether it is a target product category specifically includes: Obtain the pre-stored set of support vectors in the first layer of the hierarchical support vector machine, the Lagrange multipliers corresponding to each support vector, and the decision bias term; Calculate the inner product of the current integrated feature vector and each support vector, multiply each inner product by the corresponding Lagrange multiplier, and then sum them up. The cumulative sum is added to the decision bias term. If the resulting value is greater than zero, it is determined to be the target product category and allowed to proceed to the second level. If it is less than or equal to zero, it is determined to be a mixing anomaly and the calculation is terminated.

[0010] As a further aspect of the present invention: the output high-resolution binarized segmentation mask specifically includes: The edge extraction module inside the transform network first uses the Laplacian operator to convolve the input features to obtain a multi-directional gradient map. Then, after normalization, the gradient map is fused with the input features pixel by pixel to generate an edge enhancement feature map. The edge enhancement feature map is fed into the edge enhancement reverse attention module, and high confidence regions are erased through residual learning, forcing the network to focus on weak edges; Output a binarized segmentation mask with continuous edges.

[0011] As a further aspect of the present invention: the step of erasing high-confidence regions through residual learning, forcing the network to focus on weak edges, specifically includes: Perform a convolution operation on the edge enhancement feature map to generate a single-channel prediction confidence map, and limit the value of each pixel in the prediction confidence map to the range of 0 to 1; Pixels with values ​​greater than 0.8 in the confidence map are marked as high-confidence regions. The reverse attention weights are calculated by subtracting the original value from the confidence value in the high-confidence region by 1, thus obtaining the reverse attention map. The inverse attention map is multiplied pixel by pixel by the edge enhancement feature map, and then added element by element to the edge enhancement feature map itself. The result is used as the enhancement feature map focusing on weak edges and fed into subsequent layers.

[0012] As a further aspect of the present invention: the calculation of sub-pixel level pixel height specifically includes: First, apply a bilateral Gaussian filter to the binarized segmentation mask, and simultaneously calculate spatial proximity weights and pixel value similarity weights to smooth noise and preserve edges; The Kenny algorithm is used to extract continuous edges from the filtered image, and then the Lamer-Douglas-Puk curve approximation algorithm is applied to the extracted edge contours to remove redundant pixels. The overall contour is obtained through convex hull analysis, and the sub-pixel height of each pin is obtained by fitting the minimum bounding rectangle.

[0013] As a further aspect of the present invention: the removal of redundant pixels specifically includes: The gradient magnitude and gradient direction of each pixel are calculated for the image after bilateral Gaussian filtering. Non-maximum suppression is performed along the gradient direction to set the gradient value of non-edge pixels to zero. Set a high threshold and a low threshold, mark pixels with gradient values ​​higher than the high threshold as strong edge pixels, mark pixels between the low threshold and the high threshold as weak edge pixels, retain only weak edge pixels connected to strong edge pixels, and output a continuous single-pixel wide edge map. For each edge contour extracted from the single-pixel wide edge map, the vertical distance from each intermediate point to the connecting line is calculated by connecting the first and last pixels in sequence. If the maximum vertical distance is less than the preset threshold, the intermediate pixel point corresponding to the maximum distance is removed. Otherwise, the segmentation is recursively performed, and the simplified edge contour after removing redundant pixels is output.

[0014] As a further aspect of the present invention: the comparison with the preset process threshold specifically includes: Read the pixel equivalent coefficients pre-stored in the calibration file, multiply the subpixel-level pixel height by the subpixel-level pixel equivalent coefficients, and convert it into the actual physical size in micrometers; Based on the category label, retrieve the upper and lower threshold values ​​of the standard size corresponding to the category label; The actual physical size is compared with the upper and lower thresholds respectively. If the actual physical size is both less than the upper threshold and greater than the lower threshold, the good product judgment result is output; otherwise, the defective product judgment result is output along with the out-of-tolerance value.

[0015] The beneficial effects of this invention are: (1) By constructing an image pyramid and combining it with a zero-mean normalized cross-correlation algorithm to pass the extreme coordinates layer by layer, the port detection area can be stably located under the interference of additive noise caused by the reflection of metal surface. At the same time, the matching calculation range is compressed from the whole image to the local extreme value neighborhood, reducing the number of pixels traversed for a single location and reducing the time consumption of cropping the region of interest in online detection.

[0016] (2) By transforming the edge extraction module inside the network to generate an edge enhancement feature map, and using residual learning to erase high confidence areas, the network is forced to refocus on the weak boundaries affected by reflection. This allows the output of a continuous and unbroken binary segmentation mask in low-contrast, blurred wire port images. After combining bilateral filtering to preserve edges and noise reduction and curve approximation to remove redundant points, the pixel height obtained by fitting the minimum bounding rectangle can stably reach the sub-pixel level, providing a complete contour basis for subsequent physical size conversion. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 These are comparison images showing the effects of fine segmentation of the wire end edge before and after in this invention; Figure 3 This is a diagram of the Transformer network architecture based on edge guidance and reverse attention mechanisms in this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 As shown, this invention is a method for extracting wire port dimensions based on deep learning, comprising the following steps: S1 receives the original digital image containing the cable port, constructs an image pyramid of the source image and the template image, matches the extreme value coordinates layer by layer from the top of the pyramid downwards, and quickly locates and crops out the port detection area. S2, extract the directional gradient histogram features, gray-level co-occurrence matrix texture features and Hu's invariant rectangular features from the port detection area respectively. After performing principal component analysis to reduce the dimensionality of the directional gradient histogram features, it is concatenated and fused with the gray-level co-occurrence matrix texture features and Hu's invariant rectangular features to obtain a comprehensive feature vector. The comprehensive feature vector is input into a hierarchical support vector machine. The first layer determines whether there is material mixing, the second layer identifies the port posture, and outputs the classification label. S3: The image carrying the classification label is input into the pre-trained transformation network. The transformation network generates an edge enhancement feature map by convolving the input features. Residual learning guides the transformation network to focus on weak edge regions and outputs a high-resolution binarized segmentation mask. S4. Apply bilateral Gaussian filtering to the binarized segmentation mask to preserve edges and denoise, extract continuous edges, apply curve approximation algorithm to the extracted edge contours to remove redundant points, combine convex hull analysis to obtain the overall contour, and calculate the sub-pixel level pixel height by fitting the minimum bounding rectangle. S5 multiplies the pixel height by the system-calibrated pixel equivalent coefficient, maps it to the actual physical size, compares it with the preset process threshold, and outputs the judgment result of good or bad product.

[0021] In S1, the raw digital image containing the wire port is received, an image pyramid of the source image and template image is constructed, and extreme value coordinates are matched layer by layer from the top of the pyramid downwards to quickly locate and crop out the port detection area. Specifically, this includes: When constructing the image pyramid of source and template images, a raw digital image containing the wire port is acquired from an industrial camera as the source image, and a template image of a standard wire port is pre-stored. The source and template images are continuously downsampled at a ratio of 50%, constructing a three-layer pyramid structure.

[0022] The zeroth layer is the original size image; the first layer is the image obtained by reducing the length and width of the zeroth layer image to half of their original size; the second layer is the image obtained by reducing the length and width of the first layer image to half of their original size. The top layer of the pyramid is the second layer image.

[0023] The process of calculating the zero-mean normalized cross-correlation value at the top of the pyramid is as follows: A sliding window is set, with the same size as the template image at the top layer. Starting from the top left corner of the top layer of the source image, the sliding window is moved pixel by pixel from left to right and top to bottom, each time moving one pixel increment, covering a local area of ​​the top layer of the source image. For each sliding window position, the following operations are performed: Calculate the arithmetic mean of the gray values ​​of all pixels within the source image region covered by the current sliding window, and the standard deviation of the gray values ​​of all pixels within that region.

[0024] The standard deviation is calculated as follows: first, calculate the square of the difference between the gray value of each pixel and the average value; then, sum the squares of all the differences and divide by the total number of pixels in the window; finally, take the square root of the quotient.

[0025] Calculate the arithmetic mean and standard deviation of the gray values ​​of all pixels in the top layer of the template image, using the same method as the source image region.

[0026] For each pixel position within the window, subtract the average gray level of the window region from the pixel's gray level at that position, and then divide by the standard deviation of the gray level of the window region to obtain the normalized source image pixel value at that position. Simultaneously, subtract the average gray level of the template image from the corresponding pixel's gray level in the template image, and then divide by the standard deviation of the template image to obtain the normalized template pixel value. Multiply these two normalized values ​​together.

[0027] Fourth, sum the products at all pixel positions within the window, then divide by the total number of pixels within the window to obtain the zero-mean normalized cross-correlation metric at the current sliding window position. Fifth, record the calculated cross-correlation metric as the matching score for that window position. After traversing all sliding window positions at the top of the pyramid, compare the matching scores of all window positions, and select the coordinates of the window center corresponding to the maximum score as the candidate extreme point coordinates for the top of the pyramid.

[0028] The coordinates of the candidate extremum point are mapped from the top layer of the pyramid to the next layer (the first layer). The mapping method is as follows: multiply the x-coordinate and y-coordinate of the extremum point at the top layer by 2 to obtain the corresponding coordinates on the first layer. On the first layer image, a square search region with sides of 5 pixels is defined, centered on these mapped coordinates. Within this search region, instead of performing a full image traversal, only each possible sliding window position is traversed. Using the same zero-mean normalized cross-correlation calculation method as the top layer, the matching score for each window position is calculated, and the center coordinates of the window with the highest score are selected as the candidate extremum point for the first layer.

[0029] Repeat the coordinate mapping and local search process described above, mapping the candidate extreme point coordinates obtained in the first layer to the bottom layer of the pyramid (layer zero), multiplying the horizontal and vertical coordinates by 2 during the mapping. On the bottom layer image, a square search region with a side length of 5 pixels is delineated with the mapped coordinates as the center. Within this region, all sliding window positions are traversed, and the zero-mean normalized cross-correlation value of each window is calculated. The center coordinates of the window with the highest score are selected as the final matching coordinates of the bottom layer output.

[0030] Using the final matching coordinates of the bottom layer output as the center point of the port detection region, and the original width and height of the template image at layer zero as the width and height of the cropping window, a rectangular region is extracted from layer zero of the source image. This region is the cropped port detection region. This region is not affected by additive noise caused by metallic reflections in the original image.

[0031] In S2, histogram of oriented gradients (HARQ), gray-level co-occurrence matrix (GLCM) texture features, and Hu's invariant rectangular features are extracted from the port detection region. The HARQ features are then subjected to principal component analysis for dimensionality reduction and concatenated with the GLCM texture and Hu's invariant rectangular features to obtain a comprehensive feature vector. This comprehensive feature vector is input into a hierarchical support vector machine. The first layer determines whether material mixing exists, and the second layer identifies the port orientation and outputs classification labels, specifically including: The histogram of oriented gradients (HARQ) features are extracted from the cropped port detection region. The port detection region image is divided into several cells with a side length of 8 pixels. The cumulative gradient magnitudes in 9 gradient directions are counted in each cell, forming a 9-dimensional HARQ feature vector. Adjacent 2x2 cells are combined into a block, and contrast normalization is performed within the block. Finally, the features of all blocks are concatenated to obtain the original HARQ feature vector, which has a dimension of 1764.

[0032] Principal component analysis (PCA) was used to reduce the dimensionality of the 1764-dimensional feature. The PCA was implemented as follows: 1000 images of the port detection areas of different wire ports were pre-acquired, and the histogram of directional gradients (HCG) features of each image were extracted to form a 1764-row, 1000-column data matrix. The covariance matrix of this matrix was calculated, and the eigenvalues ​​and eigenvectors of the covariance matrix were solved. The eigenvalues ​​were sorted from largest to smallest, and the eigenvectors corresponding to the top few eigenvalues ​​with a cumulative contribution rate of 95% were selected as the projection matrix. The cumulative contribution rate was calculated as the sum of the top few eigenvalues ​​divided by the sum of all eigenvalues. It was calculated that 64 principal components were required to achieve a 95% cumulative contribution rate. The inner product operation was performed between the original 1764-dimensional HCG eigenvectors and the projection matrix composed of the 64 eigenvectors. That is, each projection vector was multiplied by the corresponding position of the original eigenvector, and the results were summed to obtain the scores of the 64 principal components, which are the 64-dimensional feature vectors after dimensionality reduction.

[0033] The gray-level co-occurrence matrix (GLCM) texture features of the port detection region are extracted. The port detection region image is converted to 256 gray levels (0-255). The joint occurrence frequency of any two pixels satisfying a specific distance and direction is statistically analyzed to construct the GLCM. The distance parameter is set to 1 pixel, and the direction parameter is the average of four directions: 0°, 45°, 90°, and 135°. Based on the GLCM, six texture statistics are calculated: second angular moment, contrast, inverse moment, entropy, autocorrelation, and variance. Each statistic is calculated using a standardized formula, resulting in a 6-dimensional GLCM texture feature vector. Simultaneously, Hu's invariant rectangular features of the port detection region are extracted. The zeroth moment, first moment, and second central moment of the image are calculated, constructing seven moment values ​​with translation, rotation, and scale invariance. Each moment value is calculated using a specific algebraic combination of the image's normalized central moments, forming a 7-dimensional Hu's invariant rectangular feature vector.

[0034] The three feature vectors are concatenated and fused. The 64-dimensional directional gradient histogram feature, the 6-dimensional gray-level co-occurrence matrix texture feature, and the 7-dimensional Hu's invariant rectangular feature are sequentially concatenated end to end to form a one-dimensional comprehensive feature vector. The total dimension of this vector is 64 + 6 + 7 = 77 dimensions.

[0035] The 77-dimensional comprehensive feature vector is input into a hierarchical support vector machine (SVM) for processing. The first layer of the hierarchical SVM is pre-trained offline to obtain model parameters. The training process is as follows: 2000 images of the port detection regions of the target product and 500 mixed images of other products are collected, and their respective 77-dimensional comprehensive feature vectors are extracted and labeled as the target class or the mixed class. A linear kernel function is used to train the first layer of the SVM, and after training, three parameters are obtained: the set of support vectors, the Lagrange multiplier corresponding to each support vector, and the decision bias term. During online discrimination, the linear kernel function is used to determine whether the current comprehensive feature vector belongs to the target product category. The specific calculation process is as follows: First, the inner product value is calculated between the current 77-dimensional comprehensive feature vector and each support vector stored in the first layer. The inner product is calculated by multiplying the first component of the comprehensive feature vector by the first component of the support vector, the second component by the second component of the support vector, and so on until the 77th component, and summing all the products to obtain an inner product value. For each support vector, this inner product value is multiplied by the Lagrange multiplier corresponding to that support vector to obtain a weighted value. The weighted values ​​of all support vectors are summed, and then the sum is added to the decision bias term to obtain the final discrimination value. If the discrimination value is greater than zero, the current feature vector is determined to belong to the target product category, and the feature vector is allowed to proceed to the second layer of the hierarchical support vector machine. If the discrimination value is less than or equal to zero, it is determined to be a mixed material anomaly, and subsequent calculations are immediately terminated and a mixed material alarm signal is output. The second layer support vector machine adopts a similar training method as the first layer. 1000 front and 1000 back pose images of the target product are pre-collected, 77-dimensional comprehensive feature vectors are extracted, and front or back labels are assigned. The second layer support vector set, Lagrange multiplier set, and decision bias term are trained to obtain the second layer's support vector set, Lagrange multiplier set, and decision bias term. When the feature vectors allowed in the first layer are input into the second layer, the same inner accumulation summation method is used to obtain the second layer's discrimination value. If the value is greater than zero, a front pose label is output; if it is less than or equal to zero, a back pose label is output. The final output classification labels include a product category qualification indicator and a front / back pose label, which are used for the detection logic allocation in subsequent steps.

[0036] In S3, images carrying classification labels are input into a pre-trained transform network. The transform network generates edge enhancement feature maps through convolution of the input features. Residual learning guides the transform network to focus on weak edge regions, outputting a high-resolution binarized segmentation mask, specifically including: The transformation network employs a multi-scale hierarchical visual transformation network as its backbone. After inputting an image carrying classification labels, it sequentially passes through multiple maximum visual transformation blocks for feature extraction, resulting in feature maps at different scales, ranging from local details to global context. The lower-level feature maps retain higher spatial resolution and detail information, while the higher-level feature maps possess larger receptive fields and semantic information.

[0037] The specific steps for edge extraction are as follows: Upsample the high-level feature map to the same spatial size as the low-level feature map using bilinear interpolation, and then adjust the number of channels using a 1x1 convolution kernel so that the number of channels in the high-level feature map matches that in the low-level feature map. Figure 1 The adjusted high-level feature map is concatenated with the low-level feature map along the channel dimension to obtain a fused feature map. Edge enhancement is then performed on this fused feature map, specifically including the following steps: A 3x3 Laplacian kernel is used to perform a convolution operation on the fused feature map. The specific values ​​of the Laplacian kernel are: -1, -1, -1 for the first row; -1, +8, -1 for the second row; and -1, -1, -1 for the third row. At each spatial location in the fused feature map, the convolution operation multiplies the pixel values ​​within the Laplacian kernel's coverage area with the kernel coefficients and sums the results to obtain the multi-directional gradient response value at that location. After performing the same operation on all spatial locations, a multi-directional gradient map with the same spatial dimensions as the fused feature map is output.

[0038] Each gradient response value in the multi-directional gradient plot is constrained to the interval between 0 and 1 using a normalization function. The specific calculation of normalization uses the following mathematical formula: ; in, In the multi-directional gradient plot, the x-axis is located... y-axis The gradient response value at that point, It is a natural constant. This is the normalized gradient value. This formula maps the gradient response of any real value to a probability value between 0 and 1. The larger the gradient response value, the closer it is to 1 after normalization; the smaller the gradient response value, the closer it is to 0.

[0039] The normalized gradient map is multiplied pixel-by-pixel with the fused feature map of the original input. That is, for each spatial location and each channel, the channel value of the fused feature map is multiplied by the normalized gradient value, and the product is used as the new value of that channel at that location. This operation enhances regions with strong gradient responses in the feature map and suppresses regions with weak gradient responses, thereby generating an edge-enhanced feature map.

[0040] The edge enhancement feature map is fed into an edge enhancement inverse attention operation, which erases high-confidence regions through residual learning, forcing the network to focus on weak edges. The specific implementation steps are as follows: First, a convolution operation is performed on the edge enhancement feature map with a 1x1 kernel size and one output channel, resulting in a single-channel prediction confidence map. Each pixel value in this prediction confidence map represents the initial confidence that the location belongs to an edge region.

[0041] A normalization function is applied to the prediction confidence map, limiting the value of each pixel to the range of 0 to 1. The normalization uses the same mathematical formula as described above: ; in, The original value is the output of the convolution. This represents the normalized confidence level.

[0042] Pixels with values ​​greater than 0.8 in the confidence map are marked as high-confidence regions. For these locations, inverse attention weights are calculated by subtracting the confidence value from the value 1. For regions with confidence values ​​less than or equal to 0.8, the inverse attention weight is set to 1. This generates an inverse attention map of the same size as the predicted confidence map. In the fourth step, the inverse attention map is multiplied pixel-by-pixel with the original edge enhancement feature map; that is, for each spatial location and each channel, the channel value of the edge enhancement feature map is multiplied by the inverse attention weight to obtain a weighted feature map. This operation reduces the response intensity of high-confidence regions.

[0043] The weighted feature map is pixel-wise added to the original edge enhancement feature map. Specifically, for each location and channel, the value in the weighted feature map is added to the corresponding value in the original edge enhancement feature map, resulting in an enhanced feature map after residual connections. This operation preserves the effective information in the original feature map while forcing the network to focus on weak edge regions not covered by high-confidence layers in subsequent layers. Finally, this enhanced feature map is passed through upsampling and convolutional layers to output a binary segmentation mask. This mask has the same spatial resolution as the input image, with each pixel taking a value of 0 or 1, where 1 indicates the pixel belongs to a port edge region and 0 indicates it belongs to the background region. The mask has continuous, unbroken edges, accurately depicting the location of weak boundaries.

[0044] A bilateral Gaussian filter is applied to the binarized segmentation mask. The output pixel value of this filter is obtained by a weighted average of all pixels in the neighborhood of the input image, with the weight coefficients depending on both spatial distance and pixel value difference. Specifically, the half-width of the filtering window is set to 3 pixels, i.e., the window size is 7 pixels by 7 pixels. For the target pixel located at x-coordinate i and y-coordinate j in the mask image, the spatial proximity weight is calculated by traversing each pixel position within the window: the square of the Euclidean distance between the target pixel and the pixels within the window is calculated, and this squared value is divided by twice the square of the spatial standard deviation, then the result is negative. The spatial standard deviation is set to 2.0. The pixel value similarity weight is calculated by calculating the square of the difference between the gray value of the target pixel and the gray value of the pixels within the window, and this squared value is divided by twice the square of the pixel value standard deviation, then the result is negative. The pixel value standard deviation is set to 25.0. The spatial proximity weight and the pixel value similarity weight are multiplied to obtain the total weight of the pixels in the window. The total weights of all pixels within the window are summed. Then, the grayscale value of each pixel in the window is multiplied by its corresponding total weight, and the sums are accumulated. Finally, the accumulated sum is divided by the sum of the total weights to obtain the grayscale value of the target pixel after filtering. This process is repeated for all pixels in the mask image to complete the filtering, and the resulting grayscale image is output after edge preservation and noise reduction.

[0045] The Cannibal algorithm is used to extract continuous edges: the gradient magnitude and gradient direction of each pixel in the filtered image are calculated. The gradient magnitude is calculated using the Sobel operator, taking the partial derivatives in the horizontal and vertical directions respectively. The gradient magnitude is the square root of the sum of the squares of the horizontal and vertical partial derivatives. The gradient direction is the arctangent of the quotient obtained by dividing the vertical partial derivative by the horizontal partial derivative, and then converting it into an angle between 0 and 180 degrees.

[0046] Non-maximum suppression is performed along the gradient direction: For each pixel, its gradient magnitude is compared with the gradient magnitudes of its two adjacent pixels in the gradient direction. If the gradient magnitude of the current pixel is not the maximum value, the gradient value of the pixel is set to zero.

[0047] A high threshold and a low threshold are set, with the high threshold set to 150 and the low threshold set to 50. Pixels with gradient magnitudes greater than the high threshold are marked as strong edge pixels, and pixels with gradient magnitudes between the low and high thresholds are marked as weak edge pixels. For weak edge pixels, they are only retained if they are connected to at least one strong edge pixel within an 8-neighborhood; otherwise, they are set to zero. After the above processing, a continuous edge map with a width of one pixel is output.

[0048] The extracted edge contours are approximated using the Lamer-Douglas-Puck curve algorithm to remove redundant pixels. The specific process is as follows: For each independent edge contour, which consists of a series of sequentially arranged pixels, the start and end points of the contour are taken, and a straight line segment is formed connecting these two points. The vertical distance from all intermediate points in the contour to this straight line segment is calculated, and the maximum vertical distance and its corresponding pixel are identified. A distance threshold of 1 pixel is set. If the maximum vertical distance is less than this threshold, all intermediate pixels between the start and end points are removed, retaining only the start and end points as the simplified contour points. If the maximum vertical distance is greater than or equal to the threshold, the contour is divided into two segments using the pixel corresponding to the maximum distance as the dividing point, and the above operation is recursively performed on each segment. After processing all edge contours, the simplified edge contours after removing redundant pixels are output.

[0049] The overall contour is obtained through convex hull analysis, and the pixel height is calculated using the minimum bounding rectangle fitting method. The specific process of convex hull analysis is as follows: all pixels on the simplified edge contour are taken as a point set, and the bottom-left corner of the point set is selected as the reference point. All points are sorted according to their polar angles from smallest to largest. Then, a stack structure is used to traverse the sorted points. For each new point, it is checked whether the direction formed by the top two points of the stack and the new point is counterclockwise. If not, the top of the stack is popped, and this step is repeated until the counterclockwise direction is satisfied. Then, the new point is pushed onto the stack. After the traversal, the points remaining in the stack constitute the convex hull contour. On the convex hull contour, the minimum bounding rectangle is calculated using the rotating caliper method: each edge of the convex hull is taken as an edge of the rectangle, and the perpendicular distance from all points on the convex hull to that edge is calculated. The maximum distance is taken as the height of the rectangle, and the span of the convex hull in the direction perpendicular to that edge is calculated as the width of the rectangle. After traversing all edges, the rectangle with the smallest area is selected as the minimum bounding rectangle. The height of this rectangle represents the height of each pin in the pixel coordinate system. Because continuous coordinate calculations are used during edge extraction and fitting, this height value is accurate to the sub-pixel level. The obtained sub-pixel level pixel height is output to step S5.

[0050] In S5, the pixel height is multiplied by the system-calibrated pixel equivalent coefficient to map to the actual physical size, and compared with a preset process threshold to output the judgment result of good or defective product, specifically including: The pixel equivalent coefficient is calibrated beforehand. A standard calibration block of known actual size is used, with equidistant parallel lines spaced 1 mm apart. The calibration block is placed within the field of view of an industrial camera, and an image of the calibration block is captured using the same object distance and focal length as the image of the wire port. After binarizing the calibration block image, the edges between the parallel lines are extracted, and the pixel spacing of the parallel lines in the image is measured using the minimum bounding rectangle fitting method described in step S4. The actual spacing value of the calibration block (1 mm) is divided by the measured pixel spacing value to obtain the pixel equivalent coefficient, which represents the actual physical size of each pixel in millimeters per pixel. This pixel equivalent coefficient is multiplied by 1000 to convert it to micrometers per pixel and then stored in the calibration file.

[0051] During each measurement, the pixel equivalent coefficient, pre-stored in the calibration file, is read. The sub-pixel height calculated in step S4 is multiplied by this pixel equivalent coefficient to obtain the actual physical size in micrometers. The specific multiplication method is as follows: the pixel height value is directly multiplied by the pixel equivalent coefficient value, and the product is retained to two decimal places as the actual physical size value.

[0052] Based on the classification label output in step S2, retrieve the corresponding standard size upper and lower threshold values. Specifically, pre-create size threshold configuration files for each port type and its front and back orientations. Each configuration file records the standard value of the pin height, the allowable upper deviation, and the allowable lower deviation for that orientation. The upper threshold equals the standard value plus the absolute value of the upper deviation, and the lower threshold equals the standard value minus the absolute value of the lower deviation. All thresholds are stored in micrometers. After obtaining the classification label, search the threshold configuration files for the upper and lower threshold values ​​that match that label.

[0053] The actual physical dimensions are compared with both the upper and lower thresholds. The comparison process is performed in the following order: First, determine if the actual physical dimension is less than the upper threshold. If so, proceed to the second step; otherwise, classify it as a defective product and record the out-of-tolerance value, which is equal to the actual physical dimension minus the upper threshold. Second, determine if the actual physical dimension is greater than the lower threshold. If so, classify it as a good product; otherwise, classify it as a defective product and record the out-of-tolerance value, which is equal to the lower threshold minus the actual physical dimension. When outputting the judgment result, if it is a good product, output a good product label and the actual dimension value; if it is a defective product, output a defective product label, the actual dimension value, and the out-of-tolerance value. The above output results can be used for production line sorting or alarm prompts.

[0054] Please see Figure 2 As shown, Figure 2The images show a comparison of the effects of fine segmentation of the wire port edge before and after the present invention. The comparison demonstrates that the algorithm of this invention can achieve high integrity and high precision segmentation of the weak edges of complex ports.

[0055] Please see Figure 3 As shown, Figure 3 This is a diagram of the Transformer network architecture based on edge guidance and reverse attention mechanisms of this invention.

[0056] The working principle of this invention is as follows: It receives the original digital image, constructs an image pyramid of the source image and the template, calculates the zero-mean normalized cross-correlation value layer by layer starting from the top, maps the extreme value coordinates layer by layer, and quickly crops out the port detection region. It extracts directional gradient histogram features, gray-level co-occurrence matrix texture features, and Hu's invariant rectangular features from this region. After dimensionality reduction by principal component analysis, the directional gradient histogram features are concatenated with the other two features to form a comprehensive feature vector, which is then input into a hierarchical support vector machine. The first layer identifies mixed materials and removes anomalies, and the second layer identifies the front and back poses of the port, outputting classification labels. Then, the image carrying the classification label is... The pre-trained transform network is input, and edge enhancement feature maps are generated through Laplacian operator convolution. Residual learning is used to erase high-confidence regions, forcing the network to focus on weak edges and outputting a high-resolution binarized segmentation mask. Bilateral Gaussian filtering is applied to the mask to preserve edges and denoise, and continuous edges are extracted using the Cannibal algorithm. Redundant points are then approximately removed using the Lamer-Douglas-Puck curve. Subpixel-level pixel heights are obtained through convex hull analysis and minimum bounding rectangle fitting. The pixel heights are multiplied by the calibrated pixel equivalent coefficient to convert them into actual physical dimensions. These dimensions are then compared with the preset process thresholds of the corresponding classification labels to output the good or defective product judgment result.

[0057] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for extracting wire port dimensions based on deep learning, characterized in that, Includes the following steps: S1 receives the original digital image containing the cable port, constructs an image pyramid of the source image and the template image, matches the extreme coordinates layer by layer from the top of the pyramid downwards, and quickly locates and crops out the port detection area. S2, extract the directional gradient histogram features, gray-level co-occurrence matrix texture features and Hu's invariant rectangular features from the port detection area respectively. After performing principal component analysis to reduce the dimensionality of the directional gradient histogram features, it is concatenated and fused with the gray-level co-occurrence matrix texture features and Hu's invariant rectangular features to obtain a comprehensive feature vector. The comprehensive feature vector is input into a hierarchical support vector machine. The first layer determines whether there is material mixing, the second layer identifies the port posture, and outputs the classification label. S3: The image carrying the classification label is input into the pre-trained transformation network. The transformation network generates an edge enhancement feature map by convolving the input features. Residual learning guides the transformation network to focus on weak edge regions and outputs a high-resolution binarized segmentation mask. S4. Apply bilateral Gaussian filtering to the binarized segmentation mask to preserve edges and denoise, extract continuous edges, apply curve approximation algorithm to the extracted edge contours to remove redundant points, combine convex hull analysis to obtain the overall contour, and calculate the sub-pixel level pixel height by fitting the minimum bounding rectangle. S5 multiplies the pixel height by the system-calibrated pixel equivalent coefficient, maps it to the actual physical size, compares it with the preset process threshold, and outputs the judgment result of good or bad product.

2. The method for extracting wire port dimensions based on deep learning according to claim 1, characterized in that, The rapid location and cropping of the port detection area specifically includes: Construct an image pyramid between the source image and the template image, calculate the zero-mean normalized cross-correlation value starting from the top of the pyramid, and determine the center coordinates with the maximum value as candidate regions; Map the coordinates of the candidate region to the next layer, traverse the extreme points only within the mapped region, and iterate layer by layer to the bottom of the pyramid. The port detection area is clipped out using the coordinates output from the underlying layer.

3. The method for extracting wire port dimensions based on deep learning according to claim 2, characterized in that, The calculation of the zero-mean normalized cross-correlation value starting from the top of the pyramid specifically includes: Select the source image region covered by the current sliding window, and calculate the grayscale average and standard deviation of all pixels within the source image region; Calculate the mean and standard deviation of gray levels for all pixels within the template image; The cross-correlation metric of the window position is obtained by subtracting the average gray value of the source image region from each pixel within the window region and dividing it by its standard deviation, and by subtracting the average gray value of the template from each pixel within the template and dividing it by the template standard deviation. The products of the corresponding positions are summed and divided by the total number of pixels within the window. Iterate through all sliding window positions at the top of the pyramid, select the center coordinates of the window with the largest cross-correlation value as the candidate extreme point, and pass the coordinates of the candidate extreme point down to the next layer.

4. The method for extracting wire port dimensions based on deep learning according to claim 1, characterized in that, The output category labels specifically include: Principal component analysis was performed on the directional gradient histogram features to reduce the dimensionality to 64 dimensions, and then concatenated with the 6-dimensional gray-level co-occurrence matrix texture features and the 7-dimensional Hu's invariant rectangular features to form a one-dimensional comprehensive feature vector. The one-dimensional comprehensive feature vector is input into the first layer of the hierarchical support vector machine, and a linear kernel function is used to determine whether it is the target product category. The feature vectors identified as the target product in the first layer are fed into the front and back orientation ports of the second layer to output classification labels.

5. The method for extracting wire port dimensions based on deep learning according to claim 4, characterized in that, The method of using a linear kernel function to determine whether a product falls within the target product category specifically includes: Obtain the pre-stored set of support vectors in the first layer of the hierarchical support vector machine, the Lagrange multipliers corresponding to each support vector, and the decision bias term; Calculate the inner product of the current integrated feature vector and each support vector, multiply each inner product by the corresponding Lagrange multiplier, and then sum them up. The cumulative sum is added to the decision bias term. If the resulting value is greater than zero, it is determined to be the target product category and allowed to proceed to the second level. If it is less than or equal to zero, it is determined to be a mixing anomaly and the calculation is terminated.

6. The method for extracting wire port dimensions based on deep learning according to claim 1, characterized in that, The output high-resolution binarized segmentation mask specifically includes: The edge extraction module inside the transform network first uses the Laplacian operator to convolve the input features to obtain a multi-directional gradient map. Then, after normalization, the gradient map is fused with the input features pixel by pixel to generate an edge enhancement feature map. The edge enhancement feature map is fed into the edge enhancement reverse attention module, and high confidence regions are erased through residual learning, forcing the network to focus on weak edges; Output a binarized segmentation mask with continuous edges.

7. The method for extracting wire port dimensions based on deep learning according to claim 6, characterized in that, The method of erasing high-confidence regions through residual learning, forcing the network to focus on weak edges, specifically includes: Perform a convolution operation on the edge enhancement feature map to generate a single-channel prediction confidence map, and limit the value of each pixel in the prediction confidence map to the range of 0 to 1; Pixels with values ​​greater than 0.8 in the confidence map are marked as high-confidence regions. The reverse attention weights are calculated by subtracting the original value from the confidence value in the high-confidence region by 1, thus obtaining the reverse attention map. The inverse attention map is multiplied pixel by pixel by the edge enhancement feature map, and then added element by element to the edge enhancement feature map itself. The result is used as the enhancement feature map focusing on weak edges and fed into subsequent layers.

8. The method for extracting wire port dimensions based on deep learning according to claim 1, characterized in that, The calculation of sub-pixel level pixel height specifically includes: First, apply a bilateral Gaussian filter to the binarized segmentation mask, and simultaneously calculate spatial proximity weights and pixel value similarity weights to smooth noise and preserve edges; The Kenny algorithm is used to extract continuous edges from the filtered image, and then the Lamer-Douglas-Puk curve approximation algorithm is applied to the extracted edge contours to remove redundant pixels. The overall contour is obtained through convex hull analysis, and the sub-pixel height of each pin is obtained by fitting the minimum bounding rectangle.

9. The method for extracting wire port dimensions based on deep learning according to claim 8, characterized in that, The process of removing redundant pixels specifically includes: The gradient magnitude and gradient direction of each pixel are calculated for the image after bilateral Gaussian filtering. Non-maximum suppression is performed along the gradient direction to set the gradient value of non-edge pixels to zero. Set a high threshold and a low threshold, mark pixels with gradient values ​​higher than the high threshold as strong edge pixels, mark pixels between the low threshold and the high threshold as weak edge pixels, retain only weak edge pixels connected to strong edge pixels, and output a continuous single-pixel wide edge map. For each edge contour extracted from the single-pixel wide edge map, the vertical distance from each intermediate point to the connecting line is calculated by connecting the first and last pixels in sequence. If the maximum vertical distance is less than the preset threshold, the intermediate pixel point corresponding to the maximum distance is removed. Otherwise, the segmentation is recursively performed, and the simplified edge contour after removing redundant pixels is output.

10. The method for extracting wire port dimensions based on deep learning according to claim 1, characterized in that, The comparison with the preset process threshold specifically includes: Read the pixel equivalent coefficients pre-stored in the calibration file, multiply the subpixel-level pixel height by the subpixel-level pixel equivalent coefficients, and convert it into the actual physical size in micrometers; Based on the category label, retrieve the upper and lower threshold values ​​of the standard size corresponding to the category label; The actual physical size is compared with the upper and lower thresholds respectively. If the actual physical size is both less than the upper threshold and greater than the lower threshold, the good product judgment result is output; otherwise, the defective product judgment result is output along with the out-of-tolerance value.

Images