Punch line CCD vision detection system based on industrial vision

Abstract

The application discloses a punch line CCD vision detection system based on industrial vision, and belongs to the technical field of industrial vision detection. The system comprises: an image acquisition module, which synchronously acquires multi-angle surface images of workpieces through multiple CCD industrial cameras and generates a panoramic splicing image; a feature extraction module, which segments a to-be-detected area from the panoramic image, extracts a gray gradient direction chain of each pixel point, and constructs a feature tensor matrix describing a surface texture direction field according to a continuous change relationship of adjacent pixel point direction chains; a difference comparison module, which calculates a difference mapping diagram of the feature tensor matrix and a standard template, and positions a connected area with broken direction chains or abnormally converged direction chains as a primary defect candidate area; and a defect recognition module, which traces back to an original image block corresponding to each candidate area in multi-angle images, inputs the defect classification convolution network, and outputs a confidence vector. The application improves the detection precision and reliability of small texture defects on the surface of punch line workpieces.

Description

Technical Field

[0001] This invention relates to the field of industrial vision inspection technology, specifically to a CCD vision inspection system for punching lines based on industrial vision. Background Technology

[0002] During the punching process, minor defects such as scratches, pits, and burrs often appear on the surface of workpieces. Traditional manual visual inspection is inefficient and has a high rate of missed detection. Existing industrial vision inspection systems mostly use a single CCD camera to acquire workpiece images from a single angle and identify surface defects through image processing algorithms. However, single-view imaging can only obtain a two-dimensional projection of a local area of ​​the workpiece. For defects parallel to the texture direction or with abnormal aspect ratios, the grayscale contrast is extremely low, making it difficult to be effectively identified by conventional edge detection or threshold segmentation algorithms. In addition, the surface of workpieces processed on punching lines usually has regular directional textures. When defects occur, they often cause local texture breaks or abnormal changes in direction. However, existing technologies lack the ability to systematically model the continuity of the texture direction field and rely solely on pixel-level grayscale differences for defect localization, easily misjudging gradient changes in normal textures as defects. At the same time, although multiple CCD cameras can provide multi-angle views, existing methods often use simple image stitching or voting decisions when fusing multi-view information, failing to fully consider the impact of differences in image block sharpness at different viewpoints on the confidence of defect classification, resulting in insufficient accuracy in identifying weak texture defects. This invention constructs a feature tensor matrix based on grayscale gradient direction chains to accurately describe the texture direction field. It also solves the problems of missed detection of texture direction anomalies and insufficient utilization of multi-view information in traditional single-view detection by backtracking original image blocks from multiple angles and fusing confidence vectors based on sharpness weighting. Summary of the Invention

[0003] The purpose of this invention is to provide a CCD vision inspection system for punching lines based on industrial vision, in order to solve the problems of low sensitivity of single-view imaging for detecting defects with abnormal texture direction and insufficient accuracy of defect classification caused by not considering the difference in image clarity from different viewpoints when fusing multi-view information.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] This invention provides a CCD vision inspection system for punching lines based on industrial vision, aiming to solve key technical problems in the surface defect detection of existing punching lines, such as insufficient imaging information from a single viewpoint, low recognition rate of subtle defects in complex texture backgrounds by traditional image processing algorithms, and insufficient utilization of multi-angle image information fusion.

[0006] As a technical solution of this invention, the CCD vision inspection system for a punching line based on industrial vision includes: an image acquisition module, a feature extraction module, a difference comparison module, a defect identification module, and a comprehensive judgment module. The image acquisition module simultaneously acquires multi-angle surface images of the workpiece processed on the punching line using multiple CCD industrial cameras and generates a complete panoramic stitched image of the workpiece. Through multi-angle synchronous acquisition and panoramic stitching, the limitations of single-camera field of view and image distortion are effectively eliminated, providing a rich and geometrically unified global view for subsequent inspection, significantly improving the comprehensiveness of the inspection.

[0007] Preferably, the multiple CCD industrial cameras are arranged in an arc around the punching line processing station, with the optical axes of each CCD industrial camera converging at the center of the workpiece. This arrangement allows the system to capture texture and lighting information of the workpiece surface from multiple directions, making it particularly suitable for workpieces with complex three-dimensional curved surfaces or high reflectivity, and effectively suppressing local overexposure or shadow blind spots caused by a single light source angle.

[0008] Furthermore, the image acquisition module performs a synchronous acquisition and stitching process including: controlling all cameras to simultaneously capture images of the workpiece at a fixed position under the same trigger pulse; reading the pre-calibrated intrinsic parameter matrix and distortion coefficients of each camera and performing distortion correction processing on each surface image; then, based on the extrinsic parameter matrix of each camera, converting the pixel coordinates in the distortion-corrected image to world coordinates; finally, arranging all pixels according to their coordinate positions in the world coordinate system, and taking the average grayscale value of the pixels in the overlapping area to generate a panoramic stitched image of the workpiece. This process ensures the pixel-level accuracy of the panoramic stitched image, providing a reliable geometric benchmark for subsequent feature extraction.

[0009] The feature extraction module is used to segment the region to be detected from the panoramic stitched image of the workpiece and extract the gray-level gradient direction chain of each pixel in the region to be detected. Based on the continuous change relationship of the gray-level gradient direction chains of adjacent pixels, a feature tensor matrix that can accurately describe the surface texture direction field of the region to be detected is constructed. This feature tensor matrix transforms the complex texture information in the image into a structured, low-dimensional mathematical representation, which is extremely sensitive to the directionality, consistency, and abrupt changes of the texture, and can capture minute texture anomalies that are difficult for the human eye to distinguish.

[0010] In a preferred embodiment of the present invention, the feature extraction module operates as follows: An edge detection operator is applied to the panoramic stitched image of the workpiece to extract the workpiece's outline and cut out the workpiece body region as the detection area. Each pixel in the detection area is traversed, and its grayscale gradient components in the horizontal and vertical directions are calculated. The gradient principal direction angle is calculated based on the ratio of the gradient components, and this direction angle is quantized to one of eight preset direction intervals to obtain the pixel's direction chain code. Taking each row of pixels in the detection area as a unit, the direction chain code of the current pixel is concatenated with the direction chain code of the previous pixel to form a direction chain code sequence for each row of pixels. Finally, all the sequences of all rows are stacked in order of row number to form a grayscale gradient direction chain. For each pair of adjacent pixel direction chain codes in the grayscale gradient direction chain, it is determined whether the difference in their direction interval indices is less than a preset difference threshold. If so, it is marked as a continuous change; otherwise, it is marked as a sudden change. Centered on each pixel, this method selects all pixel pairs marked as continuously changing within its neighborhood window. The frequency of occurrence in each directional interval within the neighborhood window is then calculated. The unit direction vector corresponding to the most frequent directional interval is taken as the texture direction vector for that pixel. Finally, the texture direction vectors of all pixels are arranged into a feature tensor matrix. By introducing the analysis of continuous changes in gradient direction chains, this method can effectively distinguish between normal texture paths and abrupt changes, significantly improving the feature tensor matrix's ability to characterize surface defects.

[0011] The difference comparison module is used to calculate the difference mapping between the feature tensor matrix and the pre-stored standard workpiece feature tensor template, and locates connected regions where the gray-level gradient direction chain breaks or converges abnormally in the difference mapping, using these regions as primary defect candidate areas. This comparison method based on the similarity of the direction field structure has stronger robustness to changes in illumination and unevenness of workpiece surface material compared to traditional gray-level value comparison, and can reliably detect defects of various shapes.

[0012] Preferably, the specific execution process of the difference comparison module is as follows: A standard workpiece feature tensor template with the same model as the current workpiece is read from the database. The template size is the same as the feature tensor matrix. The cosine value of the angle between the feature tensor matrix and the texture direction vector at the corresponding position in the standard template is calculated, and the difference value is obtained by subtracting this cosine value. All difference values ​​are arranged according to coordinates to generate a difference mapping map. Specifically, the calculation method is as follows: Let the feature tensor matrix be at position... The texture direction vector at that location is The texture direction vector of the standard template at the same position is Both are unit vectors; the cosine of the included angle Difference value The difference map is binarized, and points with difference values ​​greater than a preset threshold are marked as outliers. An eight-connected region growing algorithm is then used to aggregate all adjacent outliers into connected regions, and connected regions with an area greater than a preset minimum defect area are selected as primary defect candidate regions.

[0013] Furthermore, both the feature tensor matrix and the standard workpiece feature tensor template are stored as two-dimensional matrices of the same size, with each element in the matrix being a two-dimensional unit direction vector. This data storage format facilitates efficient vectorization operations, improving the computational efficiency of the difference comparison module.

[0014] The defect identification module is used to backtrack the corresponding original image patch in the multi-angle surface image for each primary defect candidate region, and input each original image patch into a pre-trained defect classification convolutional network, outputting a confidence vector that each original image patch belongs to a preset defect category. By backtracking to the original image, information loss that may be caused by panoramic stitching or feature extraction processes is avoided, and the powerful feature learning capability of deep learning models is utilized to achieve accurate identification of defect types, effectively reducing the false detection rate.

[0015] As a technical solution of the present invention, the specific method by which the defect recognition module backtracks the original image block is as follows: All coordinate points of the primary defect candidate area in the world coordinate system are obtained. Based on the extrinsic parameter matrix of each CCD industrial camera, these world coordinate points are projected backwards onto the pixel coordinate system of each camera to obtain the set of projected pixels corresponding to each camera. In the distortion-free image corresponding to each camera, the original image block is cut out by expanding outwards by a preset pixel width, centered on the bounding rectangle of the projected pixel set. Subsequently, each original image block is scaled to the input size of the defect classification convolutional network and input into the network. The network outputs a confidence vector with a dimension equal to the preset number of defect categories.

[0016] Preferably, the defect classification convolutional network adopts the lightweight MobileNetV3 architecture and uses the label-smooth cross-entropy loss function during the training phase. The MobileNetV3 architecture ensures high inference speed and low computational resource consumption, making it suitable for deployment in industrial settings; the label-smooth cross-entropy loss function helps improve the model's generalization ability and prevents overfitting.

[0017] The comprehensive judgment module is used to perform weighted fusion of the confidence vectors corresponding to all primary defect candidate areas. The weighting coefficient is determined by the image sharpness of each original image block in the multi-angle surface images, and the workpiece is judged as qualified based on the weighted fused confidence vector. Through weighted fusion, the system can automatically assign higher weights to images with high sharpness, effectively combining multi-view information, suppressing interference to single-view judgment caused by motion blur, inaccurate focus, or local reflections, and significantly improving the accuracy and stability of the judgment.

[0018] In a preferred embodiment of the present invention, the determination process of the comprehensive determination module is as follows: For each original image block, calculate the Laplacian variance of all pixels within it, and use this as the image sharpness of that image block. Multiply the confidence vector corresponding to each original image block by the image sharpness of that image block to obtain a weighted confidence vector; add all weighted confidence vectors element-wise to obtain a fused confidence vector; then divide each element in the fused confidence vector by the sum of the image sharpness of all original image blocks to obtain a normalized fused confidence vector. Finally, extract the maximum confidence value and its corresponding defect category from the normalized fused confidence vector. If the maximum confidence value is greater than a preset acceptable threshold and the corresponding defect category is a defect-free category, the workpiece is determined to be acceptable; otherwise, it is determined to be unacceptable.

[0019] Preferably, the Laplacian variance is calculated by convolving the original image block with a Laplacian operator template to obtain the second derivative response of each pixel, and then calculating the variance of the response values ​​of all pixels. This metric can effectively reflect the sharpness of image edges, thereby accurately characterizing the image clarity.

[0020] As another technical solution of the present invention, an initial configuration step is included before system operation. This involves installing at least two CCD industrial cameras above and to the side of the punch press conveyor belt, with the angle between the optical axis of each camera and the normal to the workpiece surface maintained within a preset angle range. A coaxial light source or ring light source is installed in front of the lens of each camera, and the illumination angle and brightness of the light source are adjusted to ensure that the reflective intensity of the workpiece surface is uniformly distributed within the dynamic response range of the CCD camera. Simultaneously, each CCD industrial camera is jointly calibrated to obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system. These preliminary steps ensure that the system can stably and reliably acquire high-quality multi-angle images, laying the foundation for the entire inspection process.

[0021] Preferably, the generation process of the standard workpiece feature tensor template includes: selecting multiple qualified standard workpiece samples, performing the same image acquisition, stitching, region segmentation, direction chain extraction, and feature tensor matrix construction steps as described above on each sample; then taking the median of the texture direction vectors at the same position in the feature tensor matrices of multiple standard samples to generate a median feature tensor matrix, and storing it in the database as the standard workpiece feature tensor template, associated with the model information of the current workpiece. Using the median statistical method to construct the template can effectively suppress random noise in individual samples and generate a more representative standard direction field.

[0022] Preferably, the training process of the defect classification convolutional network includes: collecting multiple sets of original image blocks labeled as defective and original image blocks labeled as defect-free from historical detections; labeling each original image block with its corresponding true defect category, such as scratches, dents, burrs, and defect-free categories; expanding the training sample set using data augmentation methods such as image rotation, translation, and brightness adjustment; constructing a defect classification convolutional network containing multiple convolutional layers, multiple pooling layers, and two fully connected layers, where the number of neurons in the last fully connected layer is equal to the number of preset defect categories, and a softmax activation function is used to output a confidence vector; and supervising training of the network is performed using the training sample set, updating the network weights using a cross-entropy loss function until convergence. This training process ensures that the network can learn the discriminative features of various defects and has good classification ability.

[0023] As an extended technical solution of this invention, after determining that a workpiece is unqualified, the system further performs subsequent processing steps: It extracts the defect category corresponding to the maximum confidence value from the normalized fusion confidence vector as the defect type of the workpiece, and records the position coordinates of the defect type and the primary defect candidate area in the world coordinate system into the detection log. Based on these position coordinates, it calculates the center point coordinates of the primary defect candidate area and converts them into the gripping coordinates of the punch press line robot, sending them along with the defect type to the sorting actuator to control it to remove the unqualified workpiece from the conveyor belt. This automated closed-loop process achieves complete automation from defect detection to sorting and rejection, significantly improving the intelligence level and production efficiency of the stamping production line.

[0024] The beneficial effects of this invention are:

[0025] This method employs a feature tensor matrix constructed based on grayscale gradient direction chains to describe the surface texture direction field of the area to be detected. The area to be detected is segmented from the panoramic stitched image of the workpiece. The grayscale gradient components of each pixel in the horizontal and vertical directions are calculated. The dominant gradient direction angle is obtained based on the ratio and quantized into one of eight direction intervals. This generates a direction chain code sequence for each row of pixels, which is then stacked row by row to form a grayscale gradient direction chain. By judging the continuous change relationship of the direction chain codes of adjacent pixels (whether the difference is less than a preset difference threshold), the frequency of continuously changing pixel pairs in each direction interval is counted within a neighborhood window centered on each pixel. The unit direction vector corresponding to the direction interval with the highest frequency is taken as the texture direction vector of that pixel. Finally, the texture direction vectors of all pixels are arranged into a feature tensor matrix. Compared to traditional methods that directly use grayscale gradient magnitude or single-direction features, this scheme encodes pixel-level gradient direction information into a spatially continuous direction chain and extracts the dominant texture direction in the local neighborhood, thereby constructing a high-dimensional tensor feature that reflects both the global texture direction and the consistency of local directions. The effect is as follows: when defects such as scratches and pits appear on the surface of a workpiece, the texture direction of the defect area will break or converge abnormally. This abrupt change in the direction field is manifested in the feature tensor matrix as the consistency between the texture direction vector and the surrounding normal area being disrupted. Through subsequent difference mapping with the standard template, the connected regions where the gradient direction chain changes abnormally can be accurately located. Even if the gray-level contrast of the defect area is extremely low, it can be detected as long as the texture direction changes, significantly improving the ability of traditional pixel-level gray-level detection methods to perceive weak texture defects. The method involves backtracking the original image blocks in multi-angle surface images of the primary defect candidate area and using image sharpness weighted fusion of confidence vectors to complete the final defect determination. Specifically, all coordinate points of the primary defect candidate area in the world coordinate system are obtained, and the extrinsic parameter matrix of each CCD camera is back-projected to the pixel coordinate system of each camera. In the distortion-free image, the original image block is cut out with a preset width centered on the bounding rectangle of the projected pixel point. Each original image block is scaled and input into the defect classification convolutional network, which outputs a confidence vector belonging to the preset defect category. Subsequently, the Laplacian variance of each original image patch is calculated as the image sharpness. The sharpness is multiplied by the confidence vector to obtain a weighted vector. All weighted vectors are summed element-wise and divided by the sum of sharpness to obtain a normalized fused confidence vector. Finally, the defect category and threshold corresponding to the highest confidence in the fused vector are used to determine whether the workpiece is qualified. Compared with conventional multi-view simple averaging or direct stitching methods, this scheme addresses the issue that the sharpness of image patches obtained from different angles for the same candidate defect area often varies (e.g., the defect area may be occluded or located at the edge of depth of field, causing blurring at a certain angle). If the confidence of all views is treated equally, the unreliable classification results given by blurred image patches will interfere with the final judgment.By introducing Laplace variance clarity as a weighting coefficient, the confidence of high-resolution views dominates the fusion process, while the contribution of low-resolution views is effectively suppressed. This avoids the negative impact of low-quality images on decision-making while complementing information from multiple perspectives, thus improving the robustness and accuracy of defect classification under complex lighting and workpiece posture changes. Attached Figure Description

[0026] The invention will now be further described with reference to the accompanying drawings.

[0027] Figure 1 This is a flowchart of the CCD vision inspection system for punching lines based on industrial vision described in this invention;

[0028] Figure 2 This is a flowchart of multi-view image acquisition and panoramic stitching of workpieces on a punching line;

[0029] Figure 3 This is a flowchart of the training and detection process for a convolutional network for classifying and detecting surface defects on a workpiece. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] See Figure 1This invention provides a CCD vision inspection system for a punching line based on industrial vision, including an image acquisition module, a feature extraction module, a difference comparison module, a defect recognition module, and a comprehensive judgment module. The image acquisition module simultaneously acquires multi-angle surface images of workpieces processed on the punching line using multiple CCD industrial cameras, generating a panoramic stitched image of the workpiece. The feature extraction module segments the area to be inspected from the panoramic stitched image, extracts the gray-level gradient direction chain of each pixel in the area to be inspected, and constructs a feature tensor matrix describing the surface texture direction field of the area to be inspected based on the continuous change relationship of the gray-level gradient direction chains of adjacent pixels. The difference comparison module calculates the difference mapping map between the feature tensor matrix and a pre-stored standard workpiece feature tensor template, locating connected regions where the gray-level gradient direction chains are broken or abnormally converged in the difference mapping map, as primary defect candidate areas. The defect recognition module, for each primary defect candidate area, traces back to its corresponding original image block in the multi-angle surface images, inputs each original image block into a pre-trained defect classification convolutional network, and outputs a confidence vector indicating that each original image block belongs to a preset defect category. The comprehensive judgment module performs weighted fusion on the confidence vectors corresponding to all primary defect candidate areas. The weighting coefficient is determined by the image clarity of each original image block in the multi-angle surface image. The workpiece is judged as qualified based on the weighted fusion confidence vector.

[0032] Example 1: In specific implementation, refer to Figure 2 At least two CCD industrial cameras are installed above and to the side of the conveyor belt on the punching line. The angle between the optical axis of each CCD industrial camera and the normal to the workpiece surface is maintained within a preset angle range. A coaxial light source or a ring light source is installed in front of the lens of each CCD industrial camera. The illumination angle and brightness of the light source are adjusted to ensure that the reflective intensity of the workpiece surface is uniformly distributed within the dynamic response range of the CCD industrial camera. Each CCD industrial camera is jointly calibrated to obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system.

[0033] Multiple CCD industrial cameras are arranged in an arc around the punching line processing station, with the optical axes of each CCD industrial camera converging at the center of the workpiece.

[0034] Multiple CCD industrial cameras are controlled to simultaneously capture images of workpieces at fixed positions on a punch press line under the same trigger pulse, with each CCD camera outputting a surface image. The pre-calibrated intrinsic parameter matrix and distortion coefficients of each CCD camera are read, and distortion correction processing is performed on each surface image to obtain a distortion-corrected image. Based on the extrinsic parameter matrix of each CCD camera, the pixel coordinates of each pixel in the distortion-corrected image are converted to world coordinates. All pixels are arranged according to their world coordinate positions, and the grayscale average of pixels in the overlapping world coordinate regions is taken to form a panoramic stitched image of the workpiece.

[0035] Example 2: In specific implementation, the Canny edge detection operator is applied to the panoramic stitched image of the workpiece to extract the workpiece's outline. Based on the workpiece's outline, the workpiece body area is cut out from the panoramic stitched image and used as the area to be detected.

[0036] Iterate through each pixel in the region to be detected and calculate the gray-level gradient component of that pixel in the horizontal direction. and grayscale gradient components in the vertical direction ,in: and These are obtained by convolving the image grayscale values ​​with the Sobel operator, respectively. Based on the grayscale gradient components... and The ratio is used to calculate the gradient principal direction angle of the pixel. , ,in: The gray-level gradient component is in the horizontal direction. This represents the vertical grayscale gradient component. The principal direction angle of the gradient is... The pixel is quantized to one of eight preset directional intervals to obtain the directional chain code of the pixel. The eight directional intervals are divided at 45-degree intervals and the interval numbers are 0 to 7.

[0037] For each row of pixels in the region to be detected, the direction chain code of the current pixel is concatenated with the direction chain code of the previous pixel to generate a direction chain code sequence for each row of pixels. All the direction chain code sequences are stacked in row number order to form a grayscale gradient direction chain. The grayscale gradient direction chain is a two-dimensional array with the same number of rows and columns as the region to be detected, and each element in the array is the direction chain code of the corresponding pixel.

[0038] For each pair of adjacent pixels in the grayscale gradient direction chain, determine whether the difference between their direction interval indices is less than a preset difference threshold. The preset difference threshold is set to 1. If the difference between the direction interval indices is less than 1, the direction chain codes of the pair of adjacent pixels are marked as continuously changing. If the difference between the direction interval indices is greater than or equal to 1, the direction chain codes of the pair of adjacent pixels are marked as abruptly changing.

[0039] Centered on each pixel, select all pixel pairs marked as continuously changing within its neighborhood window. The neighborhood window size is set to 3×3 pixels. Count the frequency of each directional interval within the neighborhood window. Use the unit direction vector corresponding to the most frequent directional interval as the texture direction vector for that pixel. The unit direction vectors correspond to angles of 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315° respectively, based on the directional interval numbers 0 to 7. Arrange the texture direction vectors of all pixels according to their pixel positions into a feature tensor matrix. The size of the feature tensor matrix is ​​the same as the size of the region to be detected, and each element in the matrix is ​​a two-dimensional unit direction vector.

[0040] Example 3: In the specific implementation, multiple qualified standard workpiece samples are selected. For each standard workpiece sample, the following steps are performed: synchronous image acquisition using multiple CCD industrial cameras, generation of a panoramic stitched image of the workpiece, segmentation of the area to be inspected, extraction of grayscale gradient direction chains, and construction of a feature tensor matrix, resulting in a feature tensor matrix for each standard workpiece sample. The median of the texture direction vectors at the same position is taken from the feature tensor matrices of multiple standard workpiece samples to generate a median feature tensor matrix. The median feature tensor matrix is ​​stored in the database as a standard workpiece feature tensor template and associated with the model information of the current workpiece. Both the feature tensor matrix and the standard workpiece feature tensor template are stored as two-dimensional matrices of the same size, with each element in the matrix being a two-dimensional unit direction vector.

[0041] Retrieves the standard workpiece feature tensor template corresponding to the standard workpiece with the same model as the current workpiece from the database. The size of the standard workpiece feature tensor template is the same as the size of the feature tensor matrix.

[0042] Calculate the cosine of the angle between the feature tensor matrix and the texture direction vector at the corresponding position in the standard workpiece feature tensor template. Let the feature tensor matrix be at position... The texture direction vector at that location is The texture direction vectors at the same position of the standard workpiece feature tensor template are: Both are unit vectors. Calculate the cosine of the included angle. The formula is:

[0043]

[0044] in: Represents the dot product of two vectors. and Representing vectors respectively sum vector Since both vectors are unit vectors, their magnitudes are both 1.

[0045] The difference value is obtained by subtracting the cosine of the included angle from 1. The calculation formula is: Difference value The range of values ​​is When the two directions are exactly the same When completely opposite The difference values ​​at each location are arranged according to their location coordinates to generate a difference mapping map.

[0046] The difference map is binarized, and locations with difference values ​​greater than a preset difference threshold are marked as outliers, while those less than or equal to the preset difference threshold are marked as normal points. An eight-connected region growing algorithm is used to aggregate all adjacent outliers into connected regions. The area of ​​each connected region is calculated, and connected regions with areas greater than a preset minimum defect area are selected as connected regions where the gray-level gradient direction chain has broken or abnormally converged, and are marked as primary defect candidate regions.

[0047] Example 4: In specific implementation, refer to Figure 3 The process involves obtaining all world coordinate points of the primary defect candidate region in the world coordinate system. Based on the extrinsic parameter matrix of each CCD industrial camera, each world coordinate point is projected backwards into the pixel coordinate system of each CCD industrial camera, resulting in the set of projected pixels for each CCD industrial camera. For each CCD industrial camera, in its corresponding distortion-free image, a preset extended pixel width is used as the center to expand outwards from the bounding rectangle of the projected pixel set, thus cutting out the original image block.

[0048] The original image patches cut from each CCD industrial camera are scaled down to the input size of the defect classification convolutional network. The defect classification convolutional network uses a lightweight MobileNetV3 architecture as its backbone, which contains multiple depthwise separable convolutional layers and multiple pooling layers. The backbone is followed by two fully connected layers. The number of neurons in the last fully connected layer is equal to the number of preset defect categories, and a softmax activation function is used to output a confidence vector. A label-smoothed cross-entropy loss function is used during the training phase.

[0049] We collected multiple sets of original image blocks marked as defective and original image blocks marked as defect-free from historical detections. For each original image block, we labeled its corresponding true defect category, which included scratch defects, pit defects, burr defects, and no defect category. We then expanded the number of labeled original image blocks using data augmentation techniques such as image rotation, image translation, and brightness adjustment. These expanded original image blocks were used as the training sample set.

[0050] The defect classification convolutional network is trained in a supervised manner using the training sample set. The network weights are updated using the label smooth cross-entropy loss function until the loss function value converges, resulting in a well-trained defect classification convolutional network.

[0051] The scaled original image patches are sequentially input into the defect classification convolutional network, which outputs a confidence vector with a dimension equal to the preset number of defect categories.

[0052] Example 5: In specific implementation, for each original image block, the Laplacian variance of all pixels in the original image block is calculated, and this Laplacian variance is used as the image sharpness of the original image block. The Laplacian variance is calculated by convolving the original image block with a Laplacian operator template to obtain the second derivative response of each pixel, and then calculating the variance of the response values ​​of all pixels. The Laplacian operator template uses a standard 3×3 template with the following template coefficients: the center element is 4, the adjacent elements (top, bottom, left, and right) are -1, and the remaining elements are 0. The second derivative response value of each pixel is obtained after the convolution operation. ,in For pixel indexing. The original image patch contains Pixels, Laplacian variance The calculation formula is:

[0053]

[0054] in: This represents the total number of pixels in the original image block. For pixels The second derivative response value, The mean of the second derivative response values ​​of all pixels. .

[0055] Multiply the confidence vector corresponding to each original image block by the image sharpness of the original image block to obtain a weighted confidence vector. Add all weighted confidence vectors element-wise to obtain a fused confidence vector. Divide each element in the fused confidence vector by the sum of the image sharpness of all original image blocks to obtain a normalized fused confidence vector. Extract the maximum confidence value and its corresponding defect category from the normalized fused confidence vector. If the maximum confidence value is greater than a preset acceptable threshold and the corresponding defect category is "no defect," the workpiece is deemed acceptable; otherwise, the workpiece is deemed unacceptable.

[0056] When a workpiece is determined to be defective, the defect category corresponding to the maximum confidence value is extracted from the normalized fused confidence vector as the defect type of the workpiece. The defect type and the position coordinates of the primary defect candidate area in the world coordinate system are recorded in the inspection log. Based on the position coordinates of the primary defect candidate area in the world coordinate system, the center point coordinates of the primary defect candidate area are calculated. The center point coordinates are converted into the gripping coordinates of the robot arm on the punch press line. The robot arm gripping coordinates and the defect type are sent to the sorting actuator of the punch press line to control the sorting actuator to remove the defective workpiece from the conveyor belt.

[0057] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A CCD vision inspection system for punching lines based on industrial vision, characterized in that, include: The image acquisition module simultaneously acquires multi-angle surface images of workpieces processed on the punching line using multiple CCD industrial cameras, and generates a panoramic stitched image of the workpiece. The feature extraction module segments the area to be detected from the panoramic stitched image of the workpiece, extracts the gray-level gradient direction chain of each pixel in the area to be detected, and constructs a feature tensor matrix describing the surface texture direction field of the area to be detected based on the continuous change relationship of the gray-level gradient direction chains of adjacent pixels. The difference comparison module calculates the difference mapping between the feature tensor matrix and the pre-stored standard workpiece feature tensor template. It locates connected regions where the gray-level gradient direction chain breaks or converges abnormally in the difference mapping, and uses them as primary defect candidate regions. The defect identification module backtracks the original image blocks corresponding to each primary defect candidate region in the multi-angle surface image, inputs each original image block into the pre-trained defect classification convolutional network, and outputs the confidence vector of each original image block belonging to the preset defect category. The comprehensive judgment module performs weighted fusion on the confidence vectors corresponding to all primary defect candidate areas. The weighting coefficient is determined by the image clarity of each original image block in the multi-angle surface image. The module determines whether the workpiece is qualified based on the weighted fusion confidence vector.

2. The CCD vision inspection system for punching lines based on industrial vision according to claim 1, characterized in that, The process involves simultaneously acquiring multi-angle surface images of workpieces processed on the punching line using multiple CCD industrial cameras to generate a panoramic stitched image of the workpiece. The images captured by each CCD industrial camera are mapped to a unified world coordinate system based on their corresponding spatial coordinates. Multiple CCD industrial cameras are controlled to simultaneously capture images of workpieces at fixed positions on a punching line under the same trigger pulse, with each CCD industrial camera outputting a surface image. Read the pre-calibrated intrinsic parameter matrix and distortion coefficients of each CCD industrial camera, and perform distortion correction processing on each surface image to obtain a distortion-corrected image; Based on the extrinsic parameter matrix of each CCD industrial camera, the pixel coordinates of each pixel in the distortion-free image are converted into world coordinates. All pixels are arranged in the world coordinate system according to their world coordinate positions. The average grayscale value of the pixels in the overlapping area of ​​the world coordinates is taken to form a panoramic stitched image of the workpiece.

3. The CCD vision inspection system for punching lines based on industrial vision according to claim 1, characterized in that, The multiple CCD industrial cameras are arranged in an arc around the punching line processing station, with the optical axes of each CCD industrial camera converging at the center of the workpiece.

4. The CCD vision inspection system for punching lines based on industrial vision according to claim 2, characterized in that, The process involves segmenting the area to be inspected from the panoramic stitched image of the workpiece, extracting the grayscale gradient direction chain of each pixel in the area to be inspected, and constructing a feature tensor matrix describing the surface texture direction field of the area to be inspected based on the continuous change relationship of the grayscale gradient direction chains of adjacent pixels. Specifically: An edge detection operator is applied to the panoramic stitched image of the workpiece to extract the outline of the workpiece. Based on the outline, the workpiece body area is cut out from the panoramic stitched image of the workpiece and the workpiece body area is used as the area to be detected. Traverse each pixel in the region to be detected, calculate the gray-level gradient components of the pixel in the horizontal and vertical directions, calculate the principal gradient direction angle of the pixel based on the ratio of the gray-level gradient components, quantize the principal gradient direction angle to one of the eight preset direction intervals, and obtain the direction chain code of the pixel. Taking each row of pixels in the area to be detected as a unit, the direction chain code of the current pixel is concatenated with the direction chain code of the previous pixel to generate the direction chain code sequence of each row of pixels. The direction chain code sequences of all rows are stacked in order of row number to form a grayscale gradient direction chain. For each pair of adjacent pixels in the grayscale gradient direction chain, determine whether the difference between their direction interval indices is less than a preset difference threshold. If so, mark the direction chain codes of the pair of adjacent pixels as continuously changing; otherwise, mark them as abrupt changes. Centered on each pixel, select all pixel pairs marked as continuously changing within its neighborhood window, count the frequency of occurrence of each directional interval within the neighborhood window, take the unit direction vector corresponding to the direction interval with the highest frequency as the texture direction vector of that pixel, and arrange the texture direction vectors of all pixels into a feature tensor matrix.

5. The CCD vision inspection system for punching lines based on industrial vision according to claim 4, characterized in that, The difference mapping between the calculated feature tensor matrix and the pre-stored standard workpiece feature tensor template is used to locate connected regions where the gray-level gradient direction chain breaks or converges abnormally, serving as primary defect candidate regions. Specifically: Read the standard workpiece feature tensor template corresponding to the standard workpiece with the same model as the current workpiece from the database. The size of the standard workpiece feature tensor template is the same as the size of the feature tensor matrix. Calculate the cosine of the angle between the feature tensor matrix and the texture direction vector at the corresponding position in the standard workpiece feature tensor template. Subtract the cosine of the angle from the result to obtain the difference value. Arrange the difference values ​​at each position according to their position coordinates to generate a difference mapping map. The difference map is binarized, and the positions with difference values ​​greater than a preset difference threshold are marked as outliers, while the positions with difference values ​​less than or equal to the preset difference threshold are marked as normal points. An eight-connected region growth algorithm is used to aggregate all adjacent anomalies into connected regions. The area of ​​each connected region is calculated, and connected regions with an area greater than the preset minimum defect area are selected as connected regions where the gray-level gradient direction chain has broken or converged abnormally, and are marked as primary defect candidate regions.

6. The CCD vision inspection system for punching lines based on industrial vision according to claim 5, characterized in that, The feature tensor matrix and the standard workpiece feature tensor template are both stored in two-dimensional matrices of the same size, with each element in the matrix being a two-dimensional unit direction vector.

7. The CCD vision inspection system for punching lines based on industrial vision according to claim 5, characterized in that, For each primary defect candidate region, its corresponding original image patch in the multi-angle surface image is traced back. Each original image patch is then input into a pre-trained defect classification convolutional network, and the confidence vector of each original image patch belonging to a preset defect category is output. Specifically: Obtain all world coordinate points of the primary defect candidate area in the world coordinate system. Based on the extrinsic parameter matrix of each CCD industrial camera, project each world coordinate point in reverse onto the pixel coordinate system of each CCD industrial camera to obtain the set of projected pixels corresponding to each CCD industrial camera. For each CCD industrial camera, in its corresponding distortion-free image, the original image block is cut out by extending the bounding rectangle of the projected pixel set outward by a preset extended pixel width. The original image blocks cut from each CCD industrial camera are scaled to the input size of the defect classification convolutional network. The scaled original image blocks are then sequentially input into the defect classification convolutional network, which outputs a confidence vector with a dimension equal to the preset number of defect categories.

8. The CCD vision inspection system for punching lines based on industrial vision according to claim 7, characterized in that, The defect classification convolutional network adopts a lightweight MobileNetV3 architecture and uses a label smoothing cross-entropy loss function during the training phase.

9. The CCD vision inspection system for punching lines based on industrial vision according to claim 7, characterized in that, The process involves weighted fusion of the confidence vectors corresponding to all primary defect candidate regions. The weighting coefficients are determined by the image sharpness of each original image block in the multi-angle surface image. The workpiece is then judged as qualified based on the weighted fused confidence vector. Specifically: For each original image block, calculate the Laplacian variance of all pixels in the original image block, and use the Laplacian variance as the image sharpness of the original image block. Multiply the confidence vector corresponding to each original image block by the image sharpness of that original image block to obtain a weighted confidence vector. Add all the weighted confidence vectors element by element to obtain the fused confidence vector. Divide each element in the fusion confidence vector by the sum of the image sharpness of all the original image blocks to obtain the normalized fusion confidence vector; Extract the maximum confidence value and its corresponding defect category from the normalized fusion confidence vector. If the maximum confidence value is greater than the preset qualified threshold and the corresponding defect category is the no-defect category, the workpiece is determined to be qualified; otherwise, the workpiece is determined to be unqualified.

10. The CCD vision inspection system for punching lines based on industrial vision according to claim 9, characterized in that, The Laplacian variance is calculated by convolving the original image block with the Laplacian operator template to obtain the second derivative response of each pixel, and then calculating the variance of the response values ​​of all pixels.

Images