Visual inspection method for crimping quality of automobile wire harness

Abstract

The invention relates to the field of automobile electronic product detection, in particular to a visual detection method for the crimping quality of an automobile wire harness. And performing denoising, graying and binarization processing on the wire harness image to determine a white wire harness communication area. And calculating the number of the wire harnesses through the area, and comparing the number with a preset standard value. If yes, edge extraction is carried out to determine the length of the wire harness, and whether the length is within a preset size range is judged; and after the conditions are met, segmenting the image into single wire harness images according to a preset sequence, calculating a first-order color moment and a second-order color moment of an HSV color space, and comparing the first-order color moment and the second-order color moment with a template wire harness value. And if the difference value is within the threshold range, intercepting a wire harness crimping area and generating a binary image. And utilizing the pre-trained defect instance segmentation model to detect the defect of the crimping area, generating a defect mask, and calculating the compactness of the contact surface. If no defect exists and the compactness meets the standard, qualified wire harness detection is completed, the number, length and color characteristics of the wire harnesses and the defects of the crimping area can be rapidly identified, and the detection efficiency and accuracy are remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of automotive electronic product detection, and more specifically, to a visual inspection method for the crimping quality of automotive wiring harnesses. Background Art

[0002] During the production process of automotive wiring harnesses, the detection of crimping quality is an important link to ensure the safety and reliability of automotive electrical systems. Currently, the detection of automotive wiring harness crimping quality mainly uses methods such as mechanical detection, electrical detection, and microscopic observation. Mechanical detection evaluates the crimping strength and consistency by applying external forces. Electrical detection uses means such as continuity testing and insulation resistance testing to judge the conductivity and insulation performance of the crimping points. Microscopic observation uses a microscope or industrial camera to magnify and analyze the appearance characteristics of the crimping part.

[0003] However, these traditional detection methods have obvious defects and limitations. First of all, mechanical detection and microscopic observation usually require manual participation, the detection process is cumbersome, the efficiency is low, and it is difficult to meet the production requirements of large quantities and high rhythms in the modern automotive industry. Secondly, the electrical detection method cannot visually evaluate the appearance quality of the crimping points, and it is difficult to detect problems such as over-crimping or under-crimping, lacking a comprehensive evaluation of the multi-dimensional quality characteristics of the wiring harness crimping. In addition, the traditional detection methods are affected by human subjective judgment or equipment accuracy, and it is difficult to guarantee the consistency of the detection results, which is prone to misjudgment or missed judgment.

[0004] With the rapid development of the automotive industry, higher requirements are put forward for the accuracy and efficiency of the detection of wiring harness crimping quality. Therefore, developing a detection method for the crimping quality of automotive wiring harnesses that is efficient, accurate, and intelligent has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present application provides a visual inspection method for the crimping quality of automotive wiring harnesses, which combines modern visual inspection and artificial intelligence technology to achieve the high-efficiency, intelligence, and automation of the detection of the crimping quality of automotive wiring harnesses.

[0006] The technical solution provided by the present application is as follows:

[0007] A visual inspection method for the crimping quality of automotive wiring harnesses, which sequentially detects the core wire sequence and size, and detects the crimping defect of the wiring harness based on the wiring harness image. The method includes:

[0008] Performing grayscale conversion, denoising, and binarization processing on the wiring harness image to determine the white wiring harness connected region of the wiring harness image;

[0009] Based on the white wiring harness connected region, determining the number of wiring harnesses;

[0010] If the number of wire harnesses is consistent with the first preset standard value, edge extraction is performed on the wire harness image to obtain the edge pixel points of the wire harness for determining the length of the wire harness;

[0011] If the length of the wire harness meets the preset size range, the wire harness image is segmented into multiple single wire harness images in the preset initial order;

[0012] Extract the first-order color moment and second-order color moment of each single wire harness image in the HSV color space and compare them with the difference between the template wire harness values at the corresponding positions of the single wire harness;

[0013] If the differences are all within the preset threshold range, the wire harness crimping area in the wire harness image is intercepted, and a binary image of the wire harness crimping area is generated;

[0014] Based on a pre-trained crimping area defect instance segmentation model, the binary image is detected to generate a defect mask for marking the crimping defects in the wire harness crimping area and determining the compactness of the crimping area, where the compactness is determined by the perimeter and area of the contact surface of the crimping area;

[0015] If no crimping defects are detected and the compactness is consistent with the second preset standard value, the quality inspection of the qualified wire harness is completed.

[0016] In one possible implementation, the grayscale conversion, denoising, and binarization processing of the wire harness image include:

[0017] Convert the wire harness image to the HSV color space and take the lightness V component as the grayscale image of the wire harness image;

[0018] Perform median filtering on the grayscale image to remove the noise in the image and smooth the image;

[0019] Perform binarization processing on the denoised grayscale image to segment the wire harness image into a black background area and a white wire harness connected area.

[0020] In one possible implementation, based on the white wire harness connected area, determining the number of wire harnesses includes:

[0021] Perform erosion operation on the binarized wire harness image to remove the redundant noise points to obtain the final wire harness image; wherein, the white wire harness connected area of the final wire harness image is only composed of wire harnesses for determining the number of wire harnesses.

[0022] In one possible implementation, performing edge extraction on the wire harness image to obtain the edge pixel points of the wire harness and determining the length of the wire harness includes:

[0023] Extract the edge information of the wire harness from the wire harness image based on the Canny edge detection algorithm, and determine the edge contour of the wire harness;

[0024] Randomly select N pixel points on the edge contour for random Hough transform;

[0025] By counting the peaks in the Hough space, the line with the most collinear points is taken as the entire wire harness L, including the straight part l 1 and the arc part l 2 ;

[0026] The straight part l 1 Calculate the length of the straight part through the straight line equation y 1 That is, the Euclidean distance between the two endpoints:

[0027]

[0028] where A(x a , y a ), B(x b , y b ) are the two endpoints of the straight line;

[0029] The arc part l 2 Calculate the length of the arc through the straight line equation y 2 :

[0030]

[0031] where c and d are the abscissas of the two endpoints of the arc;

[0032] Take the sum of the lengths of the straight part l 1 and the arc part l 2 as the length of the wire harness in the image;

[0033] Determine the actual length of the wire harness according to the length of the wire harness in the image and the preset proportionality coefficient.

[0034] In one possible implementation, determining the preset size range includes:

[0035] Select n groups of different sample wire harnesses, and count the error u i between the actual value A i and the measured value M i =A i -M i , and the average error Calculate the error threshold ε 0 , expressed as:

[0036]

[0037] Measure the same sample wire harness m times to obtain several measurement values M′ of the same wire harness i , determine the systematic error ε 1 , expressed as:

[0038]

[0039] According to the error threshold ε 0 , systematic error ε 1 and the preset template value to determine the preset size range. If the difference between the wire harness length and the preset template value is within the threshold range (ε 0 +ε 1 ), it is determined that the wire harness length meets the preset size range.

[0040] In one possible implementation, extract the first-order color moment and second-order color moment of each single wire harness image in the HSV color space, including:

[0041] Extract the pixel values of the single wire harness image on the three components of H, S, and V;

[0042] Based on the pixel values, determine the first-order color moment and second-order color moment corresponding to the single wire harness image; among them, the first-order color moment is expressed as:

[0043]

[0044] In the formula, N represents the total number of pixels of the single wire harness image, P i，j represents the jth pixel value of the ith color component, e i represents the mean value of all pixels of the ith color component, that is, the value of the first-order color moment;

[0045] The second-order color moment is expressed as:

[0046]

[0047] In the formula, S i represents the variance of all pixels of the ith color component, that is, the second-order color moment.

[0048] In one possible implementation, intercept the wire harness crimping area in the wire harness image and generate a binary image of the wire harness crimping area, including:

[0049] Intercept the wire harness crimping area based on the adaptive threshold segmentation algorithm;

[0050] For each pixel of the image within the wire harness crimping area, define a 3×3 neighborhood window centered on the current pixel, including the central pixel and its surrounding 8 adjacent pixels;

[0051] Calculate the mean of the grayscale values of all pixels within the 3×3 neighborhood window;

[0052] Take the mean as the threshold T(x, y) of the current pixel, expressed as:

[0053]

[0054] In the formula, T(x, y) is the threshold of the current pixel, W is the neighborhood window centered on the current pixel, I(i, j) is the grayscale value of the pixel within the window, and C is a constant used to adjust the sensitivity of segmentation;

[0055] Compare the grayscale value of the current pixel with the calculated threshold T(x, y); if the grayscale value of the current pixel is greater than the threshold T(x, y), set the current pixel to white; if the grayscale value of the current pixel is less than or equal to the threshold T(x, y), set the current pixel to black;

[0056] Repeat the above steps to process each pixel of the image within the wire harness crimping area, generating a binary image of the wire harness crimping area; wherein, the white area of the binary image represents the wire harness crimping area, and the black area represents the background or other non-crimping areas.

[0057] In one possible implementation, the pre-trained instance segmentation model for crimping area defects is the YOLACT instance segmentation model, with ResNet50 as the backbone network;

[0058] Training the instance segmentation model for crimping area defects includes:

[0059] Obtain multiple images containing various types of crimping defects to construct a crimping area defect dataset, where the types of crimping defects include crack defects, abnormal crimping shape defects, oxidation defects on the crimping surface, and exposed core wire defects in the crimping area;

[0060] Input the crimping area defect dataset into the configured YOLACT instance segmentation model, and extract feature information through the backbone network, expressed as:

[0061] F ∈ R C×H×W (9)

[0062] In the formula, F represents the feature map dimension, C represents the number of feature channels, H represents the height of the feature map, and W represents the width of the feature map;

[0063] Perform dimensionality reduction processing on the feature map dimension F, and use a 1×1 convolutional kernel to reduce the number of feature channels C of the feature map dimension F to 32 as the number of shared mask prototypes;

[0064] Determine the shared mask dimension P, expressed as:

[0065] P belongs to R M×H×W (10)

[0066] Normalize the original prediction value C of the YOLACT instance segmentation model through an activation function i to obtain the mask weight coefficient c im for characterizing the contribution degree of the shared mask prototype to a specific target instance, expressed as:

[0067] C i ={C i1 , C i2 ,..., C iM} (11)

[0068] c im =sigmoid(C i ) (12)

[0069] Perform a linear combination of the shared masks to generate the final instance mask That is, the trained crimping area defect instance segmentation model, expressed as:

[0070]

[0071] where M represents the number of shared masks, represents the final mask of the i-th instance. P m (x, y) represents the m-th shared mask prototype, C im represents the m-th weight of instance i;

[0072] Use the binary cross-entropy loss function to optimize the network for enhancing the ability of the crimping area defect instance segmentation model to distinguish different types of crimping defects, expressed as:

[0073]

[0074] In the formula, M i (x, y) is the true mask annotated by the Labelme software, is the predicted mask.

[0075] In one possible implementation, based on the defect mask, mark the crimping defects in the harness crimping area and determine the compactness of the crimping area, including:

[0076] Use opencv to extract the contours of the defect mask to obtain the point set of the crimping area image contour, expressed as:

[0077] contour={(x 1 , y 1 ), (x 2, y 2 ), ..., (x n , y n )}(15)

[0078] Based on the contour, calculate the distance d between adjacent points i , expressed as:

[0079]

[0080] In the formula, (x i , y i ) are the coordinates of the i-th point on the contour, n is the total number of contour points, x n+1 = x 1 , y n+1 = y 1 ;

[0081] Based on the distance between the adjacent points, determine the contour perimeter D, expressed as:

[0082]

[0083] Based on the calculation formula of the polygon area, determine the area A of the contact surface of the crimping area, expressed as:

[0084]

[0085] In the formula, (x i , y i ) are the coordinates of the i-th point on the contour, n is the total number of contour points, x n+1 = x 1 , y n+1 = y 1 ;

[0086] Determine the compactness of the contact surface of the crimping area through the perimeter D and the area A, expressed as:

[0087]

[0088] Compared with the prior art, the technical solution provided by this application has the following beneficial effects:

[0089] This application realizes the efficient and accurate detection of the quality of wire harnesses through the combination of image processing and deep learning. First, through preprocessing operations such as denoising, grayscaling, binarization, and edge extraction on the wire harness images, it can effectively remove image noise, clearly identify the connected regions and edge features of the wire harness, and provide an accurate basis for subsequent wire harness count and length measurement. Secondly, by using the first-order and second-order color moments in the HSV color space for feature extraction and comparing with the template wire harness values, it can quickly determine whether the color features of the wire harness meet the standards, improving the sensitivity and accuracy of detection. In addition, based on a pre-trained instance segmentation model for crimping area defects to detect the wire harness crimping area, it can accurately identify crimping defects and generate defect masks, and further verify the crimping quality by calculating the compactness of the contact surface. The entire solution not only realizes the comprehensive detection of the quality of wire harnesses, but also greatly improves the detection efficiency, reduces the errors and labor intensity of manual detection, has high practicability and reliability, and can be effectively applied to the quality control link in the wire harness production process to ensure the high quality and consistency of wire harness products. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 FIG. is a schematic structural diagram of a visual inspection system for the crimping quality of automotive wire harnesses provided in Embodiment 1 of this application.

[0091] Figure 2 FIG. is a flowchart of a method for detecting the order and size of the core wires of an automotive wire harness provided in Embodiment 2 of this application.

[0092] Figure 3 FIG. is a flowchart of a method for detecting the wire order of an automotive wire harness provided in Embodiment 2 of this application.

[0093] Figure 4 FIG. is a flowchart of a method for detecting crimping defects of a wire harness provided in Embodiment 2 of this application.

[0094] Figure 5 FIG. is a schematic diagram of the crimping area of the wire harness provided in Embodiment 2 of this application.

[0095] Figure 6 FIG. is a flowchart of a visual inspection method for the crimping quality of automotive wire harnesses provided in Embodiment 3 of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0096] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0097] Embodiment 1

[0098] Embodiment 1 of the present application provides a schematic structural diagram of a visual inspection system for the crimping quality of automotive wire harnesses. As Figure 1 shown, the overall structure of the visual inspection system for the crimping quality of automotive wire harnesses includes, from outside to inside in sequence, a frame body, a conveying module, a visual inspection module, a display and control module, and a mobile support module. The frame body is built with aluminum alloy profiles and serves as the main support framework of the entire device. It is lightweight and sturdy, providing an installation foundation and a stable operating environment for other modules. The conveying module runs through the entire device and is installed in the middle of the frame body. It is connected to the front and rear ends of the production line through a conveyor belt and is used to convey the automotive wire harnesses to be inspected from the starting end to the inspection area, and continue to convey them to the next process or the unqualified rejection area after the inspection is completed, realizing the full automation of the inspection process.

[0099] The visual inspection module is installed above the conveying module and includes an industrial camera, a high-brightness LED light source, and a mounting bracket. The industrial camera is fixed through the mounting bracket, and its inspection area is directly opposite the wire harness crimping point on the conveyor belt. The high-brightness LED light source is installed around the camera, providing uniform and high-intensity illumination for image acquisition to ensure clear and reliable image quality. By adjusting the height and angle of the visual module bracket, the inspection area of the industrial camera can completely cover the crimping point, thereby realizing the accurate inspection of the wire harness crimping quality.

[0100] The display and control module is installed on one side of the frame body and consists of a display and an industrial control computer. The industrial control computer is connected to the industrial camera of the visual inspection module through a data cable and is responsible for receiving image data and performing processing and analysis. The display is connected to the industrial control computer and is used to display the inspection results and the operation interface in real time, facilitating the operator to monitor the inspection process and results. The display and control module works in coordination with the conveying module and the visual inspection module through control signals to realize the automation and intelligence of the inspection process.

[0101] The mobile support module is installed at the bottom of the device and consists of casters, which are used to facilitate the movement of the device between different production lines. The setting of the casters makes the entire device have good flexibility and versatility, and can quickly adapt to different production environments and inspection requirements, improving the utilization rate of the equipment and the production efficiency.

[0102] In a specific application scenario, in the embodiments of the present application, the process parameters of the automotive wire harness crimping quality and the camera calibration parameters are first set. An industrial camera with five million pixels and a resolution of 2448*2048 is selected, along with a white shadowless LED high-brightness light source. The wire harness to be detected is placed on the battery fixing platform at the starting end of the conveyor belt, and the wire harness to be detected is fixed by the groove on the fixing platform. The conveyor belt runs automatically, transporting the wire harness to be detected under the industrial camera. With the assistance of the light source, the industrial camera takes pictures of the wire harness to be detected and transmits the captured images to the industrial computer. The industrial computer analyzes the images through two built-in algorithms to detect the quality parameters of the wire harness area, including the core wire sequence, size, and wire harness crimping defects, and then compares the obtained detection information with the process standard to obtain the quality detection result of the automotive wire harness.

[0103] Through the reasonable layout and collaborative work of the above modules, the visual inspection device for the crimping quality of automotive wire harnesses of the present invention realizes the full-process automated operation from wire harness conveying, image acquisition, defect detection to result display, meeting the requirements of high efficiency, precision, and intelligence for wire harness crimping quality detection in the modern automotive manufacturing industry.

[0104] The wire harness quality detection is divided into two parts: the first part detects the core wire sequence and size, and the second part detects the crimping defects. Below, the wire harness quality detection methods for the above two parts will be elaborated in detail in combination with the method embodiments.

[0105] Embodiment 2

[0106] See Figure 2 , which is a flowchart of a method for detecting the core wire sequence and size of an automotive wire harness provided in Embodiment 2 of the present application. As shown in Figure 2 , the specific implementation steps of the above method include:

[0107] Step 101, collect a complete wire harness image.

[0108] Step 102, preprocess the above wire harness image, including converting the above wire harness image into a grayscale image and performing preliminary denoising through median filtering.

[0109] Specifically, the above complete wire harness image obtained by the industrial camera is an RGB color image, including three color channels: red (R), green (G), and blue (B). In the embodiments of the present application, by converting the RGB image to the HSV color space and directly taking the value V component as the grayscale image, both the brightness information of the wire harness image is retained, and at the same time, the color information of the wire harness image is removed.

[0110] Furthermore, perform median filtering on the obtained grayscale image to remove the noise in the image and smooth the image.

[0111] Step 103: Perform binarization on the above grayscale image to segment the wire harness image into a black background area and a white wire harness connected area.

[0112] After converting the above wire harness image to the HSV color space, calculate the histograms of the three components of the image in hue H, saturation S, and brightness V respectively. The horizontal axis of the histogram represents the gray value of the pixel, and the vertical axis represents the total number of pixel points with this gray value.

[0113] Since the background area has a large proportion and the pixel values are close, a "peak" with a large peak value is formed in the low gray part of the histogram. The wire harness area accounts for a small proportion of the total image, and a "peak" with a low peak value is formed in the higher gray area of the histogram. A "valley" is formed between the high "peak" and the low "peak". Obtain the gray value corresponding to the lowest point of the "valley", and take the average of the lowest values obtained from the three-component histograms as the threshold, as shown in formula (1), T h 、T s 、T v are the gray values corresponding to the lowest points in the "valleys" of the histograms of the three components H, S, and V respectively.

[0114]

[0115] Perform binarization on the wire harness image according to the above threshold T, set the pixel gray value greater than the threshold T to 255, and set the pixel gray value less than the threshold T to 0, so as to segment the wire harness image into a black background area and a white wire harness connected area.

[0116] Step 104: Perform an erosion operation on the binarized image to remove redundant noise points and obtain the final wire harness image. The white connected domain of the final wire harness image consists only of wire harnesses, which facilitates counting the number of white connected domains and determining the number of wire harnesses.

[0117] Step 105: Determine whether the above number of wire harnesses meets the preset standard. If it meets, continue to execute Step 106. If it does not meet the preset standard, it is determined that the automotive wire harness product is unqualified.

[0118] Step 106: Perform edge extraction on the processed wire harness image using the Canny operator to obtain the edge contour of the wire harness.

[0119] Based on the Canny edge detection algorithm, this application embodiment extracts the edge information of the wire harness from the wire harness image, determines the edge contour of the wire harness, and provides a basis for further wire harness structure analysis.

[0120] Specifically, the Canny edge detection algorithm includes the following steps:

[0121] In the embodiment of this application, the Canny edge detection algorithm is used to extract the edge information of the wire harness from the wire harness image, so as to determine its edge contour, laying a foundation for subsequent wire harness structure analysis.

[0122] In practical applications, edge detection is easily interfered by image noise. Therefore, before edge detection, the image is first smoothed using a Gaussian filter to effectively reduce the influence of noise on the detection result. Subsequently, the gradient intensity and direction of each pixel point in the image are calculated through the Sobel operator. Among them, the gradient intensity reflects the significance of the edge, while the gradient direction indicates the direction of the edge.

[0123] To further refine the edge, a non-maximum suppression operation is performed, that is, only the pixel points with the maximum gradient intensity are retained, while those pixel points that are not local maxima are suppressed, thereby obtaining a clearer and narrower edge. On this basis, the Canny edge detection algorithm also performs threshold detection and edge connection. Two thresholds are used, including a low threshold and a high threshold. Pixel points with a gradient intensity higher than the high threshold are regarded as strong edges, while pixel points lower than the low threshold are suppressed. For pixel points between the two, only when they are connected to strong edge pixel points will they be recognized as part of the edge. Finally, through edge tracking technology, the discontinuous edges are connected to form a complete edge, thereby achieving the accurate extraction and description of the wire harness edge.

[0124] Step 107: Determine the wire harness length based on the edge pixel points of the edge contour.

[0125] Specifically, randomly select H pixel points on the edge contour of the wire harness to perform a random Hough transform. By statistically analyzing the peaks in the Hough space, the line with the most collinear points is found. The line with the most collinear points represents the entire wire harness L, including the straight part l 1 and the arc part l 2 . It should be noted that the above H is an adaptive threshold and needs to be adjusted according to the actual application scenario. When taking values, attention should be paid to balancing the recognition accuracy and calculation speed. If the value is too small, it may lead to large recognition errors. If the value is too large, it will affect the calculation speed.

[0126] The straight part l 1 Calculate the length of the straight line through the straight line equation y 1 , that is, the Euclidean distance between the two endpoints:

[0127]

[0128] Among them, A(x a , y a ), B(x b , y b ) are the two endpoints of the straight line.

[0129] Arc part l 2 Through the straight-line equation y 2 Calculate the length of the arc:

[0130]

[0131] Where c and d are the abscissas of the two endpoints of the arc.

[0132] Take the sum of the lengths of the straight-line part l 1 and the arc part l 2 as the length of the wire harness in the image, so that the actual length of the wire harness can be determined according to the preset proportionality coefficient.

[0133] Step 108: Determine whether the length of the above wire harness meets the preset size range. If so, it is determined that the product is qualified, and the wire harness wire sequence detection is continued. If the calculated length of the wire harness is not within the above preset size range, it is determined that the product is unqualified.

[0134] Specifically, since the Hough transform can only recognize curves that can be expressed by formulas, and the bending part of the actual wire harness is not a standard arc and cannot be expressed by a curve formula, there is an error between the measured value and the actual value. Before actual production, it is necessary to measure n groups of different sample wire harnesses first, and count the actual value A i and the measured value M i of each group of sample wire harnesses, and the error u i = A i - M i , as well as the average error Calculate the error threshold ε 0 , expressed as:

[0135]

[0136] Then measure the same sample wire harness m times to obtain several measured values M' i of the same wire harness, and the systematic error ε 1 calculated from the measured value, expressed as:

[0137]

[0138] After calculating the actual length of the wire harness through the ratio of the image to the actual value, if the difference between the wire harness length and the preset template value is within the threshold range (ε 0 + ε 1 ), it is determined that the wire harness length of the above wire harness meets the preset size range, and the wire harness size is determined to be qualified.

[0139] See Figure 3 , which is a flowchart of a method for detecting the wire sequence of an automotive wire harness provided by an embodiment of the present application. AsFigure 3 As shown in Figure 3 , the specific implementation steps of the above method include:

[0140] Step 201: Divide the complete wire harness image in the initial order from top to bottom by the method of region segmentation, and separately segment each wire harness according to the number of connected components to obtain multiple single wire harness images.

[0141] Step 202: Convert each single wire harness image from the RGB color space to the HSV color space, and extract the pixel values of the single wire harness image on the three components of H, S, and V.

[0142] Step 203: Based on the pixel values of each color component in the single wire harness image, determine the values of the first-order color moment and the second-order color moment corresponding to the single wire harness image, which are used to characterize the color characteristics and distribution characteristics of the single wire harness image.

[0143] Specifically, the value of the first-order color moment corresponding to the single wire harness image is calculated by formula (6) and is expressed as:

[0144]

[0145] In the formula, N represents the total number of pixels in the single wire harness image, P i，j represents the j-th pixel value of the i-th color component, and e i represents the mean value of all pixels of the i-th color component, that is, the value of the first-order color moment.

[0146] The value of the first-order color moment corresponding to the single wire harness image is calculated by formula (7) and is expressed as:

[0147]

[0148] In the formula, S i represents the variance of all pixels of the i-th color component, that is, the second-order color moment.

[0149] Step 204: Calculate the difference between the values of the first-order color moment and the second-order color moment corresponding to the single wire harness image and the template wire harness value at the corresponding position, and judge whether the above difference is within the preset threshold range. If so, it indicates that the wire order of the single wire harness is correct.

[0150] Step 205: Repeat steps 202 to 204, make the same judgment for each wire harness. If the wire orders of all wire harnesses are correct, then the wire order of the wire harnesses in this complete image is correct.

[0151] Precisely match the verified wire harnesses, including those with the correct number of wires and correct wire sequence, with the preset wire harness size standard. Subsequently, the system will display the detailed information of each wire harness that meets the standard, including its unique color and precise dimensions. This process not only ensures the accuracy and consistency of the wire harnesses, but also provides strong support for quality control and production monitoring through visualization, thereby significantly enhancing the automation and intelligence level in the wire harness manufacturing process and ensuring the quality and reliability of the final product.

[0152] After completing the detection of the wire harness size and wire sequence, it enters the wire harness crimping defect detection stage. The purpose of this stage is to ensure that no defects occur during the wire harness crimping process, such as loose crimping, crimping damage, etc. Next, the method for detecting wire harness crimping defects based on visual recognition provided by this application will be elaborated in detail in combination with specific embodiments.

[0153] See Figure 4 , which is a flowchart of a method for detecting wire harness crimping defects provided in Embodiment 2 of this application. As Figure 4 shown in, the specific implementation steps of the above method include:

[0154] Step 301: Set the process parameters of the crimping defect and the camera calibration parameters, and intercept the wire harness crimping area through the adaptive threshold segmentation algorithm.

[0155] Before detecting the wire harness crimping defects, it is necessary to set appropriate process parameters and camera calibration parameters. The process parameters include crimping height, width, crimping speed, crimping force, pressure, temperature, time required during the crimping process, etc., which need to be determined according to the material characteristics of the wire harness and the crimping requirements. The camera calibration parameters involve the internal parameters and external parameters of the camera. The internal parameters such as focal length, pixel size, etc., and the external parameters such as the distance and angle between the camera and the object to be photographed, accurately calibrate the above parameters to ensure the accuracy of subsequent image processing.

[0156] After completing the parameter setting, use the adaptive threshold segmentation algorithm to intercept the crimping area. Adaptive threshold segmentation can dynamically adjust the threshold according to the local characteristics of the image, thereby realizing the segmentation of the image and separating the crimping area from the background. See Figure 5 , which is a schematic diagram of the wire harness crimping area provided in Embodiment 2 of this application.

[0157] Step 302: Obtain the binary image of the intercepted wire harness crimping area.

[0158] In one achievable way, a 3×3 neighborhood window is defined with the current pixel as the center, which includes the central pixel and its surrounding 8 adjacent pixels. Calculate the mean value of the gray values of all pixels within the above-mentioned 3×3 neighborhood window. Take the above mean value as the threshold T(x, y) of the current pixel, as shown in Equation (8).

[0159]

[0160] In the formula, T(x, y) is the threshold of the current pixel, W is the neighborhood window with the current pixel as the center, l(i, j) is the gray value of the pixel within the window, and C is a constant used to adjust the sensitivity of segmentation.

[0161] Furthermore, compare the gray value of the current pixel with the calculated threshold T(x, y). If the gray value of the current pixel is greater than the threshold (i.e., I(x, y)>T(x, y)), then set the pixel to white. Otherwise, set it to black.

[0162] Repeat the above steps to process each pixel in the image of the wire harness crimping area, and finally generate a binary image of the wire harness crimping area. In the binary image, the white area represents the wire harness crimping area, while the black area represents the background or other non-crimped parts. Through this method, the wire harness crimping area can be effectively extracted from the original image, providing clear image data for subsequent defect detection and analysis. The adaptive threshold segmentation algorithm can dynamically adjust the threshold according to the local characteristics of the image, so as to better adapt to the changes of different lighting conditions and image content, and improve the accuracy and robustness of segmentation.

[0163] Step 303: Detect the binary image of the wire harness crimping area based on the pre-trained crimping area defect instance segmentation model to obtain a defect mask mask.

[0164] In the embodiments of the present application, several images containing various types of crimping defects are pre-obtained, including crack defect images, abnormal crimping shape defect images, crimping surface oxidation defect images, and exposed core wire defect images in the crimping area. All images are labeled by Labelme software.

[0165] Construct a crimping area defect data set based on the above images. Configure the YOLACT instance segmentation model, use ResNet50 as the backbone network, and set the number of masks to 32. Define the detected defect categories, including cracks, abnormal crimping shapes, crimping surface oxidation, and exposed core wires in the crimping area.

[0166] Input the above crimping area defect data set into the configured YOLACT instance segmentation model, and extract feature information through the backbone network, expressed as:

[0167] F∈R C×H×W(9)

[0168] In the formula, F represents the dimension of the feature map, C represents the number of feature channels, H represents the height of the feature map, and W represents the width of the feature map.

[0169] Furthermore, the dimension F of the feature map is reduced. A 1×1 convolutional kernel is used to reduce the number of feature channels C of the feature map dimension F to 32, which is used as the number of shared mask prototypes. The shared mask dimension P is calculated through formula (10) and is expressed as:

[0170] P ∈ R M×H×W (10)

[0171] The original predicted value C of the YOLACT instance segmentation model is normalized through the activation function to obtain the mask weight coefficient c i , and the mask weight coefficient is a set of scalar values, indicating the contribution degree of the shared mask prototype to a specific target instance. As shown in formulas (11) and (12), it is expressed as: im C

[0172] C i = {C i1 , C i2 ,..., C iM} (11)

[0173] c im = sigmoid(C i ) (12)

[0174] The shared mask is linearly combined through the above coefficients to generate the final instance mask As shown in formula (13), that is, the trained instance segmentation model for crimping area defects is obtained, which is expressed as:

[0175]

[0176] Among them, M represents the number of shared masks, represents the final mask of the i-th instance. P m (x, y) represents the m-th shared mask prototype, and C im represents the m-th weight of instance i.

[0177] Finally, the binary cross-entropy loss function is used to optimize the network. As shown in formula (14), the quality of the generated mask is improved, making it more accurately represent the shape and position of the object, improving the accuracy of object detection, and enhancing the discrimination ability for different types of targets including cracks, abnormal crimping shapes, oxidation of the crimping surface, and exposure of the core wire in the crimping area.

[0178]

[0179] Among them, Mi (x, y) is the true mask labeled by the Labelme software, and

[0180] is the predicted mask. The optimized instance segmentation model for crimping area defects is used to detect the binary image of the wire harness crimping area, obtaining a defect mask for marking the defective areas in the wire harness crimping area. Based on the above defect mask, the defect types corresponding to the defective areas in the wire harness crimping area are determined, including cracks, abnormal crimping shapes, oxidation on the crimping surface, exposure of core wires in the crimping area, etc. In the embodiments of the present application, the above recognition results are uploaded to an industrial computer, thereby realizing automated quality monitoring and defect management. This not only improves the detection efficiency and accuracy but also provides important data support for subsequent production process optimization and quality control, further enhancing the intelligent level and response ability of the production line.

[0181] Step 304: Use OpenCV to extract the contour of the above defect mask to determine the size information of the crimping area.

[0182] Specifically, the size information of the crimping area includes the perimeter, area, and compactness of the contact surface, and the specific calculation methods are as follows.

[0183] In the embodiments of the present application, contour information is extracted from the mask image of the wire harness crimping area based on OpenCV. The contour is the set of the edges of the wire harness in the image and can be represented as a series of points that define the boundary of the crimping area. Let the point set of the contour of the crimping area image be the contour, denoted as:

[0184] contour = {(x 1 , y 1 ), (x 2 , y 2 ),..., (x n , y n )} (15)

[0185] Based on the above contour, the distance d i between adjacent points is calculated, denoted as:

[0186]

[0187] In the formula, (x i , y i ) is the coordinate of the i-th point on the contour, n is the total number of contour points, x n+1 = x 1 , y n+1 = y 1 .

[0188] Based on the distance between adjacent points above, determine the contour perimeter D, expressed as:

[0189]

[0190] The area extraction method is as follows:

[0191] Based on the calculation formula of the polygon area, determine the area A of the contact surface of the crimping area, expressed as:

[0192]

[0193] where (x i , y i ) are the coordinates of the i-th point on the contour, n is the total number of contour points, x n+1 = x 1 , y n+1 = y 1 .

[0194] Determine the compactness of the contact surface of the crimping area based on the perimeter and area obtained through the above calculations, expressed as:

[0195]

[0196] Through the calculated perimeter, area, and compactness, the quality of the crimping area can be evaluated. For example, the compactness can be used as the qualification standard for the crimping contact area. The greater the compactness of the contact surface, the more irregular the shape of the contact surface represents.

[0197] In this embodiment, through contour extraction and geometric calculation, key dimension information of the crimping area is obtained from the image, and these information are used to evaluate the crimping quality, automatically detecting and analyzing the wire harness crimping area, so as to improve production efficiency and product quality.

[0198] Step 305: Refer to the process qualification standard. If no crimping defect is detected and the compactness of the crimping contact surface is qualified, the product is determined to be qualified.

[0199] Step 306: If a crimping defect is detected or the compactness of the crimping contact surface is unqualified, the product is determined to be unqualified.

[0200] In the actual application scenario, the automatic wire harness crimping detection system outputs the above visual detection results to the display and control module in real time. If the detection result shows that the wire harness crimping quality meets the preset process standard, the display will show the information of "meeting the process standard, crimping qualified". At the same time, this result will be automatically recorded and stored in the system database for subsequent quality tracking and analysis. Such an instant feedback mechanism not only improves production efficiency but also ensures the consistency of product quality.

[0201] On the contrary, if the detection result shows that there are defects in the wire harness crimping, such as cracks, abnormal crimping shapes, oxidation on the crimping surface, or exposure of the core wire in the crimping area, etc., the monitor will clearly indicate "There are defects in the crimping, and the crimping is unqualified", and will list the specific defect types in detail. This information will also be stored in the system database, providing key data for the quality control team so that they can quickly take corrective measures, optimize the production process, and reduce the generation of unqualified products.

[0202] Embodiment III

[0203] This Embodiment III is a comprehensive elaboration of all the technical solutions provided in the above Embodiment II. The method for detecting the core wire sequence, size, and crimping defects of the automotive wire harness based on the wire harness image is elaborated in detail in combination with specific embodiments.

[0204] See Figure 6 , which is a flowchart of a method for detecting the crimping quality of an automotive wire harness provided in Embodiment III of this application. As Figure 6 shown in, the specific implementation steps of the above method include:

[0205] Step 401: Perform grayscale conversion, denoising, and binarization processing on the above wire harness image to determine the white wire harness connected region of the wire harness image.

[0206] Step 402: Determine the number of wire harnesses based on the above white wire harness connected region.

[0207] Step 403: If the number of the above wire harnesses is consistent with the first preset standard value, perform edge extraction on the above wire harness image to obtain the edge pixel points of the wire harness, so as to determine the wire harness length.

[0208] Step 404: If the above wire harness length meets the preset size range, segment the wire harness image into multiple single wire harness images according to the preset initial order.

[0209] Step 405: Extract the first-order color moment and second-order color moment of each single wire harness image in the HSV color space, and compare them with the difference between the template wire harness values at the corresponding positions of the single wire harness.

[0210] Step 406: If the above differences are all within the preset threshold range, intercept the wire harness crimping area in the wire harness image and generate a binary image of the wire harness crimping area.

[0211] Step 407: Detect the above binary image based on the pre-trained defect instance segmentation model for the crimping area to generate a defect mask, which is used to mark the crimping defects in the wire harness crimping area and determine the compactness of the crimping area. The above compactness is determined by the perimeter and area of the contact surface of the crimping area.

[0212] Step 408: If no crimping defect is detected and the above-mentioned compactness is consistent with the second preset standard value, the quality inspection of the qualified wire harness is completed.

[0213] Step 409: If the number of wire harnesses is inconsistent with the first preset standard value, or if the above-mentioned wire harness length is not within the preset size range, or if the difference between the template wire harness values at the corresponding positions of any one wire harness is not within the preset threshold range, or if a crimping defect is detected, or if the above-mentioned compactness is inconsistent with the second preset standard value, the product is determined to be unqualified.

[0214] It should be noted that for the specific operation methods of each step in the embodiments of the present application, refer to the above-mentioned Embodiment 2, and this Embodiment 3 will not be elaborated.

[0215] Compared with the prior art, the technical solutions provided by the embodiments of the present application have the following beneficial effects:

[0216] The present application realizes the efficient and accurate detection of the quality of wire harnesses through the combination of image processing and deep learning. First, through preprocessing operations such as denoising, grayscale conversion, binarization, and edge extraction of the wire harness image, the image noise can be effectively removed, and the connected regions and edge features of the wire harness can be clearly identified, providing an accurate basis for subsequent wire harness number statistics and length measurement. Second, by using the first-order and second-order color moments in the HSV color space for feature extraction and comparing with the template wire harness values, the color characteristics of the wire harness can be quickly judged whether they meet the standards, improving the sensitivity and accuracy of detection. In addition, based on the pre-trained instance segmentation model for crimping area defects to detect the crimping area of the wire harness, the crimping defects can be accurately identified and defect masks can be generated, and at the same time, the crimping quality is further verified by calculating the compactness of the contact surface. The entire solution not only realizes the comprehensive detection of the quality of wire harnesses, but also greatly improves the detection efficiency, reduces the errors and labor intensity of manual detection, has high practicability and reliability, and can be effectively applied to the quality control link in the wire harness production process to ensure the high quality and consistency of wire harness products.

[0217] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principle and spirit of the present application. The scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A visual inspection method for automobile wiring harness crimping quality, characterized in that: Based on the wire harness image, the wire harness is sequentially inspected for core wire sequence and size, and for pressure notch defects, the method comprising: graying, denoising and binarizing the wire harness image to determine a white wire harness connected area of ​​the wire harness image; Determining the number of wire bundles based on the white wire bundle connected area; If the number of the wire harnesses is consistent with the first preset standard value, edge extraction is performed on the wire harness image to obtain edge pixel points of the wire harnesses for determining the length of the wire harnesses; If the length of the wire harness meets the preset size range, the wire harness image is divided into a plurality of single wire harness images according to a preset initial sequence; Extracting the first-order color moment and the second-order color moment of each single line bundle image in the HSV color space, and comparing them with the difference between the template line bundle values ​​at the corresponding position of the single line bundle; If the differences are all within a preset threshold range, intercepting a wire harness crimping area in the wire harness image, and generating a binary image of the wire harness crimping area; Detecting the binary image based on a pre-trained crimping area defect instance segmentation model to generate a defect mask for marking crimping defects in the crimping area of ​​the wire harness and determining the compactness of the crimping area, wherein the compactness is determined by the perimeter and area of ​​the contact surface of the crimping area; If no crimping defect is detected and the tightness is consistent with the second preset standard value, the quality inspection of the qualified wire harness is completed.

2. A visual inspection method for automobile wiring harness crimping quality according to claim 1, characterized in that: Graying, denoising and binarization processing are performed on the line beam image, including: Convert the line beam image to the HSV color space, and take the lightness V component as the grayscale image of the line beam image; Performing median filtering on the grayscale image to remove noise in the image and smooth the image; The denoised grayscale image is binarized to segment the wire harness image into a black background area and a white wire harness connected area.

3. A visual inspection method for automobile wiring harness crimping quality according to claim 1, characterized in that: Based on the white wire harness connected area, determining the number of wire harnesses includes: The binarized wire bundle image is corroded to remove redundant noise points to obtain a final wire bundle image; wherein the white wire bundle connected area of ​​the final wire bundle image is only composed of wire bundles, which is used to determine the number of wire bundles.

4. A visual inspection method for automobile wiring harness crimping quality according to claim 1, characterized in that: Extracting edges of the wire harness image, obtaining edge pixel points of the wire harness, and determining the length of the wire harness include: Based on the Canny edge detection algorithm, edge information of the wire harness is extracted from the wire harness image to determine the edge contour of the wire harness; Randomly select N pixel points on the edge contour to perform random Hough transform; By counting the peak values ​​in the Hough space, the line with the most collinear points is taken as the entire line bundle L, including the straight line part l1 and the arc part l2; The length of the straight line portion l1 is calculated by the straight line equation y1, that is, the Euclidean distance between the two endpoints: Among them, A(x a ,y a ), B(x b ,y b ) are the two endpoints of the straight line; The arc part l2 calculates the length of the arc using the straight line equation y2: Among them, c and d are the horizontal coordinates of the two end points of the arc; The sum of the lengths of the straight line portion l1 and the arc portion l2 is taken as the length of the line bundle in the image; The actual length of the wire harness is determined according to the length of the wire harness in the image and a preset proportionality factor.

5. A visual inspection method for automobile wiring harness crimping quality according to claim 4, characterized in that: Determining the preset size range includes: Select n groups of different sample harnesses and count the actual value A of each group of sample harnesses i and the measured value M i The error between i =A i -M i , and the mean error The error threshold ε0 is calculated and expressed as: The same sample line is measured m times, and several measurement values ​​M′ of the same sample line are obtained. i , determine the system error ε1, expressed as: The preset size range is determined according to the error threshold ε0, the system error ε1 and the preset template value. If the difference between the harness length and the preset template value is within the threshold range (ε0+ε1), it is determined that the harness length meets the preset size range.

6. A visual inspection method for automobile wiring harness crimping quality according to claim 1, characterized in that: Extracting the first-order color moment and the second-order color moment of each single wire harness image in the HSV color space, including: Extracting pixel values ​​of the single wire harness image on three components of H, S, and V; The first-order color moment and the second-order color moment corresponding to the single wire harness image are determined based on the pixel value; wherein the first-order color moment is expressed as: Where N represents the total number of pixels in a single wire harness image, P i，j represents the jth pixel value of the i-th color component, e i Represents the mean value of all pixels of the i-th color component, that is, the value of the first-order color moment; The second-order color moment is expressed as: In the formula, S i Represents the variance of all pixels of the i-th color component, that is, the second-order color moment.

7. A visual inspection method for automobile wiring harness crimping quality according to claim 1, characterized in that: Intercepting a wire harness crimping area in the wire harness image and generating a binary image of the wire harness crimping area, including: Intercepting the wire harness crimping area based on an adaptive threshold segmentation algorithm; For each pixel of the image within the harness crimping area, a 3×3 neighborhood window is defined with the current pixel as the center, including the center pixel and 8 neighboring pixels around it; Calculate the mean grayscale value of all pixels in the 3×3 neighborhood window; The mean is used as the threshold T(x, y) of the current pixel, expressed as: Where T(x, y) is the threshold of the current pixel, W is the neighborhood window centered on the current pixel, I(i, j) is the grayscale value of the pixel in the window, and C is a constant used to adjust the sensitivity of segmentation; Compare the grayscale value of the current pixel with the calculated threshold value T(x, y); if the grayscale value of the current pixel is greater than the threshold value T(x, y), set the current pixel to white; if the grayscale value of the current pixel is less than or equal to the threshold value T(x, y), set the current pixel to black; Repeat the above steps to process each pixel of the image in the wire harness crimping area to generate a binary image of the wire harness crimping area; wherein the white area of ​​the binary image represents the wire harness crimping area, and the black area represents the background or other non-crimping area.

8. A visual inspection method for automobile wiring harness crimping quality according to claim 1, characterized in that: The pre-trained crimping area defect instance segmentation model is a YOLACT instance segmentation model, with ResNet50 as the backbone network; Training the crimping area defect instance segmentation model includes: Acquire multiple images containing various types of crimping defects to construct a crimping area defect dataset, wherein the crimping defect types include crack defects, abnormal crimping shape defects, crimping surface oxidation defects, and core wire exposure defects in the crimping area; The crimping area defect dataset is input into the configured YOLACT instance segmentation model, and feature information is extracted through the backbone network, which is expressed as: F∈R C×H×W (9) In the formula, F represents the dimension of the feature map, C represents the number of feature channels, H represents the height of the feature map, and W represents the width of the feature map; Performing dimensionality reduction processing on the feature map dimension F, using a 1×1 convolution kernel to reduce the number of feature channels C of the feature map dimension F to 32 as the number of shared mask prototypes; Determine the shared mask dimension P, expressed as: P∈R M×H×W (10) The original prediction value C of the YOLACT instance segmentation model by the activation function i Normalize and get the mask weight coefficient c im , which is used to characterize the contribution of the shared mask prototype to a specific target instance, is expressed as: C i ={C i1 ,C i2 ,...,C iM }(11) c im =sigmoid(C i )(12) Linearly combine the shared masks to generate the final instance mask That is, the trained instance segmentation model of the crimping area defect is expressed as: Where M represents the number of shared masks. P represents the final mask of the i-th instance. m (x, y) represents the mth shared mask prototype, C im represents the mth weight of instance i; The binary cross entropy loss function is used to optimize the network to enhance the ability of the crimping area defect instance segmentation model to distinguish different crimping defect types, which is expressed as: Where M i (x, y) is the real mask annotated by Labelme software, is the prediction mask.

9. A visual inspection method for automobile wiring harness crimping quality according to claim 1, characterized in that: Based on the defect mask, mark the crimp defects in the crimp area of ​​the wire harness and determine the tightness of the crimp area, including: Opencv is used to extract the contour of the defect mask and obtain the point set of the image contour of the crimping area, which is expressed as: contour={(x1,y1),(x2,y2),...,(x n ,y n )} (15) Based on the contour, calculate the distance d between adjacent points i , expressed as: In the formula, (x i ,y i ) is the coordinate of the i-th point on the contour, n is the total number of contour points, x n+1 =x1,y n+1 =y1; Based on the distance between the adjacent points, the contour perimeter D is determined, which is expressed as: Based on the calculation formula of polygon area, the area A of the contact surface of the crimping area is determined, which is expressed as: In the formula, (x i ,y i ) is the coordinate of the i-th point on the contour, n is the total number of contour points, x n+1 =x1,y n+1 =y1; The tightness of the contact surface of the crimping area is determined by the perimeter D and the area A, which is expressed as:

Images