A color wiring harness quality detection method based on image recognition

Abstract

The present application discloses a method for detecting the quality of color wire harness wiring based on image recognition, which relates to the field of image technology. The method is based on the principle of image recognition, and utilizes a target detection model in combination with a semantic segmentation model to extract binary segmentation masks of connecting lines corresponding to various color labels. An initial color detection order is obtained through pixel-level detection. When it is determined that the initial color detection order does not match the predetermined color order, the color detection order is further adjusted and updated according to the position distribution of the binary segmentation masks of the connecting lines of each color label through pixel-level detection, so that the color detection order is consistent with the actual connecting line order, avoiding sequence detection errors caused by the cross-influence of different connecting line position distributions on image detection, thereby finally determining whether there is a wiring error. The method can quickly and accurately detect whether there is a wiring error and locate the connecting lines with wiring errors, with a high degree of automation and high detection efficiency.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a method for detecting the quality of color wire harness wiring based on image recognition. Background Art

[0002] Various electronic and electrical equipment need to transmit signals with other devices through wiring harnesses to complete different functions. The wiring harness connected to each electronic and electrical equipment is a collection of multiple connecting wires. Currently, the circuit industry generally uses colored wiring harnesses, that is, different colors are used to distinguish the connecting wires with different functions in the wiring harness.

[0003] As the functions of electronic and electrical equipment become more and more abundant, the colored wiring harnesses connected to electronic and electrical equipment contain more and more connecting wires, resulting in increasing wiring complexity. It is easy to cause wiring errors, which not only affects the normal operation of electronic and electrical equipment, but also poses a safety hazard.

[0004] Since different functional connecting wires in a colored wiring harness are distinguished by different colors, the existing practice is to manually visually determine whether the connecting wires of each color are connected correctly. However, this approach has low detection efficiency, and manual detection is prone to errors, resulting in unstable detection results. Summary of the Invention

[0005] In response to the above-mentioned problems and technical requirements, this application proposes a method for detecting the quality of color wire harness wiring based on image recognition. The technical solution of this application is as follows:

[0006] A color wire harness wiring quality detection method based on image recognition, the color wire harness wiring quality detection method comprising:

[0007] A color image of the device to be tested and its connected color wire harness is collected as an original image, where the color wire harness includes multiple connecting wires, any two connecting wires have different colors, and each connecting wire is connected to a different terminal on the device to be tested;

[0008] Using the target detection model to identify the terminal area in the original image, and extracting the local image of the original image in the terminal area as the circuit board image;

[0009] Use the semantic segmentation model to classify the circuit board image at the pixel level and determine the color label of each pixel. Generate a binary segmentation mask of the connection line corresponding to each color label based on the pixels of each color label.

[0010] Determine candidate endpoints of the connection lines of various color labels based on the connection line binary segmentation masks corresponding to the various color labels, and compare the arrangement order of the candidate endpoints of the connection lines of various color labels as the initial color detection order with the predetermined color order. The predetermined color order is the arrangement order of the colors of the connection lines when the color harness is correctly connected;

[0011] When the color detection sequence matches the predetermined color sequence, it is determined that the color wiring harness of the device to be detected is correct; when the color detection sequence does not match the predetermined color sequence, the color detection sequence is adjusted and updated according to the position distribution of the binary segmentation mask of the connecting line of each color label;

[0012] When the color detection order after adjustment and update is consistent with the predetermined color order, it is determined that the color wiring harness of the device to be detected is correctly connected; otherwise, it is determined that the color wiring harness of the device to be detected is incorrect, and it is determined that there is a wiring error in the connecting wire corresponding to the color label that does not match the predetermined color order in the adjusted and updated color detection order.

[0013] A further technical solution is that obtaining the initial color detection sequence includes:

[0014] The connecting line binary segmentation mask corresponding to each color label is subjected to skeleton refinement and pixel uniqueness processing to obtain the connecting line skeleton image of the color label. The connecting line skeleton image of the color label has only one foreground pixel in each column.

[0015] Determine the pixel coordinates (x, y) of the foreground pixels at both ends of the connecting line skeleton image for each color label in the pixel coordinate system, and select the foreground pixels with larger y coordinates at both ends of the connecting line skeleton image as candidate endpoints of the color label; wherein the pixel coordinate system uses the upper left corner pixel of the circuit board image as the coordinate origin, the rightward direction along the row direction is the x direction, and the downward direction along the column direction is the y direction;

[0016] The candidate endpoints of the connecting lines of each color label are sorted in ascending order of x-coordinates to obtain the initial color detection order.

[0017] A further technical solution is to perform pixel uniqueness processing on the connecting line binary segmentation mask corresponding to each color label, including:

[0018] Each column of pixels in the binary segmentation mask of the connecting line after skeleton thinning is traversed in turn. For any column of pixels traversed, the foreground pixel with the largest y coordinate is retained, and all other pixels in the current column are set as background pixels. After traversing all columns, the pixel uniqueness processing is completed and the connecting line skeleton image with color label is obtained.

[0019] A further technical solution is to adjust and update the color detection order according to the position distribution of the binary segmentation mask of the connecting line of each color label, including:

[0020] Traverse each group of color labels in sequence according to the initial color detection order, where each group of color labels traversed includes a first color label and a second color label that are adjacent in the color detection order, and the x-coordinate of the candidate endpoint of the connecting line of the first color label is smaller than the candidate endpoint of the connecting line of the second color label;

[0021] For each group of color labels traversed, check whether the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label have completely staggered position distributions;

[0022] When it is determined that the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label do not have a completely staggered position distribution, a line intersection detection is performed on the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label, and when it is detected that the order of the first color label and the second color label is wrong due to the line intersection, the order of the first color label and the second color label in the color detection order is swapped; otherwise, the order of the first color label and the second color label in the color detection order remains unchanged;

[0023] When it is determined that the position distribution of the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label are completely staggered, the average value of the x-coordinates of all the first color foreground pixels and the average value of the x-coordinates of all the second color foreground pixels are calculated. When the average value of the x-coordinates of all the first color foreground pixels is less than or equal to the average value of the x-coordinates of all the second color foreground pixels, the order of the first color label and the second color label in the color detection order is maintained unchanged; otherwise, the order of the first color label and the second color label in the color detection order is swapped; wherein the first color foreground pixel is a foreground pixel in the connecting line binary segmentation mask of the first color label, and the second color foreground pixel is a foreground pixel in the connecting line binary segmentation mask of the second color label.

[0024] A further technical solution is that detecting whether the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label have completely staggered position distributions includes:

[0025] Detecting whether the number of overlapping distribution rows of the first color label and the second color label reaches a row number threshold, where the overlapping distribution row is a pixel row including both the first color foreground pixel and the second color foreground pixel in the same row;

[0026] When the overlapping distribution rows reach the row number threshold, it is determined that the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label do not have a completely staggered position distribution; otherwise, it is determined that the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label have a completely staggered position distribution.

[0027] A further technical solution is that performing cross detection on the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label includes:

[0028] Sequentially traverse each overlapping distribution row of the first color label and the second color label, and for each overlapping distribution row traversed, count the number of abnormal pixels in the overlapping distribution row, where the abnormal pixel in each overlapping distribution row is a foreground pixel of the first color whose x coordinate is greater than the x coordinate of the foreground pixel of the second color in the overlapping distribution row;

[0029] When the number of abnormal pixels in all overlapping distribution lines is not greater than the pixel number threshold, it is determined that there is no line intersection between the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label, and the order of the first color label and the second color label in the color detection order remains unchanged;

[0030] When the number of abnormal pixels in the overlapping distribution row exceeds the pixel number threshold, and when the average value of the x-coordinates of all the first color foreground pixels is less than or equal to the average value of the x-coordinates of all the second color foreground pixels, it is determined that the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label cross but do not cause an order error, and the order of the first color label and the second color label in the color detection order remains unchanged;

[0031] When the number of abnormal pixels in the overlapping distribution rows exceeds the pixel number threshold, and when the calculated average value of the x-coordinates of all first color foreground pixels is greater than the average value of the x-coordinates of all second color foreground pixels, it is determined that there is a line intersection between the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label, resulting in an order error, and the order of the first color label and the second color label in the color detection order is exchanged.

[0032] A further technical solution is to generate a connecting line binary segmentation mask corresponding to each color label based on the pixels of each color label, including:

[0033] Extract the pixels of each color label and convert them into grayscale images to obtain the original segmentation mask area of ​​the color label;

[0034] The original segmentation mask area of ​​each color label is binarized and the interference area with an area smaller than the area threshold is deleted to obtain the connecting line binary segmentation mask corresponding to the color label.

[0035] Its further technical solution is that the target detection model adopts the lightweight PICO series target detection model, which includes a backbone network, a neck network and a head network cascaded in sequence. The backbone network uses LCNet for feature extraction and outputs feature maps of three different scales; the neck network uses LCPAN to fuse the feature maps of three different scales. The output channel number of LCPAN is 128 and it uses depthwise separable convolution; the head network uses PicoHeadV2 for target detection. PicoHeadV2 includes convolutional layers for feature extraction and fully connected layers for category prediction and bounding box regression.

[0036] Its further technical solution is that the semantic segmentation model uses the semantic segmentation model of the PPLite series. The semantic segmentation model uses STDC1 as the backbone network, and three increasing output channels are designed between the backbone network and the semantic segmentation head to gradually extract and integrate image features. The semantic segmentation head sets three intermediate layers to refine the feature representation.

[0037] A further technical solution is that identifying the terminal area in the original image using the target detection model also includes:

[0038] Downsample the original image;

[0039] For the original image that has completed the downsampling process, linear interpolation is first performed on two adjacent pixels in the horizontal direction, and then linear interpolation is performed again in the vertical direction to obtain the original image that has completed the linear interpolation process;

[0040] The original image after linear interpolation is smoothed using median filtering to obtain the preprocessed original image;

[0041] The object detection model is used to identify the terminal block area in the preprocessed raw image.

[0042] The beneficial technical effects of this application are:

[0043] The present application discloses a method for detecting the quality of color wiring harness connections based on image recognition. The method is based on an image recognition method, utilizes a target detection model in combination with a semantic segmentation model to extract binary segmentation masks of connection lines corresponding to various color labels, obtains an initial color detection order through pixel-level detection, and then further adjusts and updates the color detection order according to the position distribution of the binary segmentation masks of the connection lines of each color label through pixel-level detection, so that the color detection order is consistent with the actual connection line order, avoiding sequence detection errors caused by the cross-influence of different connection line position distributions on image detection, and finally determining whether there is a wiring error. The method can quickly and accurately detect whether there is a wiring error and locate the connection line with the wiring error, with a high degree of automation and high detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a method for detecting the quality of color wire harness wiring according to an embodiment of the present application.

[0045] Figure 2 This is an image of a circuit board in action.

[0046] Figure 3 is based on Figure 2 Schematic diagram of the original segmentation mask area of ​​each color label extracted from the circuit board image and the determined candidate endpoints of the connection line.

[0047] Figure 4 is Figure 3 Schematic diagram of the skeleton refinement process of the binary segmentation mask of the connecting lines corresponding to each color label in the example.

[0048] Figure 5 This is a schematic diagram of a circuit board image in another example, the extracted connection line binary segmentation mask areas of each color label, and the determined candidate endpoints of the connection line.

[0049] Figure 6 This is a flow chart of a method for adjusting and updating the color detection sequence in another embodiment of the present application.

[0050] Figure 7 It is a schematic diagram of the distribution of different positions of the binary segmentation mask area of ​​the connecting lines of each color label in an example. DETAILED DESCRIPTION

[0051] The specific implementation of this application will be further described below with reference to the accompanying drawings.

[0052] This application discloses a color wiring harness quality detection method based on image recognition, please refer to Figure 1 As shown in the flowchart, the color wiring harness wiring quality detection method includes the following steps:

[0053] Step 110 : Acquire a color image of the device to be inspected and the color wire harness connected thereto as an original image.

[0054] The colored wire harness connected to the device to be tested includes multiple connecting wires, any two connecting wires have different colors, and each connecting wire is connected to a different terminal on the device to be tested.

[0055] Step 120 : Using the target detection model to identify the wiring terminal area in the original image, and extracting a local image of the original image in the wiring terminal area as a circuit board image.

[0056] Since the device to be inspected often includes other background areas in addition to the terminal area, or the device to be inspected itself is located in a complex background, the original image directly obtained often also includes a lot of background images that can easily cause interference. Therefore, it is necessary to identify and locate the circuit board image of the terminal area from the original image to reduce background interference.

[0057] In order to improve the accuracy of target detection, in one embodiment, the target detection model is not directly used to identify the terminal area in the original image. Instead, the original image is first preprocessed to obtain a preprocessed original image, and then the target detection model is used to identify the terminal area in the preprocessed original image. The preprocessing of the original image includes:

[0058] (1) Downsample the original image.

[0059] (2) For the original image that has completed the downsampling process, linear interpolation is first performed on two adjacent pixels in the horizontal direction to generate a new pixel value. Then, linear interpolation is performed again on the two pixel values ​​obtained in the previous step in the vertical direction to obtain the original image that has completed the linear interpolation. In this way, the information of the four adjacent pixels can be comprehensively considered to generate new pixels, thereby achieving image reduction while maintaining image quality.

[0060] (3) The original image after linear interpolation is smoothed using median filtering to reduce the nonlinear noise in the image and finally obtain the preprocessed original image.

[0061] In one embodiment, the target detection model adopts a lightweight PICO series target detection model, which includes a backbone network, a neck network, and a head network cascaded in sequence. The backbone network adopts LCNet for feature extraction and outputs feature maps of three different scales; the neck network adopts LCPAN to fuse feature maps of three different scales. The output channel number of LCPAN is 128 and it adopts depthwise separable convolution; the head network adopts PicoHeadV2 for target detection. PicoHeadV2 includes a convolutional layer for feature extraction and a fully connected layer for category prediction and bounding box regression.

[0062] When training the object detection model, pretrained weights are loaded to initialize the model parameters. Then, an object detection dataset is created using the COCO dataset format for network training. The training cycle is set to 500 epochs, with a model snapshot saved every 10 epochs. During training, appropriate training, evaluation, and test batch sizes are set to prevent insufficient memory. A base learning rate is set and adjusted using a cosine decay and linear warmup strategy. The model parameters are optimized using the Momentum optimizer and L2 regularization is applied. During training, input images are preprocessed, including decoding, random cropping, flipping, color distortion, normalization, and permutation transformations. The exponential moving average (EMA) method is used to update model parameters, allowing for unused parameters during model training. GPU hardware acceleration is used to leverage its parallel computing capabilities to improve the training process of deep learning models.

[0063] In a schematic diagram of an example, taking the circuit board as an example, the terminal area obtained by performing target detection and recognition on the original image is as follows: Figure 2 As shown in this Figure 2 In the example, the colored wiring harness connected to the device to be tested includes 5 connecting wires of different colors, namely Figure 2 In the figure, there are yellow connecting line L1, green connecting line L2, blue connecting line L3, black connecting line L4 and red connecting line L5.

[0064] Step 130: Use a semantic segmentation model to perform pixel-level classification on the circuit board image and determine the color label of each pixel.

[0065] In one embodiment, the semantic segmentation model uses the semantic segmentation model of the PPLite series. The semantic segmentation model uses STDC1 as the backbone network, and three increasing output channels are designed between the backbone network and the semantic segmentation head to gradually extract and integrate image features. The semantic segmentation head sets three intermediate layers to refine the feature representation and improve the accuracy of segmentation. When training the semantic segmentation model, in order to fully utilize the advantages of pre-training, the official pre-training weights are used to initialize the model parameters, and then the segmentation dataset is prepared. The labelme annotation tool is used to perform semantic segmentation annotation on the colored line area in the sample image in the segmentation dataset, and each color of the connecting line area contained in the colored line is marked as a corresponding color label. After the annotation is completed, the segmentation dataset is divided into a training set and a validation set, and stored in a specified directory and indexed by the training data path.

[0066] During training and validation, each sample image undergoes a series of preprocessing steps. First, the sample image is resized by varying the scaling factor from 1.0 to 2.5 in steps of 0.25 to simulate different views and resolutions. Next, random padding crops are performed with a size of [512, 256] to reduce edge artifacts and enhance the model's generalization. Furthermore, random horizontal flips are applied to improve the model's invariance to sample image flipping. To enhance the model's adaptability to varying lighting and color conditions, random color distortion is applied to adjust brightness, contrast, and saturation. Finally, the image is normalized to standardize pixel values.

[0067] During the initial training phase, the batch size was set to 8 and the total number of iterations was set to 20,000. SGD was selected as the optimizer, with a momentum of 0.9 and a weight decay parameter of 6e-4 to accelerate training and prevent overfitting. An initial learning rate of 0.002, a decay exponent of 0.9, 1000 warm-up iterations, and a warm-up starting learning rate of 1e-5 were set to optimize the training process. The loss function used was the online hard example mining cross entropy loss (OhmCrossEntropyLoss). By setting the minimum number of retained samples to 130,000, the focus was placed on difficult-to-classify examples and balancing the contributions of different loss functions.

[0068] Step 140, based on the pixels of each color label, generates a binary segmentation mask for the connecting lines corresponding to the color label, including: first extracting the pixels of each color label and converting them into a grayscale image to obtain the original segmentation mask area of ​​the color label, each color label is represented by a fixed grayscale value to distinguish connecting lines of different colors, such as Figure 2 The original segmentation mask areas of various color labels extracted from the circuit board image are as follows Figure 3As shown in the figure, the original segmentation mask area S1 corresponds to the yellow label, the original segmentation mask area S2 corresponds to the green label, the original segmentation mask area S3 corresponds to the blue label, the original segmentation mask area S4 corresponds to the black label, and the original segmentation mask area S5 corresponds to the red label. Then, the original segmentation mask area of ​​each color label is binarized and the interference area with an area smaller than the area threshold is deleted to obtain the connecting line binary segmentation mask corresponding to the color label. The area threshold is generally set to 200.

[0069] Step 150 : determining candidate endpoints of connection lines of various color labels according to the connection line binary segmentation masks corresponding to the various color labels, and comparing the arrangement order of the candidate endpoints of connection lines of various color labels as the initial color detection order with the predetermined color order.

[0070] For the convenience of description, a pixel coordinate system is defined. The pixel coordinate system takes the upper left corner pixel of the circuit board image as the coordinate origin, the rightward direction along the row direction is the positive x direction, and the downward direction along the column direction is the positive y direction. Then, the pixel coordinates (x, y) of each pixel in the pixel coordinate system can be determined.

[0071] Then, the binary segmentation mask of the connecting line corresponding to each color label is first subjected to skeleton refinement processing. The skeleton refinement processing can adopt the existing method, which will not be described in detail in this application. Figure 3 In the example, the binary segmentation mask of the connection line corresponding to each color label is as follows after the skeleton thinning process Figure 4 shown.

[0072] Then, the connecting line binary segmentation mask after the skeleton thinning process is further pixel-uniquely processed to obtain a connecting line skeleton image with a color label. After the pixel-uniquely processed, the connecting line skeleton image with a color label has only one foreground pixel in each column. In one embodiment, the pixel-uniquely processed includes: sequentially traversing each column of pixels of the connecting line binary segmentation mask after the skeleton thinning process, for any column of pixels traversed, retaining the foreground pixel with the largest y coordinate, that is, retaining the pixel at the bottom of the current column, and setting all other pixels in the current column as background pixels. After traversing all columns, the pixel-uniquely processed is completed and the connecting line skeleton image with a color label is obtained.

[0073] After obtaining the skeleton images of the connecting lines of each color label, the pixel coordinates (x, y) of the foreground pixels at the two end points of the skeleton images of the connecting lines of each color label are determined in the pixel coordinate system, and the foreground pixels with larger y coordinates at the two end points are used as candidate endpoints of the connecting lines of the color label. Figure 3In the example, the connection line candidate endpoint T1 with a yellow label, the connection line candidate endpoint T2 with a green label, the connection line candidate endpoint T3 with a blue label, the connection line candidate endpoint T4 with a black label, and the connection line candidate endpoint T5 with a red label can be extracted.

[0074] Ideally, the candidate endpoints of the connection lines of each color label to be located in this step should be the terminal locations of the connection lines of the corresponding colors. In actual applications, the terminal area in the circuit board image is located below the colored wire harness area, such as Figure 2 As shown, the terminal position of each connecting line is located closer to the bottom of the image, that is, at a position with a larger y coordinate. Therefore, when performing pixel uniqueness processing, this step retains the foreground pixel with the largest y coordinate in each column and the foreground pixel with a larger y coordinate selected here as the candidate endpoint of the connecting line. If the actual terminal area and the colored harness area adopt other relative positions, the screening method can also be adjusted accordingly.

[0075] Then, the candidate endpoints of the connecting lines of each color label are sorted in ascending order of x coordinates to obtain the initial color detection order, for example, Figure 3 In this example, the five color-labeled connecting line candidate endpoints T1-T5 are arranged in ascending x-coordinate order as T1, T2, T3, T4, and T5. The resulting initial color detection order is yellow, green, blue, black, and red from left to right along the image. This initial color detection order is then compared with a predetermined color order. The predetermined color order is the color order of each connecting line when the color wiring harness is correctly connected. Typically, the terminal blocks connected to each connecting line are arranged in order from left to right as the predetermined color order. This predetermined color order is known in advance.

[0076] Step 160: If the initial color detection sequence matches the predetermined color sequence, the color wiring harness of the device to be detected is determined to be correctly connected. If the color detection sequence does not match the predetermined color sequence, the color detection sequence is adjusted and updated based on the position distribution of the binary segmentation mask of the connecting line of each color label.

[0077] That is, when it is determined that the initial color detection order does not match the predetermined color order, it is not directly determined that there is a wiring error, because in actual applications, the distribution of the terminal blocks of the device to be detected is not fixed, the connection of the color harness is also relatively complex, and there are often crossovers and interferences between different connecting wires, which will cause the initial color detection order determined above to be not the actual arrangement order of the connecting wires. Ideally, the candidate endpoints of the connecting wires of each color label are the terminal positions of the connecting wires of the corresponding color, and the color detection order obtained by sorting the candidate endpoints of the connecting wires is the actual arrangement order of the connecting wires. However, this ideal situation will only occur when the connecting wires are not severely obstructed, such as in Figure 2 In the example of the colored wiring harness, the five connecting wires in the harness are not seriously blocked from each other, and the terminal blocks at the end of each connecting wire are exposed. Figure 3 The candidate ends of the connection wires with various color labels extracted are indeed the terminal positions of the connection wires with corresponding colors.

[0078] However, in most cases of actual applications, due to the occlusion of other components or the occlusion of connecting lines, it will interfere with target recognition and semantic segmentation, resulting in the binary segmentation mask of the connecting line corresponding to the extracted color label not being a complete connecting line, and thus resulting in the extracted connecting line candidate endpoints of each color label not necessarily being the terminal position of the connecting line of the corresponding color. Figure 5 In the example of Figure 5 As shown in (a), the colored wire harness connected to the device to be detected also includes yellow connecting wire L1, green connecting wire L2, blue connecting wire L3, black connecting wire L4 and red connecting wire L5. The binary segmentation mask of the connecting wire corresponding to each color label is as follows: Figure 5 As shown in (b), it should be noted that for the convenience of illustration, Figure 5 (b) in the figure uses different colors to distinguish the connection line binary segmentation masks of different color labels, including the connection line binary segmentation mask S1 of the yellow label, the connection line binary segmentation mask S2 of the green label, the connection line binary segmentation mask S3 of the blue label, the connection line binary segmentation mask S4 of the black label, and the connection line binary segmentation mask S5 of the red label. However, since the blue connection line L3 in this example seriously blocks the green connection line L2, it can be seen that the extracted connection line binary segmentation mask S2 of the green label is not the complete green connection line area. Further, the connection line candidate endpoint T1 of the yellow label, the connection line candidate endpoint T2 of the green label, the connection line candidate endpoint T3 of the blue label, the connection line candidate endpoint T4 of the black label, and the connection line candidate endpoint T5 of the red label can be extracted. Comparison Figure 5As can be seen from (a) and (b) in the figure, T1, T3, T4, and T5 are still the actual terminal positions of the corresponding connecting lines. However, the green-labeled connecting line candidate endpoint T2 extracted at this time is not the actual terminal position of the green connecting line L2, but only the endpoint position of the outer part of the green connecting line. Therefore, the five color-labeled connecting line candidate endpoints T1 to T5 are arranged in the order of x-coordinate from small to large as T2, T1, T3, T4, and T5. The initial color detection order is green, yellow, blue, black, and red from left to right along the image. Figure 5 As can be seen from (a) in FIG, the initial color detection order obtained at this time is not the actual arrangement order of the connecting lines. Therefore, the color detection order must first be adjusted and updated to restore the actual arrangement order of the connecting lines.

[0079] Adjusting and updating the initial color detection order according to the position distribution of the binary segmentation mask of the connecting line of each color label includes the following steps, please refer to Figure 6 Flowchart:

[0080] Step 601: traverse each group of color labels in sequence according to the initial color detection order.

[0081] Each group of color labels traversed includes the first color label and the second color label that are adjacent in the color detection order, and the x coordinate of the connecting line candidate endpoint of the first color label is smaller than the connecting line candidate endpoint of the second color label, that is, the connecting line candidate endpoint of the first color label is located to the left of the connecting line candidate endpoint of the second color label. For example, Figure 5 In the example, the initial color detection order is green, yellow, blue, black, and red. The color labels traversed for the first time include the green label as the first color label and the yellow label as the second color label, and the same is true for the traversal in sequence.

[0082] Step 602: For each group of color labels traversed, check whether the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label have completely staggered position distribution. The purpose of adjusting and updating the initial color detection order is to restore the actual connection line arrangement order. As mentioned above, the inconsistency between the initial color detection order and the actual connection line arrangement order is often due to the misidentification of the order caused by the interlacing of the connecting lines. When the position distribution of the binary segmentation mask of the connecting line of the two color labels is completely staggered, there is no line sequence intersection in the position of the binary segmentation mask of the connecting line of the two color labels. Therefore, first check whether the position distribution of the binary segmentation mask of the connecting line of the two color labels is completely staggered, including:

[0083] Check whether the number of overlapping distribution rows of the first color label and the second color label reaches a row number threshold. An overlapping distribution row is a pixel row that includes both the first color foreground pixel and the second color foreground pixel in the same row. The first color foreground pixel is the foreground pixel in the binary segmentation mask for the connected lines of the first color label, and the second color foreground pixel is the foreground pixel in the binary segmentation mask for the connected lines of the second color label.

[0084] When the number of overlapping rows reaches a threshold, the binary segmentation mask for the connecting lines of the first color label and the second color label is determined to be consistent with each other. Otherwise, the binary segmentation mask for the connecting lines of the first color label and the second color label is determined to be consistent with each other. The row threshold here can be customized, for example, to 8 rows or other empirical values.

[0085] In an example, the binary segmentation mask of the connection lines of each color label is extracted as follows: Figure 7 As shown, Figure 7 For ease of illustration, different colors are used to represent the binary segmentation masks of the connecting lines of each color label. The overlapping distribution rows of the connecting line binary segmentation mask of the yellow label and the connecting line binary segmentation mask of the green label include all pixel rows between the two dotted lines. In this case, the position distribution of the connecting line binary segmentation mask of the yellow label and the connecting line binary segmentation mask of the green label do not completely stagger, and there may be a situation where the lines cross. However, there are no overlapping distribution rows between the connecting line binary segmentation mask of the green label and the connecting line binary segmentation mask of the red label. Therefore, there is a situation where the position distribution is completely staggered, and the lines often do not cross.

[0086] Step 603: When it is determined that the binary segmentation mask for the connecting lines of the first color label and the binary segmentation mask for the connecting lines of the second color label are not completely misaligned in position distribution, a line intersection detection is performed on the binary segmentation mask for the connecting lines of the first color label and the second color label. If it is detected that the order of the first color label and the second color label is incorrect due to line intersection, the order of the first color label and the second color label in the color detection sequence is swapped; otherwise, the order of the first color label and the second color label in the color detection sequence remains unchanged. This includes:

[0087] (1) Traverse the overlapping distribution rows of the first color label and the second color label in sequence, and for each overlapping distribution row traversed, count the number of abnormal pixels in the overlapping distribution row.

[0088] The abnormal pixel in each overlapping distribution row is a foreground pixel of the first color whose x-coordinate is greater than the x-coordinate of the foreground pixel of the second color in the overlapping distribution row. Since the candidate endpoint of the connecting line labeled with the first color is located to the left of the candidate endpoint of the connecting line labeled with the second color, when there is no line crossing, the x-coordinate of the foreground pixel of the first color should be smaller than the x-coordinate of the foreground pixel of the second color. When the x-coordinate of the foreground pixel of the first color is greater than the x-coordinate of the foreground pixel of the second color, it indicates that a crossing may occur and the foreground pixel of the first color is considered an abnormal pixel.

[0089] (2) When the number of abnormal pixels in an overlapping distribution row is not greater than the pixel number threshold, it can be considered that the first color foreground pixels in the current overlapping distribution row are all located to the left of the second color foreground pixels within the error range. Further, when the number of abnormal pixels in all overlapping distribution rows is not greater than the pixel number threshold, it can be determined that there is no line crossing between the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label, and the order of the first color label and the second color label in the color detection sequence remains unchanged. The pixel number threshold here can be customized, such as set to 5 pixels or 10 pixels, to reduce the detection error caused by interfering pixels.

[0090] (3) When the number of abnormal pixels in the overlapping distribution row exceeds the pixel number threshold, the average value of the x coordinates of all first color foreground pixels and the average value of the x coordinates of all second color foreground pixels are further calculated.

[0091] (4) When the number of abnormal pixels in the overlapping distribution row exceeds the pixel number threshold, and when the average value of the x-coordinates of all the first color foreground pixels is less than or equal to the average value of the x-coordinates of all the second color foreground pixels, it is determined that there is a line intersection between the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label but no sequence error is caused, and the order of the first color label and the second color label in the color detection order remains unchanged.

[0092] (5) When the number of abnormal pixels in the overlapping distribution row exceeds the pixel number threshold, and when the average value of the x-coordinates of all the first color foreground pixels is calculated to be greater than the average value of the x-coordinates of all the second color foreground pixels, it is determined that the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label have line intersections and cause an order error, and the order of the first color label and the second color label in the color detection order is exchanged.

[0093] In step 604, if it is determined that the binary segmentation mask for the connecting lines of the first color label and the binary segmentation mask for the connecting lines of the second color label are completely misaligned in their distribution, since there is no intersection between the two lines, the average x-coordinates of all first color foreground pixels and the average x-coordinates of all second color foreground pixels are directly calculated. If the average x-coordinates of all first color foreground pixels is less than or equal to the average x-coordinates of all second color foreground pixels, the order of the first color label and the second color label in the color detection order is maintained unchanged; otherwise, the order of the first color label and the second color label in the color detection order is swapped.

[0094] As described above, each group of color labels is traversed in turn and it is detected whether the order needs to be adjusted. After all groups of color labels are traversed, the adjusted and updated color detection order is obtained.

[0095] In step 170, the color detection order after the aforementioned adjustment and update is the actual arrangement order of the connecting wires, which is then compared with the predetermined color order. If the adjusted and updated color detection order matches the predetermined color order, the color wiring harness of the device to be detected is determined to be correctly wired. Otherwise, the color wiring harness of the device to be detected is determined to be incorrectly wired. Furthermore, it can be determined that the connecting wires corresponding to the color labels in the adjusted and updated color detection order that do not match the predetermined color order have wiring errors, thereby locating the incorrectly wired connecting wires.

[0096] The above description is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and variations directly derived or imagined by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the scope of protection of the present application.

Claims

1. A color wire harness wiring quality detection method based on image recognition, characterized in that: The color wire harness connection quality detection method comprises: Collecting a color image of the device to be detected and the color wiring harness connected thereto as an original image, wherein the color wiring harness includes a plurality of connecting wires, any two connecting wires have different colors, and each connecting wire is connected to a different terminal on the device to be detected; Identifying a terminal block region in the original image using a target detection model, and extracting a partial image of the original image in the terminal block region as a circuit board image; Using a semantic segmentation model to perform pixel-level classification on the circuit board image and determine the color label of each pixel, and generating a binary segmentation mask of the connection line corresponding to each color label based on the pixels of each color label; Determining candidate endpoints of connection lines of various color labels based on the connection line binary segmentation masks corresponding to the various color labels, and comparing the arrangement order of the candidate endpoints of the connection lines of various color labels as an initial color detection order with a predetermined color order, wherein the predetermined color order is the arrangement order of the colors of the connection lines when the color wiring harness is correctly connected; When the color detection sequence matches the predetermined color sequence, it is determined that the color wiring harness of the device to be detected is correctly connected; when the color detection sequence does not match the predetermined color sequence, the color detection sequence is adjusted and updated according to the position distribution of the binary segmentation mask of the connecting line of each color label; When the color detection order after adjustment and update is consistent with the predetermined color order, it is determined that the color wiring harness of the device to be detected is correctly connected; otherwise, it is determined that the color wiring harness of the device to be detected is incorrect, and it is determined that there is a wiring error in the connecting wire corresponding to the color label that does not match the predetermined color order in the adjusted and updated color detection order.

2. The color wire harness connection quality detection method according to claim 1, characterized in that: The initial color detection sequence includes: Performing skeleton thinning and pixel uniqueness processing on the connecting line binary segmentation mask corresponding to each color label to obtain a connecting line skeleton image of the color label, wherein the connecting line skeleton image of the color label has only one foreground pixel in each column; Determine the pixel coordinates (x, y) of the foreground pixels at both end points of the connecting line skeleton image for each color label in a pixel coordinate system, and select the foreground pixels with larger y coordinates at both end points of the connecting line skeleton image as candidate endpoints of the connecting line of the color label; wherein the pixel coordinate system has the upper left corner pixel of the circuit board image as the coordinate origin, the rightward direction along the row direction is the x direction, and the downward direction along the column direction is the y direction; The candidate endpoints of the connecting lines of each color label are sorted in ascending order of x-coordinates to obtain the initial color detection order.

3. The color wire harness connection quality detection method according to claim 2, characterized in that: The pixel uniqueness processing of the binary segmentation mask of the connecting line corresponding to each color label includes: Each column of pixels in the connecting line binary segmentation mask after the skeleton thinning process is traversed in sequence. For any column of pixels traversed, the foreground pixel with the largest y coordinate is retained, and the other pixels in the current column are set as background pixels. After traversing all columns, the pixel uniqueness processing is completed and the connecting line skeleton image with the color label is obtained.

4. The method for detecting the quality of color wire harness connection according to claim 2, wherein: Adjusting and updating the color detection order according to the position distribution of the binary segmentation mask of the connecting line of each color label includes: Traversing each group of color labels in sequence according to the initial color detection order, wherein each group of color labels traversed includes a first color label and a second color label that are adjacent in the color detection order, and the x-coordinate of the candidate endpoint of the connecting line of the first color label is smaller than the candidate endpoint of the connecting line of the second color label; For each group of color labels traversed, detecting whether the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label have completely staggered position distributions; When it is determined that the binary segmentation mask of the connecting lines of the first color label and the binary segmentation mask of the connecting lines of the second color label do not have a completely staggered position distribution, performing line intersection detection on the binary segmentation mask of the connecting lines of the first color label and the binary segmentation mask of the connecting lines of the second color label, and when it is detected that the order of the first color label and the second color label is wrong due to line intersection, swapping the order of the first color label and the second color label in the color detection order; otherwise, maintaining the order of the first color label and the second color label in the color detection order unchanged; When it is determined that the position distribution of the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label are completely staggered, the average value of the x-coordinates of all the first color foreground pixels and the average value of the x-coordinates of all the second color foreground pixels are calculated. When the average value of the x-coordinates of all the first color foreground pixels is less than or equal to the average value of the x-coordinates of all the second color foreground pixels, the order of the first color label and the second color label in the color detection order remains unchanged; otherwise, the order of the first color label and the second color label in the color detection order is swapped; wherein the first color foreground pixel is the foreground pixel in the connecting line binary segmentation mask of the first color label, and the second color foreground pixel is the foreground pixel in the connecting line binary segmentation mask of the second color label.

5. The method for detecting the quality of color wire harness connection according to claim 4, characterized in that: Detecting whether the position distribution of the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label are completely staggered includes: Detecting whether the number of overlapping distribution rows of the first color label and the second color label reaches a row number threshold, wherein the overlapping distribution rows are pixel rows that include both first color foreground pixels and second color foreground pixels in the same row; When the overlapping distribution rows reach the row number threshold, it is determined that the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label do not have a completely staggered position distribution; otherwise, it is determined that the binary segmentation mask of the connecting line of the first color label and the binary segmentation mask of the connecting line of the second color label do not have a completely staggered position distribution.

6. The method for detecting the quality of color wire harness connection according to claim 5, characterized in that: Performing intersection detection on the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label includes: Sequentially traverse each overlapping distribution row of the first color label and the second color label, and for each traversed overlapping distribution row, count the number of abnormal pixels in the overlapping distribution row, where the abnormal pixel in each overlapping distribution row is a foreground pixel of the first color whose x coordinate is greater than the x coordinate of a foreground pixel of the second color in the overlapping distribution row; When the number of abnormal pixels in all overlapping distribution rows is not greater than the pixel number threshold, determining that there is no line intersection between the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label, and maintaining the order of the first color label and the second color label in the color detection order unchanged; When the number of abnormal pixels in the overlapping distribution row exceeds the pixel number threshold, and when the average value of the x-coordinates of all first color foreground pixels is less than or equal to the average value of the x-coordinates of all second color foreground pixels, it is determined that the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label have line intersections but do not cause an order error, and the order of the first color label and the second color label in the color detection order remains unchanged; When the number of abnormal pixels in the overlapping distribution rows exceeds the pixel number threshold, and when the average value of the x-coordinates of all the first color foreground pixels is calculated to be greater than the average value of the x-coordinates of all the second color foreground pixels, it is determined that the connecting line binary segmentation mask of the first color label and the connecting line binary segmentation mask of the second color label have line intersections and cause an order error, and the order of the first color label and the second color label in the color detection order is swapped.

7. The method for detecting the quality of color wire harness connection according to claim 1, characterized in that: Generating a connecting line binary segmentation mask corresponding to each color label based on pixels of the color label includes: Extracting pixels of each color label and converting them into a grayscale image to obtain the original segmentation mask area of ​​the color label; The original segmentation mask area of ​​each color label is binarized and the interference area with an area smaller than the area threshold is deleted to obtain the connecting line binary segmentation mask corresponding to the color label.

8. The method for detecting the quality of color wire harness connection according to claim 1, characterized in that: The target detection model adopts the lightweight PICO series target detection model, which includes a backbone network, a neck network and a head network cascaded in sequence. The backbone network adopts LCNet for feature extraction and outputs feature maps of three different scales; the neck network adopts LCPAN to fuse the feature maps of three different scales. The output channel number of LCPAN is 128 and it adopts depthwise separable convolution; the head network adopts PicoHeadV2 for target detection. PicoHeadV2 includes a convolutional layer for feature extraction and a fully connected layer for category prediction and bounding box regression.

9. The method for detecting the quality of color wire harness connection according to claim 1, characterized in that: The semantic segmentation model uses the semantic segmentation model of the PPLite series. The semantic segmentation model uses STDC1 as the backbone network, and three increasing output channels are designed between the backbone network and the semantic segmentation head to gradually extract and integrate image features. The semantic segmentation head sets three intermediate layers to refine the feature representation.

10. The method for detecting the quality of color wire harness connection according to claim 1, characterized in that: The identifying the terminal area in the original image by using the target detection model further includes: Performing downsampling processing on the original image; For the original image that has completed the downsampling process, linear interpolation is first performed on two adjacent pixels in the horizontal direction, and then linear interpolation is performed again in the vertical direction to obtain the original image that has completed the linear interpolation process; The original image after linear interpolation is smoothed using median filtering to obtain the preprocessed original image; The object detection model is used to identify the terminal block area in the preprocessed raw image.

Images