A method, device and medium for intelligent identification of wire harness defects based on deep learning

Abstract

This invention discloses a method, device, and medium for intelligent identification of wire harness defects based on deep learning, relating to the field of industrial defect identification technology. The method includes: using a deep learning recognition network to perform joint deep learning representation and defect candidate identification on topology-aligned image sequences, outputting an initial list of defect candidates; verifying the structural relationships of the initial list of defect candidates based on the wire harness topology prior graph, and incorporating the consistency verification conclusion into the defect candidates as a judgment criterion, while correcting the target region identifier, generating a structural constraint defect judgment list; and performing online inference calibration on the structural constraint defect judgment list and re-judging boundary samples to generate a wire harness defect judgment conclusion. This invention uses a deep learning recognition network to perform joint deep learning representation and defect candidate identification on topology-aligned image sequences, enabling defect candidate identification to simultaneously consider local texture anomalies and global morphological relationship information, and improving the distinguishability of defects in complex critical connection areas.

Description

Technical Field

[0001] This invention relates to the field of industrial defect identification technology, and in particular to a method, device and medium for intelligent identification of wire harness defects based on deep learning. Background Technology

[0002] In automotive and equipment manufacturing scenarios, the appearance and position quality inspection of critical connection areas of wiring harnesses typically relies on industrial vision. Conventional methods often acquire overall views of the wiring harness and images of key local areas, combining these with process configuration records to complete area localization, status interpretation, and quality traceability. In the identification stage, deep learning networks such as convolutional neural networks, ResNet, U-Net, or YOLO are often introduced to perform defect detection and classification identification of crimp terminal areas, sealing ring assembly areas, and connector mating areas.

[0003] However, the above-mentioned conventional methods still have room for improvement: on the one hand, their interpretation criteria are mostly concentrated on local texture or spatial location features, and the collaborative utilization of topological information such as terminal configuration relationships and branch connection relationships is relatively limited, and the ability to distinguish structural consistency under complex connection relationships still needs to be enhanced; on the other hand, under conditions of wire harness model switching, light fluctuations or machine differences, the consistency of detection results and the stability of regional attribution are also easily affected. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a deep learning-based intelligent identification method for wire harness defects to solve the problem that the accuracy and stability of defect identification under complex connection relationships are limited due to insufficient utilization of wire harness topology information in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for intelligent identification of wire harness defects based on deep learning, which includes: acquiring an overall view and a key connection area view of the wire harness to be inspected, and constructing a wire harness topology prior map in combination with process configuration records; Based on the harness topology prior diagram, pose normalization and structural alignment are performed on the overall view and the key connection area view, and the target region is segmented to generate a topology-aligned image sequence. Based on a deep learning recognition network, deep learning joint representation and defect candidate recognition are performed on topology-aligned image sequences, and an initial list of defect candidates is output. Based on the prior diagram of the wire harness topology, the structural relationship of the initial list of defect candidates is reviewed, and the consistency verification conclusion is written into the defect candidates as the basis for judgment and the target area is corrected to generate a list of structural constraint defects. The list of structural constraint defects is used for online inference calibration and boundary sample re-judgment to generate a conclusion on the determination of wire harness defects.

[0007] As a preferred embodiment of the deep learning-based intelligent identification method for wire harness defects described in this invention, the steps for constructing a wire harness topology prior graph by combining process configuration records are as follows: An industrial area array camera is used to acquire an overall view and a view of the key connection area of ​​the wire harness to be inspected, and the exposure clarity quality is verified. Using product identification information from the overall view and key connection area view as search criteria, read the corresponding process configuration records, perform field parsing and extract terminal configuration relationships and branch connection relationships, and generate topology construction information; Based on the topology construction information and the structural outlines of the overall view and the key connection area view, node relationships and connection relationships are established, and a priori diagram of the harness topology is generated.

[0008] As a preferred embodiment of the deep learning-based intelligent identification method for wire harness defects described in this invention, the steps of performing pose normalization and structural alignment on the overall view and the key connection area view, and segmenting the target region to generate a topologically aligned image sequence are as follows: Based on the prior diagram of the wire harness topology, benchmark positioning and attitude parameter extraction are performed on the overall view and the key connection area view to obtain benchmark positioning information and attitude parameters; Based on the baseline positioning information and attitude parameters, the overall view and the critical connection area view are rotated and translated and scaled to generate a standardized overall view and critical connection area view. Based on the standardized overall view and key connection area view, structural alignment mapping is performed, and key connection areas are located and target regions are segmented to obtain a topology-aligned image sequence.

[0009] As a preferred embodiment of the intelligent identification method for wire harness defects based on deep learning described in this invention, the deep learning identification network refers to a dual-branch identification network that uses a convolutional backbone and a Swing Transformer global encoder in parallel, and is then connected to a decoupled detection head after feature pyramid fusion.

[0010] As a preferred embodiment of the deep learning-based intelligent identification method for wire harness defects described in this invention, the steps of performing deep learning joint representation and defect candidate identification on the topologically aligned image sequence to output an initial list of defect candidates are as follows: Using a convolutional backbone branch, multi-scale feature extraction is performed on the topologically aligned image sequence, and noise suppression and illumination normalization are completed to obtain local texture defect feature maps. Using the global encoding branch of the Swing Transformer, global structural relationship modeling is performed on the topologically aligned image sequence and long-range dependency features are extracted to obtain a global morphological relationship feature map; Based on the local texture defect feature map and the global morphological relationship feature map, feature pyramid fusion and scale alignment are performed to obtain a joint representation feature map; Based on the joint characterization feature map, defect category prediction and defect location prediction are performed by decoupling the detection head to obtain defect candidates; Based on the defect candidates, perform candidate purification and overlap reduction, and associate the defect candidates with the target region identifier of the topologically aligned image sequence to output an initial list of defect candidates.

[0011] As a preferred embodiment of the deep learning-based intelligent identification method for wire harness defects described in this invention, the steps of verifying the structural relationships of the initial list of defect candidates and correcting the target region identification based on the wire harness topology prior graph to generate a list of structural constraint defects are as follows: Based on the harness topology prior diagram, each defect candidate in the initial defect candidate list is mapped to the corresponding target area identifier and topology node relationship, the topology mapping defect candidate is obtained and the connection relationship consistency check is performed to generate the consistency check conclusion; Based on the consistency verification conclusion, consistency filtering is performed on the topology mapping defect candidates and the target area identification is corrected to obtain the corrected defect candidates. At the same time, the defect candidates that need to be re-judged are marked. The revised defect candidates are compiled and their association with the target area identifiers in the harness topology prior diagram is maintained to generate a list of structural constraint defects.

[0012] As a preferred embodiment of the deep learning-based intelligent identification method for wire harness defects described in this invention, the steps of performing online inference calibration on the structural constraint defect judgment list and re-judging boundary samples to generate wire harness defect judgment conclusions are as follows: Calculate the defect confidence level of each defect candidate in the structural constraint defect judgment list, and use the defect candidates with defect confidence levels higher than the confidence level threshold as the online calibration sample set, while using the defect candidates that need to be re-judged as the boundary re-judgment sample set; Online inference calibration is performed on the deep learning recognition network based on the online calibration sample set to obtain the calibrated deep learning recognition network. Using the calibrated deep learning recognition network, the defect candidate recognition is performed again on the topology aligned image sequence corresponding to the boundary re-judgment sample set, and the structural constraint defect judgment list is corrected to obtain the re-judgment defect item list. The list of re-judged defects is then filled back into the corresponding entries of the list of structural constraint defects, while maintaining consistency in the target area identification, ultimately generating the harness defect judgment conclusion.

[0013] As a preferred embodiment of the deep learning-based intelligent identification method for wire harness defects described in this invention, the following means that maintaining the consistency of the target region identifier means that each defect candidate in the re-judgment defect list continues to use the same target region identifier as the wire harness topology prior map without changing the correspondence of the target region identifiers when merging it into the structural constraint defect judgment list.

[0014] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the deep learning-based intelligent identification method for wire harness defects as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the deep learning-based intelligent identification method for wire harness defects as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: by performing deep learning joint representation and defect candidate identification on topology-aligned image sequences through a deep learning recognition network, the defect candidate identification can simultaneously take into account local texture anomalies and global morphological relationship information and improve the distinguishability of defects in complex key connection areas; and by performing online inference calibration through an online calibration sample set to obtain a calibrated deep learning recognition network, a list of re-judged defect items is formed on the boundary re-judgment sample set and incorporated into the wire harness defect judgment conclusion, thereby improving the stability of judgment under changing working conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of intelligent identification of wire harness defects based on deep learning.

[0019] Figure 2 This is a schematic diagram of a deep learning network for identifying network structures.

[0020] Figure 3 This is a line graph showing the impact of the interval scaling factor on the number of online calibration sample sets and the number of boundary re-judgment sample sets.

[0021] Figure 4 This is a heatmap showing the impact of the ratio of defect area to target area and the aspect ratio of defect on the recall rate of defect identification in complex critical connection areas. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a method for intelligent identification of wire harness defects based on deep learning, including the following steps: S1. Collect the overall view and key connection area view of the wire harness to be inspected, and construct the wire harness topology prior diagram in combination with the process configuration record.

[0026] An industrial area scan camera is used to capture an overall view and a view of the key connection area of ​​the wire harness under inspection, and the exposure clarity quality is verified.

[0027] Furthermore, the exposure sharpness quality verification specifically includes: performing brightness range checks and sharpness checks on the overall view and the key connection area view respectively. The brightness range check determines whether there is overexposure or underexposure by statistically analyzing the grayscale distribution. For example, the percentage of pixels with grayscale values ​​close to the maximum value in the overall view and the key connection area view is used to determine overexposure and saturation, and the percentage of pixels with grayscale values ​​close to the minimum value is used to determine underexposure. The sharpness check uses the Laplacian variance method to calculate the edge gradient intensity to determine whether there is obvious out-of-focus or motion blur. For example, first calculate the Laplacian response for the overall view and the key connected area view and take the absolute value. Then, count the percentage of pixels with response values ​​in the high response range of the whole image and the length of their connected edges. When the percentage of high response pixels is low and the connected edges are scattered short segments (e.g., the percentage of high response pixels is less than 1% and the total length of connected edges is less than 200 pixels), it is determined that there is obvious out-of-focus or motion blur. When the percentage of high response pixels is high and the connected edges form continuous long sides along the contour (e.g., the percentage of high response pixels is about 5% to 12% and the total length of connected edges is about 800 to 2000 pixels), it is determined that there is no obvious out-of-focus or motion blur.

[0028] Using product identification information from the overall view and key connection area view as search criteria, the corresponding process configuration records are read, field parsing is performed, and terminal configuration relationships and branch connection relationships are extracted to generate topology construction information.

[0029] Furthermore, the product identification information in the overall view and the key connection area view is used as the retrieval key to read the corresponding process configuration record. The process configuration record is parsed according to the preset field mapping table to extract the terminal number field, terminal model field, wire specification field and branch number field. The terminal number field and the terminal model field are paired according to the same number association rule to generate the terminal configuration relationship. The branch number field and the terminal number field are paired according to the connection mapping rule to generate the branch connection relationship. The terminal configuration relationship and the branch connection relationship are merged to obtain the topology construction information. The topology construction information is used for the subsequent construction of the wire harness topology prior diagram and supports the generation of topology alignment image sequence.

[0030] It should be noted that the field mapping table is configured and set according to the field dictionary and data interface specifications of the process configuration record, specifying the correspondence between the field names and data types of the terminal number field, terminal model field, wire specification field, and branch number field.

[0031] The same-number association rule refers to the rule of matching and associating the terminal model field value corresponding to the same terminal number field value with the value of the terminal number field as the primary key. It is set according to the uniqueness constraint of the terminal number field in the process configuration record (the same terminal number field value can only correspond to one terminal model field value in the same process configuration record and the terminal number field value cannot be repeated).

[0032] Connection mapping rules refer to the rules for matching and associating the terminal number fields pointed to by the branch number field with the corresponding values ​​of the branch number field and the terminal number field. These rules are set according to the coding specifications of the branch connection definition field in the process configuration record.

[0033] Based on the topology construction information and the structural outlines of the overall view and the key connection area view, node relationships and connection relationships are established, and a priori diagram of the harness topology is generated.

[0034] The critical connection area refers to the injection-molded area, crimped terminal area, sealing ring assembly area, and connector mating area in the wire harness under inspection, which are directly related to the reliability of electrical connections and assembly positioning.

[0035] Product identification information refers to identification fields such as work order number, product serial number, and batch number, which are used to uniquely associate the wire harness under inspection.

[0036] The process configuration record includes the terminal model, wire specification configuration parameters, injection molding configuration parameters, and crimping configuration parameters corresponding to the wire harness to be inspected.

[0037] S2. Based on the harness topology prior diagram, perform attitude normalization and structural alignment on the overall view and the key connection area view, and segment the target region to generate a topology-aligned image sequence.

[0038] Based on the prior diagram of the wire harness topology, benchmark positioning and attitude parameter extraction are performed on the overall view and the key connection area view to obtain benchmark positioning information and attitude parameters.

[0039] Furthermore, based on the prior topology diagram of the wire harness, the reference target area identifiers that need to be aligned in the overall view and the critical connection area view are determined, and the corresponding reference features are selected. The reference features are represented by a set of feature points of the fixture positioning hole edge, the inflection point of the wire harness trunk contour, and the corner point of the terminal shape. The overall view and the critical connection area view obtain a set of candidate feature points through edge detection and corner detection, and match them with the reference features to determine the reference point position and reference direction. The reference point position and reference direction are summarized to form the reference positioning information. The reference positioning information is used to calculate the rotation angle and translation amount of the overall view and the critical connection area view and generate attitude parameters. The reference positioning information and attitude parameters are subsequently used to perform rotation and translation correction on the overall view and the critical connection area view and complete the scale unification to obtain the standardized overall view and the standardized critical connection area view.

[0040] Based on the reference positioning information and attitude parameters, the overall view and the critical connection area view are rotated and translated for correction and scale unification, generating standardized overall view and critical connection area view.

[0041] Furthermore, based on the reference positioning information, the reference point positions and reference directions of the overall view and the key connection area view are determined. Based on the attitude parameters, the rotation angle and translation amount of the overall view and the key connection area view are determined, and a geometric transformation matrix is ​​constructed. The overall view and the key connection area view perform rotation correction and translation correction according to the geometric transformation matrix. After rotation and translation correction, the overall view and the key connection area view perform scale unification according to the reference scale conversion relationship in the reference positioning information so that the overall view and the key connection area view reach a consistent pixel scale and a unified coordinate origin. The overall view and the key connection area view after scale unification are used as standardized overall view and standardized key connection area view for subsequent structure alignment mapping and target region segmentation driven by the harness topology prior map and to generate a topology aligned image sequence.

[0042] It should be noted that the reference scale conversion relationship refers to the conversion correspondence between the measured length of the reference feature in pixel coordinates and the corresponding physical calibration length in the reference positioning information.

[0043] Based on the standardized overall view and key connection area view, structural alignment mapping is performed, and key connection areas are located and target regions are segmented to obtain a topology-aligned image sequence.

[0044] Furthermore, based on the harness topology prior diagram, the terminal configuration relationship and branch connection relationship are read, and the target area identifier set is determined. The standardized overall view and the standardized critical connection area view establish a coordinate mapping relationship according to the target area identifier set to complete the structural alignment mapping. The structural alignment mapping is achieved by mapping the target area identifiers in the harness topology prior diagram to the expected position range in the standardized overall view and the standardized critical connection area view. The harness topology prior diagram drives the standardized overall view and the standardized critical connection area view to perform critical connection area positioning and determine the area boundaries corresponding to the injection molding covering area, crimp terminal area, sealing ring assembly area and connector insertion area. The area boundaries are used to segment the target areas of the standardized overall view and the standardized critical connection area view and obtain a topology aligned image sequence arranged according to the target area identifier. The topology aligned image sequence is subsequently used by the deep learning recognition network to perform defect candidate recognition and output an initial defect candidate list.

[0045] S3. Based on a deep learning recognition network, perform deep learning joint representation and defect candidate recognition on topology-aligned image sequences, and output an initial list of defect candidates.

[0046] Deep learning recognition networks refer to dual-branch recognition networks that use a convolutional backbone and a Swing Transformer (shifting window visual Transformer network) for global encoding in parallel, and then fuse them through a feature pyramid before connecting to a decoupled detection head. It should be noted that the deep learning recognition network uses historical topology-aligned image sequences as training data and organizes training batches uniformly according to an input resolution of 1024×1024 and a batch size of 16. During training, a transfer learning training method is used to initialize the parameters of the convolutional backbone and the Swin Transformer global encoding parallel structure, and the AdamW optimizer is used to perform end-to-end updates with an initial learning rate of 0.0001 and a weight decay of 0.05. Within 100 training rounds, 5 rounds of warm-up and cosine annealing are used to adjust the learning rate and continuously output fused defect representation feature maps by fusing feature pyramids. The decoupled detection head calculates classification loss and localization loss simultaneously in each iteration to jointly optimize defect category prediction and defect location prediction. The Swin Transformer uses a window size of 7 and an embedding dimension of 96 and is configured with a four-stage depth of 2-2-6-2 to enhance the global morphological relationship representation.

[0047] By employing a convolutional backbone branch, multi-scale feature extraction is performed on the topologically aligned image sequence, and noise suppression and illumination normalization are completed to obtain local texture defect feature maps.

[0048] Furthermore, the convolutional backbone performs convolutional downsampling and feature stacking on the topology-aligned image sequence frame by frame to form local texture responses at different receptive field scales. Noise suppression reduces the interference of random noise and stripe noise on the texture response through smooth convolution and normalization in the shallow stage of the convolutional backbone. Illumination normalization ensures that the topology-aligned image sequence maintains consistent texture contrast characteristics under different exposure conditions through intensity normalization and contrast stretching in the middle stage of the convolutional backbone. Local texture responses at different scales are scale-aligned and converged in the final layer of the convolutional backbone to form a local texture defect feature map.

[0049] It should be noted that texture contrast characteristics refer to the degree of difference in grayscale intensity and local gradient changes between defective textures and surrounding normal areas in a topologically aligned image sequence.

[0050] Local texture defect feature maps refer to feature representations used to characterize fine-grained texture anomalies such as crack edges, bubble boundaries, and burr protrusions in topologically aligned image sequences, which are used to improve the distinguishability of minute defects in subsequent defect candidate identification.

[0051] The global encoding branch of the Swing Transformer is used to model the global structural relationship of the topologically aligned image sequence and extract long-range dependency features to obtain a global morphological relationship feature map.

[0052] Furthermore, the Swing Transformer global encoding branch is used to perform in-window self-attention calculation on the topology-aligned image sequence according to the shift window division method, and shift interaction is performed between adjacent windows to establish cross-regional correlation. After position encoding, the topology-aligned image sequence continuously aggregates the structural information of terminal area contour, injection molding covering area shape and branch direction in multi-layer self-attention to form a global structural relationship modeling result. Long-range dependent features write the morphological changes and relative positional relationships of distant areas into the same feature representation through cross-window interaction to reflect the overall morphological consistency. The global structural relationship modeling result is scale-aligned at the resolution level to obtain a global morphological relationship feature map.

[0053] It should be noted that long-range dependency features refer to the correlation features between target regions that are far apart in a topologically aligned image sequence in terms of morphological changes and relative positional relationships. These features are used to maintain a stable representation of the overall structural consistency and reduce the risk of misjudging structural defects even when there is local occlusion or local noise interference.

[0054] Global morphological relationship feature map refers to the feature representation used to characterize the overall morphological consistency and relative positional relationship between the terminal area contour and branch direction and the shape of the injection-molded covering area in the topologically aligned image sequence, thereby improving the identifiability of structural defects such as terminal offset and injection deformation in subsequent defect candidate identification.

[0055] Based on the local texture defect feature map and the global morphological relationship feature map, feature pyramid fusion and scale alignment are performed to obtain a joint representation feature map.

[0056] Furthermore, based on the local texture defect feature map and the global morphological relationship feature map, multi-resolution feature layers are constructed in the feature pyramid fusion process, and upsampling and downsampling are performed to align spatial dimensions. Scale alignment is achieved by converting the local texture defect feature map and the global morphological relationship feature map into consistent feature dimensions through channel mapping, while maintaining the consistency of the spatial position corresponding to the target region identifier. The multi-resolution feature layers with scale alignment are fused layer by layer at the same level to simultaneously retain local texture details and global morphological relationship information. The layer-by-layer fusion results are then aggregated between layers to obtain a joint representation feature map.

[0057] It should be noted that the joint characterization feature map is used to simultaneously carry local texture defect information and global morphological relationship information, avoiding misjudgment of structural defects caused by relying solely on local texture and missing detection of minute texture defects caused by relying solely on global morphology. This improves the ability to distinguish defects of different scales and the robustness to illumination noise interference, thereby providing a more complete and interference-resistant feature basis for subsequent defect candidate identification.

[0058] Based on the joint characterization feature map, defect category prediction and defect location prediction are performed by decoupling the detection head to obtain defect candidates.

[0059] Furthermore, based on the joint representation feature map, a defect category prediction branch and a defect location prediction branch are established in the decoupled detection head. The defect category prediction branch generates a category discrimination vector through convolutional layer stacking and global pooling, and outputs the defect category label through a multi-class discrimination function. The defect location prediction branch generates a location regression vector through convolutional layer stacking and outputs the defect location range parameter. The category discrimination of the defect category prediction branch is based on the texture anomaly response and morphological relationship response of the candidate region in the joint representation feature map. For example, when the candidate region presents a continuous, thin, high-gradient edge and forms obvious fracture features with the surrounding background, the defect category label corresponding to the crack is output. When the candidate region presents a closed boundary and an internal gray-level abrupt change with a smooth boundary, the defect category label corresponding to the bubble is output. The defect category label and the defect location range parameter are correlated in the decoupled detection head to form a candidate description. The candidate description is transformed to generate defect candidates and retains the defect category label and the defect location range parameter.

[0060] It should be noted that the multi-class discriminant function is as follows: the defect category prediction branch processes the joint representation feature map corresponding to the candidate region through several layers of convolution and normalized activation to obtain a fixed-dimensional class discriminant vector. Then, it generates a class response vector with the same number of defect categories through linear mapping. Finally, the maximum response selection strategy is used to map the class index with the largest response value in the class response vector to the defect category label and output it. The maximum response selection strategy means selecting the class index corresponding to the item with the largest value in the class response vector and mapping the class index to the output defect category label.

[0061] Based on the defect candidates, perform candidate purification and overlap reduction, and associate the defect candidates with the target region identifier of the topologically aligned image sequence to output an initial list of defect candidates.

[0062] Furthermore, candidate cleanup determines whether the defect location range parameter falls within the effective range of the target area based on the inclusion relationship between the defect location range parameter and the target area boundary (e.g., all four boundary points of the defect location range parameter are located inside the target area boundary). It also determines whether the defect candidate meets the defect morphology constraints based on the area range and aspect ratio of the defect location range parameter (e.g., the defect location range parameter corresponding to a crack has a high aspect ratio and a slender shape, while the defect location range parameter corresponding to a bubble has a near-circular shape with an aspect ratio close to one). For overlap elimination, the overlap degree of defect candidates with overlapping defect location range parameters is calculated using the intersection-union ratio (IUU). Defect candidates whose defect location range parameter area is more consistent with the common size range of the corresponding defect category label and whose aspect ratio of the defect location range parameter conforms to the morphological characteristics of the corresponding defect category label are retained to eliminate duplicate candidates.

[0063] like Figure 4As shown, under different defect area ratios and defect aspect ratios, the overall recall rate for identifying defects in complex critical connection regions remains at a high level. In particular, it can still maintain effective identification even under the more difficult conditions of a small defect area ratio and a large defect aspect ratio. This indicates that deep learning recognition networks can simultaneously take into account local texture anomalies and global morphological relationship information, forming a relatively stable discrimination ability for defects of different scales and shapes, thereby improving the distinguishability of defects in complex critical connection regions.

[0064] S4. Based on the prior diagram of the wire harness topology, the initial list of candidate defects is structurally verified and the target area identification is corrected to generate a list of structural constraint defects.

[0065] Based on the harness topology prior diagram, each defect candidate in the initial defect candidate list is mapped to the corresponding target area identifier and topology node relationship. The topology mapping defect candidate is obtained and the connection relationship consistency check is performed to generate the consistency check conclusion.

[0066] Furthermore, based on the harness topology prior diagram, the target area identifier set and topology node relationship are read. Each defect candidate in the initial defect candidate list is matched with the target area identifier according to the correspondence between the defect location range parameter and the area coordinates of the topology aligned image sequence, and written into the topology node relationship to obtain the topology mapping defect candidate. The connection relationship consistency check compares the target area identifier and topology node relationship of the topology mapping defect candidate based on the branch connection relationship and terminal configuration relationship in the harness topology prior diagram. It determines whether the target area identifier satisfies the connection adjacency constraint and sequence constraint and records the check status: When the harness topology prior diagram stipulates that the crimp terminal area and the connector insertion area are adjacent connection relationships, the topology mapping defect candidate corresponding to the crimp terminal area and the topology mapping defect candidate corresponding to the connector insertion area satisfy the adjacent connection relationship and are judged as consistent. When the harness topology prior diagram stipulates that the terminal number order is A1 before A2, the case where the target area identifier of A2 is located upstream of the target area identifier of A1 is judged as inconsistent. The check status is summarized to generate a consistency check conclusion.

[0067] It should be noted that the connection adjacency constraint refers to the requirement that there must be a direct connection between adjacent target area identifiers as specified in the harness topology prior graph, and that there must be no connection jump constraint that crosses non-adjacent target area identifiers.

[0068] The correspondence between regional coordinates refers to the correspondence between the position range of the defect location range parameter in the coordinate system of the topology-aligned image sequence and the regional boundary corresponding to each target region identifier.

[0069] Sequence constraints refer to the constraints that the order of target region identifiers in the connection link, as specified in the harness topology prior graph, must not be reversed or crossed.

[0070] Based on the consistency verification results, consistency filtering is performed on the topology mapping defect candidates and the target area identifier is corrected to obtain the corrected defect candidates. At the same time, the defect candidates that need to be re-judged are marked.

[0071] Furthermore, based on the consistency verification conclusion, the verification status corresponding to the topology mapping defect candidates is read and consistency filtering is performed. Consistency filtering removes topology mapping defect candidates with inconsistent verification status to eliminate those that do not meet the connection adjacency constraints and order constraints. For topology mapping defect candidates with correctable verification status, the target region identifier is re-matched and updated based on the target region identifier set of the harness topology prior graph and the topology node relationship. Topology mapping defect candidates with conflicting verification status are retained and marked as defect candidates that need to be re-judged. After consistency filtering and target region identifier correction are completed, the corrected defect candidates are obtained.

[0072] The revised defect candidates are compiled and their association with the target area identifiers in the harness topology prior diagram is maintained to generate a list of structural constraint defects.

[0073] It should be noted that the target area identification association relationship refers to the one-to-one correspondence between the target area identification recorded in the corrected defect candidate and the target area identification with the same name and the corresponding topology node relationship in the wire harness topology prior diagram. For example, when the target area identification in the corrected defect candidate is the crimp terminal area, the topology node corresponding to the crimp terminal area is located in the wire harness topology prior diagram and the connection relationship between the crimp terminal area and the connector insertion area is associated.

[0074] S5. Perform online inference calibration on the list of structural constraint defects and re-judge the boundary samples to generate a conclusion on the determination of wire harness defects.

[0075] Calculate the defect confidence level of each defect candidate in the structural constraint defect judgment list, and use the defect candidates with defect confidence levels higher than the confidence level threshold as the online calibration sample set, while using the defect candidates that need to be re-judged as the boundary re-judgment sample set.

[0076] Furthermore, when calculating the defect confidence score for each defect candidate in the structural constraint defect judgment list, the defect confidence score is obtained by normalizing the category response output by the deep learning recognition network in the defect candidate recognition stage and writing it into the corresponding entry in the structural constraint defect judgment list. After comparing the defect confidence score with the confidence score threshold, defect candidates with a defect confidence score higher than the confidence score threshold are selected as the online calibration sample set. At the same time, defect candidates marked as needing re-judgment in the structural constraint defect judgment list are summarized as the boundary re-judgment sample set.

[0077] When calculating the defect confidence score for each defect candidate in the structural constraint defect assessment list, the expression is: ; in, It is the first The defect confidence level of each defect candidate; It is the first The index of the defect category label in the category discrimination vector of the defect candidate. The component at the location; It is the first The category discrimination vector of the defect candidate, excluding the index Largest competing index The component at that location is used to represent the component corresponding to the closest competing defect category; It is the interval scaling factor, used to adjust... Sensitivity to changes in defect confidence level.

[0078] It should be noted that the interval scaling factor is based on the calibration result that best matches the defect confidence with the actual recognition accuracy on the validation set of historical topology-aligned image sequences. The corresponding value is selected by performing a grid search within the candidate value range and using minimizing the calibration error as the criterion.

[0079] The confidence threshold is based on the statistical distribution of the structural constraint defect judgment list generated from historical topology aligned image sequences. It is set by traversing the candidate values ​​and selecting the value that ensures the online calibration sample set contains only defect candidates whose consistency verification conclusion is consistent and who are not marked as needing re-judgment.

[0080] Online inference calibration is performed on the deep learning recognition network based on the online calibration sample set to obtain the calibrated deep learning recognition network.

[0081] like Figure 3 As shown, with the interval scaling factor As the value increases from 0.50 to 0.90, the number of samples in the online calibration dataset decreases, while the number of samples in the boundary re-judgment dataset increases, indicating that the interval scaling factor... This system can effectively adjust the distribution of defect confidence levels, allowing high-confidence defect candidates to be more concentrated in the online calibration sample set, and allowing defect candidates with higher uncertainty to be more fully included in the boundary re-judgment sample set. As a result, the sample selection for online inference calibration is more targeted, and the coverage of boundary re-judgment is enhanced, thus improving the stability of harness defect determination conclusions under changing operating conditions.

[0082] Furthermore, online inference calibration is a process of slightly calibrating the deep learning recognition network using the topology-aligned image sequence corresponding to the online calibration sample set without using historical topology-aligned image sequences to adapt to the current operating condition drift. The deep learning recognition network performs forward inference on the online calibration sample set and applies intensity perturbation and geometric perturbation to the online calibration sample set before inference again. Based on the consistency of the defect candidates obtained from the two inferences, the normalized statistics and lightweight adjustable parameters are iteratively updated, and the backbone parameters of the convolutional backbone and the global encoding of the Swing Transformer are frozen. The iteration continues until the defect candidate output is stable, resulting in the calibrated deep learning recognition network.

[0083] Using the calibrated deep learning recognition network, the defect candidate recognition is performed again on the topology aligned image sequence corresponding to the boundary re-judgment sample set, and the structural constraint defect judgment list is corrected to obtain the re-judgment defect item list.

[0084] Furthermore, when the calibrated deep learning recognition network is used to perform defect candidate recognition again on the topology-aligned image sequence corresponding to the boundary re-judgment sample set, the target region identifier corresponding to the boundary re-judgment sample set guides the region localization of the topology-aligned image sequence and outputs new defect candidates. The new defect candidates form a re-judgment defect item list based on candidate purification and overlap resolution. The re-judgment defect item list and the structural constraint defect judgment list are matched according to the target region identifier and inconsistent items are replaced and updated, thereby completing the correction of the structural constraint defect judgment list.

[0085] By combining the list of re-evaluated defects with the revised list of structural constraint defects, and ensuring consistency in the target area identification, a final conclusion on the determination of wire harness defects is generated.

[0086] Maintaining consistency in target area identifiers means that each defect candidate in the list of reassessed defect items continues to use the same target area identifier as the harness topology prior diagram when it is merged into the structural constraint defect assessment list, without changing the correspondence between target area identifiers.

[0087] This embodiment also provides a computer device applicable to the intelligent identification method for wire harness defects based on deep learning, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the intelligent identification method for wire harness defects based on deep learning as proposed in the above embodiment.

[0088] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0089] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the deep learning-based intelligent identification method for wire harness defects as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0090] In summary, this invention utilizes a deep learning recognition network to perform joint deep learning representation and defect candidate identification on topologically aligned image sequences. This enables defect candidate identification to simultaneously consider local texture anomalies and global morphological relationship information, while improving the distinguishability of defects in complex critical connection areas. Furthermore, by performing online inference calibration using an online calibration sample set, a calibrated deep learning recognition network is obtained. This network then forms a list of re-judged defect items from the boundary re-judgment sample set and incorporates it into the harness defect judgment conclusion, thereby improving the stability of judgment under changing operating conditions.

[0091] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent identification of wire harness defects based on deep learning, characterized in that, include: Collect an overall view and a view of the key connection area of ​​the wire harness to be inspected, and construct a priori topology diagram of the wire harness by combining the process configuration record; Based on the harness topology prior diagram, pose normalization and structural alignment are performed on the overall view and the key connection area view, and the target region is segmented to generate a topology-aligned image sequence. Based on a deep learning recognition network, deep learning joint representation and defect candidate recognition are performed on topology-aligned image sequences, and an initial list of defect candidates is output. Based on the prior diagram of the wire harness topology, the structural relationship of the initial list of defect candidates is reviewed, and the consistency verification conclusion is written into the defect candidates as the basis for judgment and the target area is corrected to generate a list of structural constraint defects. The list of structural constraint defects is used for online inference calibration and boundary sample re-judgment to generate a conclusion on the determination of wire harness defects.

2. The intelligent identification method for wire harness defects based on deep learning as described in claim 1, characterized in that, The steps for constructing the harness topology prior graph by combining process configuration records are as follows: An industrial area array camera is used to acquire an overall view and a view of the key connection area of ​​the wire harness to be inspected, and the exposure clarity quality is verified. Using product identification information from the overall view and key connection area view as search criteria, read the corresponding process configuration records, perform field parsing and extract terminal configuration relationships and branch connection relationships, and generate topology construction information; Based on the topology construction information and the structural outlines of the overall view and the key connection area view, node relationships and connection relationships are established, and a priori diagram of the harness topology is generated.

3. The intelligent identification method for wire harness defects based on deep learning as described in claim 1, characterized in that, The steps for performing pose normalization and structural alignment on the overall view and the key connection region view, and segmenting the target region to generate a topologically aligned image sequence are as follows: Based on the prior diagram of the wire harness topology, benchmark positioning and attitude parameter extraction are performed on the overall view and the key connection area view to obtain benchmark positioning information and attitude parameters; Based on the baseline positioning information and attitude parameters, the overall view and the critical connection area view are rotated and translated and scaled to generate a standardized overall view and critical connection area view. Based on the standardized overall view and key connection area view, structural alignment mapping is performed, and key connection areas are located and target regions are segmented to obtain a topology-aligned image sequence.

4. The intelligent identification method for wire harness defects based on deep learning as described in claim 1, characterized in that: The deep learning recognition network refers to a dual-branch recognition network that uses a convolutional backbone and a Swing Transformer global encoder in parallel, and is then connected to a decoupled detection head after feature pyramid fusion.

5. The intelligent identification method for wire harness defects based on deep learning as described in claim 3 or 4, characterized in that, The steps for performing deep learning joint representation and defect candidate identification on the topologically aligned image sequence, and outputting an initial list of defect candidates, are as follows: Using a convolutional backbone branch, multi-scale feature extraction is performed on the topologically aligned image sequence, and noise suppression and illumination normalization are completed to obtain local texture defect feature maps. Using the global encoding branch of the Swing Transformer, global structural relationship modeling is performed on the topologically aligned image sequence and long-range dependency features are extracted to obtain a global morphological relationship feature map; Based on the local texture defect feature map and the global morphological relationship feature map, feature pyramid fusion and scale alignment are performed to obtain a joint representation feature map; Based on the joint characterization feature map, defect category prediction and defect location prediction are performed by decoupling the detection head to obtain defect candidates; Based on the defect candidates, perform candidate purification and overlap reduction, and associate the defect candidates with the target region identifier of the topologically aligned image sequence to output an initial list of defect candidates.

6. The intelligent identification method for wire harness defects based on deep learning as described in claim 1, characterized in that, The steps for verifying the structural relationships of the initial list of candidate defects based on the harness topology prior diagram and correcting the target region identification to generate a list of structural constraint defects are as follows: Based on the harness topology prior diagram, each defect candidate in the initial defect candidate list is mapped to the corresponding target area identifier and topology node relationship, the topology mapping defect candidate is obtained and the connection relationship consistency check is performed to generate the consistency check conclusion; Based on the consistency verification conclusion, consistency filtering is performed on the topology mapping defect candidates and the target area identification is corrected to obtain the corrected defect candidates. At the same time, the defect candidates that need to be re-judged are marked. The revised defect candidates are compiled and their association with the target area identifiers in the harness topology prior diagram is maintained to generate a list of structural constraint defects.

7. The intelligent identification method for wire harness defects based on deep learning as described in claim 6, characterized in that, The process of reviewing the structural constraint defect assessment list, performing online inference calibration, and re-assessing boundary samples to generate wiring harness defect assessment conclusions is as follows: Calculate the defect confidence level of each defect candidate in the structural constraint defect judgment list, and use the defect candidates with defect confidence levels higher than the confidence level threshold as the online calibration sample set, while using the defect candidates that need to be re-judged as the boundary re-judgment sample set; Online inference calibration is performed on the deep learning recognition network based on the online calibration sample set to obtain the calibrated deep learning recognition network. Using the calibrated deep learning recognition network, the defect candidate recognition is performed again on the topology aligned image sequence corresponding to the boundary re-judgment sample set, and the structural constraint defect judgment list is corrected to obtain the re-judgment defect item list. The list of re-judged defects is then filled back into the corresponding entries of the list of structural constraint defects, while maintaining consistency in the target area identification, ultimately generating the harness defect judgment conclusion.

8. The intelligent identification method for wire harness defects based on deep learning as described in claim 7, characterized in that: Maintaining consistency of target area identifiers means that each defect candidate in the list of reassessed defect items continues to use the same target area identifier as the wiring harness topology prior diagram when it is merged into the structural constraint defect judgment list, without changing the correspondence of target area identifiers.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the deep learning-based intelligent identification method for wire harness defects as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the deep learning-based intelligent identification method for wire harness defects as described in any one of claims 1 to 8.

Images