A harness coating defect detection method and system based on visual recognition

Abstract

The application discloses a kind of line bundle coating defect detection method and system based on visual identification, it is related to visual detection technical field.The system is aimed at the difficult problem of defect positioning caused by bending, twisting and the change of illumination and shielding in detection for complex shape line bundle, by synchronously acquiring multi-view image sequence, using convolutional neural network to extract features and combining normalization processing, generate enhanced feature map that can capture line bundle spatial information and potential defects.The application tracks line bundle center axis by iterative fitting, constructs unified axis model using optical flow algorithm to fuse multi-view data, and calculates the radial offset of defect area based on the model to carry out preliminary positioning, finally uses multi-view geometric constraint to accurately reconstruct three-dimensional for defect.The application significantly improves the accuracy and robustness of line bundle defect detection in complex environment through multi-view information fusion and three-dimensional geometric optimization, and provides reliable spatial position data support for automation processing.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, and in particular to a method and system for detecting defects in wire harness wrapping in industrial automated production. Background Technology

[0002] As a core component in electrical systems that transmits signals and power, the quality of the wiring harness's sheathing directly affects the system's safety and reliability. In fields such as automotive, aerospace, and industrial automation, defects such as damage and cracking in the wiring harness sheathing can lead to short circuits or even system failures. Therefore, visual recognition-based defect detection technology has become crucial for ensuring product quality.

[0003] However, existing inspection methods often struggle to accurately capture the spatial location of defects when dealing with complex wire harness morphologies, limiting the accuracy and automation of inspections. Most current vision inspection systems rely on single-view image analysis, which is ill-suited to handling common bending and twisting patterns in wire harnesses, easily leading to the loss of spatial information and inaccurate defect localization. A deeper technical challenge lies in accurately determining the central axis of the wire harness as the localization reference. Since wire harnesses may exhibit irregular bends, inaccurate axis tracking will result in deviations in defect location calculations. Furthermore, inconsistencies often arise during multi-view image fusion due to lighting variations or occlusion, causing errors in the 3D reconstruction of defect locations.

[0004] Therefore, how to accurately track the central axis in complex wire harnesses and achieve accurate three-dimensional positioning of defects through multi-view image fusion has become a key issue in improving the performance of wire harness covering defect detection systems.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a method and system for detecting wire harness covering defects based on visual recognition, in order to solve the problem that the existing technology is difficult to accurately track the central axis and achieve accurate three-dimensional positioning of defects through multi-view image fusion when dealing with wire harnesses with complex shapes, thereby achieving high-precision and robust detection of wire harness covering defects.

[0007] This invention provides a method for detecting wire harness covering defects based on visual recognition, comprising: Acquire harness image data from multiple cameras taken from different angles, align timestamps using a preset synchronization mechanism, and obtain an initial multi-view image sequence; A convolutional neural network is used to extract features from the initial multi-view image sequence, and normalization processing is applied to address illumination changes and occlusion issues to obtain enhanced feature maps. The path of the central axis of the harness is traced based on the enhanced feature map. When the bending intensity exceeds a preset threshold, the axis curve is adjusted by iterative fitting to obtain stable axis path data. The optical flow algorithm is used to fuse axis path data from multiple perspectives, and the visible area for cross-validation is calculated to address the occlusion problem, resulting in a unified fused axis model. Defect candidate regions are extracted from the fused axis model. When the radial offset of the candidate region from the axis path is greater than a preset threshold, it is marked as a potential defect point, and the preliminary defect location coordinates are obtained. Based on the preliminary defect location coordinates, a 3D reconstruction was performed, and the offset value was optimized using multi-view geometric constraints to obtain accurate 3D defect location data. Determine whether the accurate 3D defect location data meets the location accuracy requirements. If the deviation value is lower than the preset threshold, output the final defect detection result.

[0008] In some optional embodiments, the step of using a convolutional neural network to extract features from the initial multi-view image sequence and normalizing the features to obtain an enhanced feature map includes: Gaussian filtering is used to smooth the noise preprocessing of the initial multi-view image sequence to obtain the preprocessed image sequence; A convolutional neural network is used to extract features from the preprocessed image sequence to generate an initial feature map. The channel weights of the initial feature map are calculated using the SE attention mechanism, normalized, and then multiplied with the initial feature map to obtain the enhanced feature map.

[0009] In some optional embodiments, the step of tracing the axis path based on the enhanced feature map and adjusting the axis path through iterative fitting to obtain a stable axis path representing a three-dimensional geometric reference includes: The line bundle features of the enhanced feature map are extracted using image processing techniques to obtain a clear line bundle feature map; Based on the clear feature map of the wire harness, the central axis of the wire harness is located using a path tracing method to obtain the initial axis path; The initial skeleton of the wire bundle is generated by identifying the linear structure of the initial axis path using the Hessian matrix. If the bending strength of the initial skeleton exceeds a preset strength threshold, the initial axis path is adjusted by iterative fitting using a cubic B-spline curve to obtain a stable axis path representing the three-dimensional geometric reference.

[0010] In some optional embodiments, the step of fusing the stable axis paths from multiple perspectives using an optical flow algorithm, and analyzing the visible regions of each perspective using a cross-validation method to obtain a unified fused axis model, includes: Based on the stable axis path, the motion vectors of each viewpoint are calculated using an optical flow algorithm to obtain the initial axis path set; Based on the initial set of axis paths, the visible area of ​​each viewpoint is calculated using a cross-validation method to determine the boundary of the occluded area; A weighted average method is used to fuse the visible regions of each viewpoint to generate a spatial mapping matrix between viewpoints; An optical flow algorithm is applied to the spatial mapping matrix to adjust the path deviation, resulting in an optimized set of axis paths; Based on the optimized set of axis paths, a unified fusion axis model is constructed.

[0011] In some optional embodiments, the step of extracting defect candidate regions from the unified fusion axis model and calculating the radial offset of the defect candidate regions from the stable axis path includes: Candidate defect regions are extracted from the unified fusion axis model using a region segmentation algorithm in image processing technology. The radial offset between the candidate defect region and the path of the stable axis is calculated using the Euclidean distance algorithm.

[0012] In some optional embodiments, the step of performing three-dimensional reconstruction based on the preliminary defect location coordinates and optimizing the radial offset using a multi-view geometric constraint model to determine accurate three-dimensional defect location data includes: Three-dimensional spatial equations are constructed using a multi-view geometric constraint model; The three-dimensional coordinates of the initial defect location and the radial offset are optimized using a bundle adjustment algorithm to obtain accurate three-dimensional defect location data that reflects the actual spatial location under the complex shape of the wire bundle.

[0013] In some optional embodiments, if the deviation between the precise three-dimensional defect location data and the reference location data is lower than a preset location deviation threshold, then the final detection result containing the defect location is output, including: The mean square error algorithm is used to calculate the deviation between the precise three-dimensional defect location data and the reference location data; If the deviation value is lower than the preset positioning deviation threshold, the final detection result containing the defect location will be output.

[0014] This invention provides a vision-based wire harness covering defect detection system, comprising: The data acquisition module is used to acquire an initial multi-view image sequence containing the bending and twisting features of the wire harness, captured by multiple cameras from multiple perspectives. The feature extraction module is used to extract features from the initial multi-view image sequence using a convolutional neural network, and to normalize the features to obtain an enhanced feature map. The axis tracking operation module is used to track the axis path according to the enhanced feature map, and adjust the axis path through iterative fitting to obtain a stable axis path representing the three-dimensional geometric reference. The data fusion operation module is used to fuse the stable axis paths from multiple perspectives using an optical flow algorithm, and to analyze the visible areas of each perspective using a cross-validation method to obtain a unified fused axis model. The radial offset calculation module is used to extract defect candidate regions from the unified fusion axis model and calculate the radial offset between the defect candidate regions and the stable axis path. The preliminary positioning operation module is used to mark the position coordinates of the defect candidate area as potential defect points when the radial offset is greater than a preset offset threshold, so as to obtain the preliminary defect position coordinates. The 3D reconstruction and optimization module is used to perform 3D reconstruction based on the preliminary defect location coordinates, and to optimize the radial offset using a multi-view geometric constraint model to determine accurate 3D defect location data. The result output module is used to output the final detection result containing the defect location if the deviation between the accurate three-dimensional defect location data and the reference location data is lower than a preset location deviation threshold.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention.

[0016] Compared to existing technologies, this invention offers the following advantages: Addressing the problem of accurately locating wire harness defects in complex conditions such as bending and torsion, this invention simultaneously acquires multi-view image sequences using multiple cameras, employs a convolutional neural network to extract features robust to illumination and occlusion, and precisely tracks and fuses these features to construct a unified three-dimensional model of the central axis. This approach provides a stable three-dimensional geometric benchmark for defect localization, fundamentally solving the core problem of positioning deviations caused by inaccurate benchmarks in existing technologies. Therefore, this invention significantly improves the accuracy and robustness of wire harness defect detection in complex environments, achieving precise three-dimensional localization and providing reliable spatial location data support for subsequent automated processing. Attached Figure Description

[0017] Figure 1 This is a schematic flowchart of a visual recognition-based wire harness covering defect detection method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a wire harness covering defect detection system based on visual recognition provided in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0019] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0020] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0021] This invention provides a visual recognition-based method for detecting defects in wire harness coverings, mainly comprising: S101, acquire an initial multi-view image sequence containing the bending and twisting features of the wire harness, captured by multiple cameras from multiple perspectives; S102, a convolutional neural network is used to extract features from the initial multi-view image sequence, and the features are normalized to obtain an enhanced feature map; S103, trace the axis path according to the enhanced feature map, and adjust the axis path through iterative fitting to obtain a stable axis path representing the three-dimensional geometric reference; S104, the stable axis paths from multiple perspectives are fused using an optical flow algorithm, and the visible areas of each perspective are analyzed using a cross-validation method to obtain a unified fused axis model; S105, Extract the defect candidate region from the unified fusion axis model, and calculate the radial offset between the defect candidate region and the stable axis path; S106, when the radial offset is greater than a preset offset threshold, the position coordinates of the defect candidate region are marked as potential defect points to obtain preliminary defect position coordinates; S107, Based on the preliminary defect location coordinates, perform three-dimensional reconstruction and use a multi-view geometric constraint model to optimize the radial offset to determine accurate three-dimensional defect location data; S108, if the deviation between the precise three-dimensional defect location data and the reference location data is lower than the preset location deviation threshold, then the final detection result including the defect location is output.

[0022] like Figure 1 This embodiment of a visual recognition-based method for detecting defects in wire harness coverings may specifically include: S101. Acquire an initial multi-view image sequence containing the bending and twisting features of the wire harness, captured by multiple cameras from multiple perspectives.

[0023] Multiple cameras synchronously acquire image data of the wire harness. A timestamp alignment mechanism is used to generate a time-consistent multi-view image sequence, ultimately yielding an initial multi-view image sequence containing the bending and twisting features of the wire harness. If the timestamp deviation exceeds a preset threshold, an interpolation algorithm is used to adjust the image frame time to obtain an aligned multi-view image sequence. For the aligned multi-view image sequence, optical flow is used to extract the bending and twisting features of the wire harness, verifying that the image sequence effectively contains the spatial morphological features of the wire harness, requiring no additional post-processing.

[0024] For example, multiple cameras can be used to acquire wire harness image data from multiple perspectives. Four industrial cameras with a resolution of 1920×1080 can be positioned at the top, bottom, left, and right of the wire harness in space, with an angle of 90 degrees between each camera to ensure full coverage of the wire harness. The cameras simultaneously capture images at a frame rate of 30 frames per second, using a PTP (Precise Time Protocol) synchronization mechanism to align camera timestamps to millisecond-level accuracy (error <1ms) via Ethernet. After image acquisition, the images are stored in JPEG format, with each frame containing approximately 2MB of data.

[0025] To align timestamps, a timestamp matching algorithm can be used: First, extract the UTC timestamp (format: year-month-day hour:minute:second:millisecond) embedded in the metadata of each camera image, and then calculate the time difference between images from multiple cameras at the same moment. (t1 and t2 are the timestamps of the two cameras, in units of: );like If the image is synchronized, it is considered synchronized; if there is a deviation ( Then, intermediate frames are generated using linear interpolation (e.g., when camera A is in...). Camera B in Image available, interpolation generation Alignment frames were used to generate an initial multi-view image sequence. Each sequence contained 4 synchronized images, with a sequence length of 1000 frames, covering the dynamic changes of the wire harness under complex shapes. To extract and verify the bending and twisting features of the wire harness, an image processing algorithm was used: first, each frame of the image was preprocessed, and Gaussian filtering was used to remove noise. Then, Canny edge detection (low threshold 50, high threshold 150) was used to extract the wire harness contour. Based on the contour point set, 50 sampling points evenly distributed on the contour were selected. ), using a cubic spline interpolation function: Where x is the independent variable, corresponding to the one-dimensional coordinates of the wire harness contour sampling point, and n is the total number of wire harness contour sampling points. The x-coordinate of the k-th sampling point after sorting. Let y be the y-coordinate of the k-th sampling point. For the k-th segment of the curve The first derivative value at that point, For the k-th segment of the curve at x k The second derivative value at the point is related to the numerical correlation coefficient. Let be the coefficient of the cubic term in the k-th segment. This function is used to fit the centerline, i.e. This ensures that the curve is continuous and twice differentiable. The bending characteristics (curvature) and torsional characteristics (torsion) of the wire harness are calculated based on the fitted centerline curve, where curvature ,unit: This reflects the degree of local curvature; Torque Based on the spatial mapping relationship of multi-view images, and the rate of change of the normal vector of the center line of adjacent frames, the method is used to analyze the spatial mapping relationship of multi-view images. Calculation, unit: ,in The angle between the normal vectors, The arc length reflects the degree of torsion.

[0026] For example, in a certain frame of an image, curvature Torque This indicates that the frame contains effective bending and twisting features. If more than 80% of the frames in the sequence... and The entire initial multi-view image sequence is deemed to meet the requirements, requiring no additional operations. Furthermore, through multi-view geometric reconstruction based on triangulation, the 3D coordinates of the line bundle are calculated from the contour points of the four views, generating a 3D point cloud (approximately...). (Points) are used to assist in verifying the effectiveness of features and ensure that the initial multi-view image sequence can accurately reflect the complex shape of the harness.

[0027] S102. Use a convolutional neural network to extract features from the initial multi-view image sequence, and normalize the features to obtain an enhanced feature map.

[0028] Gaussian filtering is used to smooth the noise preprocessing of the initial multi-view image sequence to obtain the preprocessed image sequence; A convolutional neural network is used to extract features from the preprocessed image sequence to generate an initial feature map. The channel weights of the initial feature map are calculated using the SE attention mechanism, normalized, and then multiplied with the initial feature map to obtain the enhanced feature map.

[0029] Gaussian filtering is used to smooth noise in the initial multi-view image sequence, resulting in a preprocessed image sequence. A convolutional neural network is then used to extract features from the preprocessed image sequence, generating an initial feature map to capture the spatial information of the wire bundle. The SE attention mechanism is used to calculate the channel weights of the initial feature map. To address illumination variations and occlusion issues, the normalized channel weights are multiplied by the initial feature map to obtain an enhanced feature map. The enhanced feature map is then used to reduce its dimensionality using pooling operations, preserving key information about the wire bundle structure and potential defect regions, generating a dimensionality-reduced feature map. If the pixel intensity in the dimensionality-reduced feature map exceeds a preset threshold, a region segmentation algorithm is used to determine potential defect regions, generating candidate defect regions. Morphological operations are then used to smooth the boundaries and remove noise, resulting in an optimized defect region. The final enhanced feature map contains both the spatial information of the wire bundle and potential defect regions.

[0030] For example, a convolutional neural network is used to extract features from the initial multi-view image sequence. First, the input multi-view image sequence is preprocessed. Assume the image sequence contains 5 views, each image has a resolution of 512×512 pixels, and is in RGB format. The preprocessing stage uses a noise smoothing algorithm, specifically a 3×3 Gaussian filter kernel (standard deviation...). To reduce image noise interference, a preprocessed image sequence is obtained. Subsequently, a convolutional neural network (CNN) is applied for feature extraction. ResNet-18 is selected as the base network, containing 18 convolutional layers with a kernel size of 3×3, a stride of 1, and padding of 1. The output feature map dimension is 512×16×16, generating an initial feature map to capture the spatial information of the line bundle. To address the issue of illumination variation, batch normalization is used to normalize the features of each layer. The calculation formula is as follows: ,in This is the batch average. For variance, To avoid division by zero, the range of eigenvalues ​​is compressed to [value range] after normalization. This effectively mitigates feature shifts caused by illumination variations. For the occlusion problem, a Squeeze-and-Excitation (SE) attention mechanism is used to calculate the channel weights of the initial feature map, using the following formula: ,in For global pooling features, For the weights of the fully connected layer, For the Sigmoid function, It is an activation function The normalized channel weights are multiplied by the initial feature map to highlight the response of defective regions, resulting in an enhanced feature map. This enhanced feature map is then reduced from 16×16 to 1×1 using global average pooling, generating a 512-dimensional reduced feature map that retains key information about the wire bundle structure and potential defective regions. When the pixel intensity in the reduced feature map exceeds a preset threshold (e.g., 0.6), a region segmentation algorithm (e.g., threshold segmentation) is used to determine potential defective regions and generate candidate defective regions. Morphological closing operations (3×3 structuring elements) are then used to smooth the boundaries of the candidate defective regions, eliminating noise regions with an area <5 pixels, resulting in optimized defective regions. The entire process is implemented using the deep learning framework PyTorch, with the Adam optimizer used during training. The learning rate is 0.001, the batch size is 32, and the iterations are performed for 100 epochs to ensure the stability and accuracy of feature extraction. In the final enhanced feature map, the pixel values ​​of the wire harness edges and defect areas (such as breaks or wear) are significantly enhanced. For example, the average pixel value of the defect area is increased to 0.85, while the average pixel value of the background area is reduced to 0.2. This indicates that the feature extraction effectively captures the spatial information of the wire harness and potential defect features. Furthermore, through Gaussian filtering preprocessing, batch normalization, and SE attention mechanism, the robustness to changes in illumination and local occlusion (such as occlusion area ≤ 64×64 pixels) is significantly improved.

[0031] S103. Trace the axis path according to the enhanced feature map, and adjust the axis path by iterative fitting to obtain a stable axis path representing the three-dimensional geometric reference.

[0032] The line bundle features of the enhanced feature map are extracted using image processing techniques to obtain a clear line bundle feature map; Based on the clear feature map of the wire harness, the central axis of the wire harness is located using a path tracing method to obtain the initial axis path; The initial skeleton of the wire bundle is generated by identifying the linear structure of the initial axis path using the Hessian matrix. If the bending strength of the initial skeleton exceeds a preset strength threshold, the initial axis path is adjusted by iterative fitting using a cubic B-spline curve to obtain a stable axis path representing the three-dimensional geometric reference.

[0033] First, image processing techniques are used to extract wire harness features from the enhanced feature map, filtering out irrelevant information to obtain a wire harness feature map with clear contours and prominent features. Based on this clear wire harness feature map, a path tracing method is used to locate the central axis that reflects the core position of the wire harness, obtaining a preliminary initial axis path. Then, the Hessian matrix is ​​used to analyze the initial axis path, accurately identifying the linear structures within it, thereby generating an initial skeleton that can characterize the overall shape of the wire harness. Finally, the bending strength of the initial skeleton is judged. If it exceeds a preset strength threshold, a cubic B-spline curve is used to iteratively fit and adjust the initial axis path, continuously optimizing the path shape, and finally obtaining an axis path that can serve as a three-dimensional geometric reference, with a stable and accurate shape.

[0034] For example, to trace the central axis path of the harness based on the enhanced feature map, the input 3D feature map is first preprocessed using image processing techniques. Assuming the feature map is a 128×128×128 voxel grayscale image, a 3D Gaussian filter (standard deviation) is then applied. Noise is smoothed to enhance the edge features of the wire harness. Next, Hessian matrix analysis is used to calculate the eigenvalues ​​of each voxel, and a threshold is set. To identify linear structures, an initial skeleton for the wire bundle is generated. Curvature is defined for bending morphology strength. , where T is the tangent vector. Let be the arc length, if If the bending morphology strength exceeds the threshold, iterative fitting is triggered. Iterative fitting uses cubic B-spline curves, with initial control points sampled from the skeleton points (one point for every five voxels). The control point positions are optimized using the least squares method, and the error function is... ,in For skeleton points, For spline curve points, iterate 10 times or the error Stop when the time is right. The final output axis path data is a three-dimensional coordinate sequence. The distance between each point is approximately 0.5mm This ensures the accuracy of the geometric datum. The analysis process verifies curvature continuity and the maximum rate of change of curvature. This ensures a smooth path. If the line bundles in the feature map are discontinuous, a graph cut algorithm (with weights based on grayscale gradients) is used to fill the gaps, ensuring path integrity. This method, through numerical constraints and algorithm optimization, forms a logically rigorous central axis tracing process, suitable for 3D modeling of line bundles.

[0035] S104. The stable axis paths from multiple perspectives are fused using an optical flow algorithm, and the visible areas of each perspective are analyzed using a cross-validation method to obtain a unified fused axis model.

[0036] Based on the stable axis path, the motion vectors of each viewpoint are calculated using an optical flow algorithm to obtain the initial axis path set; Based on the initial set of axis paths, the visible area of ​​each viewpoint is calculated using a cross-validation method to determine the boundary of the occluded area; A weighted average method is used to fuse the visible regions of each viewpoint to generate a spatial mapping matrix between viewpoints; An optical flow algorithm is applied to the spatial mapping matrix to adjust the path deviation, resulting in an optimized set of axis paths; Based on the optimized set of axis paths, a unified fusion axis model is constructed.

[0037] First, motion vectors from each viewpoint are calculated using an optical flow algorithm to obtain an initial set of axis paths. Cross-validation is then used to analyze the visible region of each viewpoint, determining the boundaries of occlusion areas and obtaining the visible region distribution. If occlusion areas exist within the visible region distribution, the dynamic changes of these occlusion areas are calculated through time-series comparison of multi-view data, yielding the dynamic visible region. Next, the spatial information from each viewpoint is fused using the dynamic visible region to generate a spatial mapping matrix between viewpoints, resulting in a unified set of spatial information. Using this unified spatial information set, an optical flow algorithm is applied to optimize the axis paths, adjusting path deviations to obtain an optimized set of axis paths. Finally, based on the optimized set of axis paths, a unified fused axis model is constructed, integrating the spatial information from multiple viewpoints to generate the final fused axis model.

[0038] For example, by fusing multi-view axis path data using optical flow algorithms, the pixel displacement field of each viewpoint is first calculated using a dense optical flow algorithm (such as the Farneback algorithm). Assuming the input consists of video frames from three viewpoints with a resolution of 1280x720 and a frame rate of 30fps, feature points of the axis path in each frame are selected.

[0039] For example, the SIFT algorithm is used to extract 1000 keypoints (with a coordinate accuracy of 0.1 pixels). Optical flow tracing is used to calculate the displacement vector of each point between adjacent frames, with the displacement error controlled within 0.5 pixels. To address the occlusion problem, a cross-validation method is employed to construct visibility masks for each viewpoint. Assume viewpoint 1 is in a certain area... When an area is occluded, the pixel consistency of the region is detected using optical flow data from other viewpoints (such as viewpoints 2 and 3) (with a similarity threshold set to 0.8), generating a visibility weight map with a weight range of [0,1]. Next, a weighted average method is used to fuse the multi-viewpoint data and calculate a unified axis path. Assume the axis points of viewpoints 1, 2, and 3 are respectively... The fusion formula is ,in Visibility weights are used. After fusion, the spatial coordinate error of the axis points is reduced from the initial 2.0 pixels to 0.3 pixels. To reduce reconstruction error, the least squares method is used to optimize the axis model, assuming the model is a quadratic curve. By fitting 1000 fusion points and optimizing parameters a, b, and c, the sum of squared errors decreased from 10.5 to 1.2. Analysis shows that the visibility weight of occluded areas effectively reduced the fusion error, and spatial information integration improved the model's robustness. If noise exists in the data, Kalman filtering can be introduced to smooth the trajectory, further reducing the error to 0.2 pixels. This method significantly improves the accuracy of axis reconstruction through multi-view complementarity.

[0040] S105. Extract the defect candidate region from the unified fusion axis model and calculate the radial offset between the defect candidate region and the stable axis path.

[0041] Candidate defect regions are extracted from the unified fusion axis model using a region segmentation algorithm in image processing technology. The radial offset between the candidate defect region and the path of the stable axis is calculated using the Euclidean distance algorithm.

[0042] Candidate defect regions are obtained by fusing an axis model. Region boundaries are extracted using a region segmentation algorithm, resulting in a set of candidate defect regions. For each candidate defect region, the radial offset from the axis path is calculated, and the Euclidean distance algorithm is used to determine the offset value. If the radial offset is greater than a preset threshold, the corresponding region is marked as a potential defect point, resulting in a set of potential defect points. Based on the potential defect point set, defect location coordinates are generated using a reference datum, and coordinate transformation algorithms are used to determine the coordinate values. For the defect location coordinates, the geometric consistency between the coordinates and the axis path is verified. If the consistency verification passes, the coordinates are confirmed as valid, resulting in a set of valid defect coordinates. Based on the valid defect coordinate set, cluster analysis is performed, and the K-means algorithm is used to divide the defect point clusters, resulting in a defect point cluster distribution. For the defect point cluster distribution, spatial distribution characteristics of the defect points are generated, and a grid partitioning algorithm is used to determine the distribution range.

[0043] For example, when extracting candidate defect regions from a fused axis model, the axis model is first analyzed using point cloud data processing technology. Assume the point cloud data contains 1 million points with coordinates ranging from [missing information]. The axis is generated by a fitting algorithm (such as the least squares method), with axis points at (x, y, z), and the axis path is represented by parametric equations. ,in Using a stereomicroscope algorithm, the point cloud was divided into 1000 grids (each grid with a side length of 0.1 meters). The Euclidean distance from each point within a grid to the axis was calculated using the following formula: If the average distance within a grid cell is greater than the threshold of 0.5 meters, it is marked as a candidate region. For example, if the center point of a grid cell is (5, 0.6, 0), and the corresponding axis point is (5, 0.25, 0), the distance is calculated. The first grid point (6, 1.2, 0) is not marked as a candidate region; the other grid center point (6, 0.36, 0) corresponds to the axis point (6, 0.36, 0), with a distance d = 0.84. 0.5, marked as a candidate region. Radial offset analysis is performed on the candidate region, calculating the offset based on the normal vector of the axis path. The normal vector is calculated using its derivative. The result after normalization. For example, normal vector The offset is the projected distance from the point to the axis, calculated as follows: .

[0044] S106. When the radial offset is greater than a preset offset threshold, the position coordinates of the defect candidate region are marked as potential defect points to obtain preliminary defect position coordinates.

[0045] If the offset is greater than the preset threshold of 0.7 meters, it is marked as a potential defect point. The coordinates (6, 1.2, 0) meet the condition, and their coordinates are recorded. Finally, preliminary defect location coordinates are generated, based on the axis reference datum, and represented as (t, offset), such as (6, 0.84). The above calculation is repeated for all candidate areas to obtain a defect point set, stored in JSON format: [{"t":6,"offset":0.84}, ...]. To ensure accuracy, k-means clustering (k=5) is used to classify the defect points and analyze the defect distribution pattern. The mean offset of the center point is 0.75 meters, and the standard deviation is 0.1 meters, indicating that the defects are concentrated in a specific segment of the axis, for reference in subsequent process optimization.

[0046] S107. Based on the preliminary defect location coordinates, perform three-dimensional reconstruction and use a multi-view geometric constraint model to optimize the radial offset to determine accurate three-dimensional defect location data.

[0047] Three-dimensional spatial equations are constructed using a multi-view geometric constraint model; The three-dimensional coordinates of the initial defect location and the radial offset are optimized using a bundle adjustment algorithm to obtain accurate three-dimensional defect location data that reflects the actual spatial location under the complex shape of the wire bundle.

[0048] The initial position coordinates of the wire harness are obtained. Using a multi-view image acquisition device, two-dimensional coordinate data of the wire harness defect are extracted from multiple angles to obtain the initial position coordinate set. A multi-view geometric constraint model is used to construct three-dimensional geometric constraint equations based on the initial position coordinate set, determining the preliminary three-dimensional model of the wire harness defect. Through constraint adjustment algorithms, the offsets in the preliminary three-dimensional model are optimized by combining multi-view geometric constraints, resulting in an optimized three-dimensional model. If the optimized three-dimensional model does not match the preset features of the complex shape of the wire harness, the parameters of the geometric constraint model are adjusted based on complex shape analysis, and a new three-dimensional model is generated. Based on the optimized three-dimensional model, the spatial position data of the wire harness defect is extracted to generate accurate three-dimensional defect location data. By comparing the deviation between the three-dimensional defect location data and the initial position coordinates, the degree of conformity of the defect position accuracy is judged, obtaining the final location data. For the final location data, complex shape analysis is used to verify the spatial position consistency under the complex shape of the wire harness, generating verified three-dimensional defect location data.

[0049] For example, in the 3D reconstruction of defect locations in complex wire harness shapes, firstly, 2D image data of the wire harness is acquired using a multi-view imaging system. Assume images are acquired from three viewpoints (0°, 45°, and 90°) with a pixel resolution of 1920×1080. The Canny edge detection algorithm is used to extract the wire harness contour, with high and low thresholds set to 150 and 50 respectively, to obtain the preliminary defect coordinates for each viewpoint. For example, the defect center point detected by viewpoint 1 is (500, 300), viewpoint 2 is (520, 310), and viewpoint 3 is (480, 290). Based on these coordinates, a 3D reconstruction model is constructed using multi-view geometric constraints. A bundle adjustment algorithm is used to optimize the camera's intrinsic and extrinsic parameters. Assuming a camera focal length of 50mm and a pixel size of 5.5μm, the initially estimated 3D coordinates of the defect are (x0, y0, z0) = (10.5, 8.2, 15.7)mm. Next, the intersection points of light rays from each viewpoint were calculated using triangulation, and the offset value was iteratively optimized. A maximum of 100 iterations and a convergence threshold of 0.001 mm were set, resulting in optimized 3D coordinates (x, y, z) = (10.52, 8.25, 15.68) mm. The analysis process included calculating the reprojection error, assuming errors of 0.12, 0.15, and 0.09 pixels for the three viewpoints, all less than the threshold of 0.2 pixels, to verify positioning accuracy. For the complex shape of the wire harness, a B-spline curve was introduced to fit the harness path, with 10 control points and the fitting error controlled within 0.05 mm to ensure the defect location data reflected the actual spatial position. Finally, accurate 3D defect coordinates were output, which, combined with the wire harness morphology model, were used for subsequent automated inspection system planning of repair paths. The path error was controlled within 0.1 mm, forming a closed-loop business logic.

[0050] S108. If the deviation between the precise three-dimensional defect location data and the reference location data is lower than the preset location deviation threshold, the final detection result containing the defect location is output.

[0051] The mean square error algorithm is used to calculate the deviation between the precise three-dimensional defect location data and the reference location data; If the deviation value is lower than the preset positioning deviation threshold, the final detection result containing the defect location will be output.

[0052] To acquire 3D defect location data, raw data containing axial and radial information is collected from sensors or scanning equipment to obtain a location dataset. Data preprocessing methods are used to denoise and format the location dataset, resulting in a cleaned location dataset. The deviation between the cleaned location dataset and reference data is calculated using a mean squared error algorithm to determine the deviation value. If the deviation value is below a preset threshold, the positioning accuracy is considered to meet the requirements, and the defect detection result, including axial and radial position data, is output. If the deviation value is higher than or equal to the preset threshold, a secondary analysis is performed on the cleaned location dataset, using a clustering algorithm to identify outliers and obtain the outlier distribution. Based on the outlier distribution, the 3D defect location parameters are adjusted, and location data is re-acquired to obtain an updated location dataset. Using the updated location dataset, the deviation calculation and threshold determination are repeated to output the final axial and radial position data.

[0053] For example, receiving 3D defect location data, assuming the input data is the position of the defect point in the workpiece coordinate system (x=10.5mm, y=20.3mm, z=15.2mm), and setting the positioning accuracy threshold to 0.5mm, the deviation is calculated using the Euclidean distance formula. ,in For the standard position, assume it is (10.0mm, 20.0mm, 15.0mm). Substitute the data... Comparing the deviation value of 0.616mm with the threshold of 0.5mm, it was found that the deviation exceeded the threshold, indicating that the positioning accuracy did not meet the standard. At this time, the system automatically triggered a calibration algorithm, using the least squares method to optimize the positioning data. Assuming that the new coordinates are obtained after calibration (x=10.2mm, y=20.1mm, z=15.1mm), the deviation is recalculated: The deviation of 0.245mm is below the threshold of 0.5mm, meeting the accuracy requirements. The final output defect detection results include the axial position z=15.1mm and the radial position... The results are encapsulated in JSON format as {"axial":15.1,"radial":22.53} and transmitted to the subsequent processing module for defect repair path planning. The entire process is implemented through automated algorithms, requiring no manual intervention, ensuring rigorous logic and data traceability.

[0054] like Figure 2 As shown, this embodiment of the invention also provides a visual recognition-based wire harness covering defect detection system. The system may include a processor, a memory, and multiple industrial cameras in terms of hardware. When the processor executes the program stored in the memory, it implements the steps of the above-described method. Logically, the system may include: The data acquisition module M301 uses multiple industrial cameras (e.g., four 1920×1080 resolution cameras) and a PTP (Precise Time Protocol) synchronization mechanism to acquire images of the wire harness from different angles in real time. After acquiring the data, the module precisely aligns the images from each viewpoint using a timestamp matching algorithm, ensuring a time error of less than 1 millisecond. For example, it uses Canny edge detection (low threshold 50, high threshold 150) and cubic spline interpolation algorithms to extract the wire harness contour and centerline from the aligned images and calculate two-dimensional morphological features such as curvature κ and torsion τ. This module outputs an initial multi-view image sequence, which serves as the raw data foundation for all subsequent three-dimensional analysis and defect identification.

[0055] The feature extraction module M302 receives image sequences from M301 and performs deep feature extraction on the line bundles using a pre-trained convolutional neural network (such as ResNet-18). This module inputs each image into the network and generates initial feature maps through multiple convolutional operations. To address uneven illumination and occlusion issues, it employs batch normalization to standardize the features of each layer and can introduce random occlusion data augmentation during training to improve robustness. Furthermore, through the SE attention mechanism, the module can automatically learn and enhance the weights of feature channels related to defects. The final enhanced feature map (e.g., the average pixel value in the defect region is increased to 0.85) provides high-quality input for subsequent geometric analysis.

[0056] The axis tracking module M303 first identifies the linear structure using Hessian matrix analysis based on the enhanced feature map provided by M302, generating an initial 3D skeleton for the line bundle. Next, the module calculates the curvature κ of the skeleton. When the curvature exceeds a preset threshold (e.g., ...), ... When the error is less than 0.01 mm, the iterative fitting process is triggered. It uses cubic B-spline curves and optimizes the control point positions using the least squares method until the error is less than 0.01 mm, ultimately generating a smooth and accurate centerline path data for each viewpoint.

[0057] The data fusion module M304 fuses the axis data from multiple viewpoints output by M303 to construct a globally unified model. This module employs a dense optical flow algorithm to track the motion of axis feature points from each viewpoint. When occlusion is detected (e.g., cross-validation reveals that pixel consistency in a certain area is below 0.8), the module reduces the visibility weight of the data in that area. Finally, through a weighted average, all visible data is fused into a unified 3D axis model with less error (e.g., spatial coordinate error reduced from 2.0 pixels to 0.3 pixels).

[0058] The radial offset calculation module M305 and the preliminary positioning operation module M306 perform geometric analysis based on the unified axis model generated by M304 to screen potential defect areas. It treats the wire harness surface data as a point cloud and calculates the radial offset (Euclidean distance d) of each point to the central axis. When the average radial offset of a region exceeds a preset threshold (e.g., 0.7 meters), the region is marked as a defect candidate area, and its preliminary position coordinates based on the axis reference system are recorded (e.g., {"t":6,"offset":0.84}).

[0059] The 3D reconstruction and optimization module M307 and the result output module M308 convert the preliminary positioning results into precise physical coordinates and verify them. Module M307 uses a bundle adjustment algorithm, combined with multi-view geometric constraints, to iteratively optimize the 3D coordinates of the defect point until the reprojection error is less than a threshold (e.g., 0.2 pixels), obtaining accurate 3D positioning data. Subsequently, the accuracy verification and output module M308 compares this data with the standard reference position, calculates the deviation value d, and if d is less than the accuracy threshold (e.g., 0.5 mm), it outputs the final detection result including axial and radial positions, forming a complete closed-loop detection process.

[0060] The vision-based wire harness covering defect detection system provided in this embodiment acquires complete spatial information through simultaneous acquisition by multiple cameras, performs robust feature extraction based on deep learning, accurately tracks and fuses the data to construct a unified central axis model, identifies defects through geometric analysis based on this model, and finally achieves high-precision three-dimensional positioning by using multi-view geometric constraints. This significantly improves the accuracy and robustness of wire harness defect detection in complex environments and provides a reliable quality control method for industrial automated production.

[0061] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for detecting defects in wire harness covering based on visual recognition, characterized in that, include: Acquire an initial multi-view image sequence containing features of wire harness bending and twisting, captured from multiple cameras from multiple perspectives; A convolutional neural network is used to extract features from the initial multi-view image sequence, and the features are normalized to obtain an enhanced feature map. The axis path is traced based on the enhanced feature map, and the axis path is adjusted by iterative fitting to obtain a stable axis path representing the three-dimensional geometric reference. The stable axis paths from multiple perspectives are fused using an optical flow algorithm, and the visible areas of each perspective are analyzed using a cross-validation method to obtain a unified fused axis model. Defect candidate regions are extracted from the unified fusion axis model, and the radial offset between the defect candidate regions and the stable axis path is calculated. When the radial offset is greater than a preset offset threshold, the position coordinates of the defect candidate region are marked as potential defect points to obtain preliminary defect position coordinates; Based on the preliminary defect location coordinates, a three-dimensional reconstruction is performed, and the radial offset is optimized using a multi-view geometric constraint model to determine accurate three-dimensional defect location data. If the deviation between the precise three-dimensional defect location data and the reference location data is lower than a preset location deviation threshold, the final detection result containing the defect location will be output.

2. The method for detecting wire harness covering defects based on visual recognition according to claim 1, characterized in that, The step of using a convolutional neural network to extract features from the initial multi-view image sequence and then normalizing the features to obtain an enhanced feature map includes: Gaussian filtering is used to smooth the noise preprocessing of the initial multi-view image sequence to obtain the preprocessed image sequence; A convolutional neural network is used to extract features from the preprocessed image sequence to generate an initial feature map. The channel weights of the initial feature map are calculated using the SE attention mechanism, normalized, and then multiplied with the initial feature map to obtain the enhanced feature map.

3. The method for detecting wire harness covering defects based on visual recognition according to claim 1, characterized in that, The step of tracing the axis path based on the enhanced feature map and adjusting the axis path through iterative fitting to obtain a stable axis path representing a three-dimensional geometric reference includes: The line bundle features of the enhanced feature map are extracted using image processing techniques to obtain a clear line bundle feature map; Based on the clear feature map of the wire harness, the central axis of the wire harness is located using a path tracing method to obtain the initial axis path; The initial skeleton of the wire bundle is generated by identifying the linear structure of the initial axis path using the Hessian matrix. If the bending strength of the initial skeleton exceeds a preset strength threshold, the initial axis path is adjusted by iterative fitting using a cubic B-spline curve to obtain a stable axis path representing the three-dimensional geometric reference.

4. The method for detecting wire harness covering defects based on visual recognition according to claim 1, characterized in that, The process of fusing stable axis paths from multiple perspectives using an optical flow algorithm, and then analyzing the visible regions of each perspective using cross-validation to obtain a unified fused axis model, includes: Based on the stable axis path, the motion vectors of each viewpoint are calculated using an optical flow algorithm to obtain the initial axis path set; Based on the initial set of axis paths, the visible area of ​​each viewpoint is calculated using a cross-validation method to determine the boundary of the occluded area; A weighted average method is used to fuse the visible regions of each viewpoint to generate a spatial mapping matrix between viewpoints; An optical flow algorithm is applied to the spatial mapping matrix to adjust the path deviation, resulting in an optimized set of axis paths; Based on the optimized set of axis paths, a unified fusion axis model is constructed.

5. The method for detecting wire harness covering defects based on visual recognition according to claim 1, characterized in that, The step of extracting defect candidate regions from the unified fusion axis model and calculating the radial offset of the defect candidate regions from the stable axis path includes: Candidate defect regions are extracted from the unified fusion axis model using a region segmentation algorithm in image processing technology. The radial offset between the candidate defect region and the path of the stable axis is calculated using the Euclidean distance algorithm.

6. The method for detecting wire harness covering defects based on visual recognition according to claim 1, characterized in that, The process of performing 3D reconstruction based on the preliminary defect location coordinates and optimizing the radial offset using a multi-view geometric constraint model to determine accurate 3D defect location data includes: Three-dimensional spatial equations are constructed using a multi-view geometric constraint model; The three-dimensional coordinates of the initial defect location and the radial offset are optimized using a bundle adjustment algorithm to obtain accurate three-dimensional defect location data that reflects the actual spatial location under the complex shape of the wire bundle.

7. The method for detecting wire harness covering defects based on visual recognition according to claim 1, characterized in that, If the deviation between the precise three-dimensional defect location data and the reference location data is lower than a preset location deviation threshold, then the final detection result containing the defect location is output, including: The mean square error algorithm is used to calculate the deviation between the precise three-dimensional defect location data and the reference location data; If the deviation value is lower than the preset positioning deviation threshold, the final detection result containing the defect location will be output.

8. A visual recognition-based wire harness covering defect detection system, characterized in that, include: The data acquisition module is used to acquire an initial multi-view image sequence containing the bending and twisting features of the wire harness, captured by multiple cameras from multiple perspectives. The feature extraction module is used to extract features from the initial multi-view image sequence using a convolutional neural network, and to normalize the features to obtain an enhanced feature map. The axis tracking operation module is used to track the axis path according to the enhanced feature map, and adjust the axis path through iterative fitting to obtain a stable axis path representing the three-dimensional geometric reference. The data fusion operation module is used to fuse the stable axis paths from multiple perspectives using an optical flow algorithm, and to analyze the visible areas of each perspective using a cross-validation method to obtain a unified fused axis model. The radial offset calculation module is used to extract defect candidate regions from the unified fusion axis model and calculate the radial offset between the defect candidate regions and the stable axis path. The preliminary positioning operation module is used to mark the position coordinates of the defect candidate area as potential defect points when the radial offset is greater than a preset offset threshold, so as to obtain the preliminary defect position coordinates. The 3D reconstruction and optimization module is used to perform 3D reconstruction based on the preliminary defect location coordinates, and to optimize the radial offset using a multi-view geometric constraint model to determine accurate 3D defect location data. The result output module is used to output the final detection result containing the defect location if the deviation between the accurate three-dimensional defect location data and the reference location data is lower than a preset location deviation threshold.

Images