High-speed wire harness welding alignment method and system based on multi-modal vision

Abstract

The present application relates to the technical field of wire harness welding visual alignment, a high-speed wire harness welding alignment method and system based on multi-modal vision, comprising: confirming a wire harness welding alignment environment based on a wire harness welding alignment instruction, performing image acquisition on the welding terminal based on a visual information acquisition unit to obtain a multi-modal data set, performing data preprocessing on the multi-modal data set, obtaining a multi-modal fusion target function based on an ideal model, a two-dimensional feature point set, a three-dimensional feature point set and a key height feature point set, and calculating an optimal rigid body transformation based on the multi-modal fusion target function, calculating a translation delay compensation number based on a delay data set and a motion control unit, and performing wire harness welding based on the welding execution unit, the translation delay compensation number and the pose deviation. The present application can improve the precision, speed, stability and automation level of high-speed wire harness welding alignment.

Description

Technical Field

[0001] This invention relates to the field of visual alignment technology for wire harness welding, and in particular to a high-speed wire harness welding alignment method and system based on multimodal vision. Background Technology

[0002] With the widespread application of multimodal vision technology in high-speed wire harness welding, and the stringent requirements for welding quality in precision manufacturing scenarios such as new energy vehicles, 5G communications, and energy storage systems, higher demands are being placed on the synergistic improvement of the alignment accuracy, response speed, and welding consistency of high-speed wire harness welding.

[0003] Existing alignment technologies have significant limitations: traditional mechanical fixture hard positioning and single-camera vision alignment methods are inefficient and struggle to handle complex scenarios such as wire bending, terminal reflection, and welding fumes under high-speed transmission conditions; while conventional single-modal vision alignment technologies mostly focus on the static positioning of single terminal features, lacking the fusion and utilization of multimodal visual information (such as visible light, infrared, and 3D depth information). Therefore, there is an urgent need for a high-speed wire harness welding alignment method and system based on multimodal vision that can integrate multimodal visual information fusion, high-speed dynamic pose accurate recognition, and real-time compensation for alignment deviations. Summary of the Invention

[0004] This invention provides a high-speed wire harness welding alignment method and system based on multimodal vision, the main purpose of which is to improve the accuracy, speed, stability and automation level of high-speed wire harness welding alignment.

[0005] To achieve the above objectives, the present invention provides a high-speed wire harness welding alignment method based on multimodal vision, comprising: The system confirms receipt of the wire harness welding alignment command and confirms the wire harness welding alignment environment based on the command. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. The visual information acquisition unit acquires images of the welding terminal to obtain a multimodal dataset. The multimodal dataset is preprocessed to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set. Obtain an ideal model, and based on the ideal model, a two-dimensional feature point set, a three-dimensional feature point set, and a key height feature point set, obtain a multimodal fusion objective function, calculate the optimal rigid body transformation based on the multimodal fusion objective function, and calculate the pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose. Acquire a delay dataset, calculate translation delay compensation based on the delay dataset and motion control unit, and perform wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain a welded wire harness; The welding harness is subjected to quality analysis to obtain evaluation results, which include excessive deviation or no deviation. If the evaluation result is no deviation, the welding harness is identified as a precision welding harness. Based on the precision welding harness, high-speed welding alignment of the welding terminals is achieved.

[0006] Optionally, the step of acquiring images of the welding terminal based on the visual information acquisition unit to obtain a multimodal dataset includes: Obtain the welding station; when the welding terminal arrives at the welding station, obtain a visual information acquisition command. The visual information acquisition unit receives the visual information acquisition command, wherein the visual information acquisition unit includes a 2D image acquisition device, a 3D visual acquisition device, and a line laser contour acquisition device, and controls the visual information acquisition unit to perform the following operations based on the visual information acquisition command: The welding terminal is acquired using the 2D image acquisition device to obtain a two-dimensional image of the terminal. The 3D vision acquisition device acquires images of the welding terminal to obtain an original depth image. Based on the pre-built 3D reconstruction algorithm, cloud coordinates are constructed from the original depth image to obtain a 3D point cloud coordinate set. The line laser profile acquisition device performs line laser scanning along a preset welding path to obtain a one-dimensional height profile of the welding terminal. By summarizing the two-dimensional images, three-dimensional point cloud coordinate sets, and one-dimensional height contours of the terminals, a multimodal dataset is obtained.

[0007] Optionally, the step of preprocessing the multimodal dataset to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour includes: A unified time standard is obtained based on a pre-built system clock. The two-dimensional image, three-dimensional point cloud coordinate set, and one-dimensional height profile of the terminal are time-calibrated based on the unified time standard to obtain a calibrated two-dimensional image, a calibrated three-dimensional point cloud coordinate set, and a calibrated one-dimensional height profile. The calibrated 2D image is denoised based on a pre-built image filtering algorithm to obtain a denoised 2D image. The contrast of the denoised 2D image is then enhanced to obtain a pre-processed 2D image. For each calibration 3D point cloud coordinate in the calibration 3D point cloud coordinate set, the following operation is performed: Based on the calibrated 3D point cloud coordinates, obtain multiple surrounding 3D point cloud coordinates, calculate multiple Euclidean distances based on the multiple surrounding 3D point cloud coordinates, and calculate the mean and standard deviation of the distances based on the multiple Euclidean distances. Based on the mean and standard deviation of the distance, the mean distance and the mean standard deviation of the overall distance are calculated. The mean distance and the standard deviation of each calibration 3D point cloud coordinate in the calibration 3D point cloud coordinate set are compared with the mean distance and the mean standard deviation of the overall distance. If the mean distance is greater than the mean distance and the standard deviation is greater than the mean standard deviation of the overall distance, the calibration 3D point cloud coordinates are removed to obtain a preprocessed 3D coordinate set. The calibrated one-dimensional height profile is smoothed based on a pre-built smoothing algorithm to obtain a pre-processed height profile.

[0008] Optionally, the step of extracting features from the preprocessed 2D image, preprocessed 3D coordinate set, and preprocessed height contour based on the multimodal feature extraction unit to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set includes: Edge detection is performed on the preprocessed 2D image based on a multimodal feature extraction unit and a pre-constructed two-dimensional feature detection algorithm to obtain two-dimensional contour edges. Feature points are then extracted from the two-dimensional contour edges to obtain two-dimensional feature points. Multiple principal curvatures are calculated based on a pre-constructed surface fitting algorithm and a pre-processed 3D coordinate set. The multiple principal curvatures are then filtered based on a preset curvature threshold, and a set of 3D feature points is obtained based on the pre-processed 3D coordinates corresponding to the filtered principal curvatures. The pre-built inflection point detection algorithm is used to detect inflection points in the preprocessed height contour to obtain a set of key height feature points.

[0009] Optionally, the step of obtaining a multimodal fusion objective function based on the ideal model, the two-dimensional feature point set, the three-dimensional feature point set, and the key height feature point set, and calculating the optimal rigid body transformation based on the multimodal fusion objective function, includes: Based on the ideal model, an ideal three-dimensional coordinate set is obtained. Based on the two-dimensional feature point set and the ideal three-dimensional coordinate set, feature point matching is performed to obtain a two-dimensional to three-dimensional feature point set, wherein the two-dimensional to three-dimensional feature point set includes M two-dimensional to three-dimensional feature points. Based on the ideal model, an ideal model point set and an ideal height value set are obtained. A multimodal fusion objective function is constructed based on the aforementioned 2D-to-3D feature point set, ideal model point set, ideal height value set, 2D feature point set, 3D feature point set, and key height feature point set. The 3D feature point set includes L 3D feature points, and the key height feature point set includes P key height feature points. The multimodal fusion objective function is shown below: in, Indicates the target score. Represents the rotation matrix. Represents the translation vector. Represents the pixel scaling factor. Represents the camera projection model. This represents the camera intrinsic parameter matrix. Represents the first digit in the set of 2D to 3D feature points. Two-dimensional to three-dimensional feature points Represents the first feature point in the two-dimensional feature point set. Two-dimensional feature points, Represents the 3D feature point set. Three-dimensional feature points, Let j represent the j-th ideal model point in the ideal model point set. Represents component functions, Represents the first key height feature point in the set. Key height feature points, Represents the set of ideal height values. An ideal height value, , and Indicates the weighting coefficient. , and Indicates subscript, Represents norm operations; The rotation matrix and translation vector in the multimodal fusion objective function are iterated until the objective score is less than a preset convergence threshold. Then, the rotation matrix and translation vector are integrated to obtain the optimal rigid body transformation, wherein the optimal rigid body transformation includes the optimal rotation matrix and the optimal translation vector.

[0010] Optionally, the calculation of pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose includes: Based on the ideal pose, obtain the ideal rotation matrix and the ideal translation vector; The rotation deviation is calculated based on the optimal rotation matrix and the ideal rotation matrix, and the displacement deviation is calculated based on the ideal translation vector and the optimal translation vector. The positional deviation is obtained by summing the rotational and displacement deviations.

[0011] Optionally, the step of acquiring the delay dataset and calculating the translation delay compensation number based on the delay dataset and the motion control unit includes: A delay dataset is obtained based on the motion control unit, wherein the delay dataset includes visual processing delay, communication delay and robot response delay, and the total delay time is calculated based on the delay dataset; The terminal transmission line speed and terminal transmission acceleration are obtained based on the motion control unit. The translational delay compensation number is calculated based on the total delay time, terminal transmission line speed, and terminal transmission acceleration, using the following formula: in, This represents the translation delay compensation number. Indicates the total delay time. Indicates transmission acceleration. This indicates the terminal transmission line speed.

[0012] Optionally, the step of welding the wire harness based on the welding execution unit, the translation delay compensation number, and the pose deviation to obtain the welded wire harness includes: The compensated displacement is calculated based on the translation delay compensation number and the displacement deviation in the pose deviation. The initial welding position is obtained based on the pre-constructed welding gun, and the initial welding position is adjusted based on the compensated displacement and rotation deviation to obtain the adjusted welding position. The welding execution unit controls the welding gun to weld the welding terminal at the adjusted welding position to obtain a welding wire harness.

[0013] Optionally, the quality analysis of the welding wire harness to obtain the evaluation results includes: Based on the visual information acquisition unit, images of the welding wire harness are acquired to obtain images after welding. The image after welding is segmented to obtain the welding point region. The pixel center coordinates are obtained based on the welding point region. The physical position deviation is calculated based on the pixel center coordinates and the preset ideal position. The area of ​​the welding region is calculated based on the welding point region. The physical position deviation and the area of ​​the welding area are compared with the preset position deviation threshold and the preset area of ​​the welding area, respectively. If the physical position deviation is less than the position deviation threshold and the area of ​​the welding area is within the area of ​​the welding area, the evaluation result is confirmed as no deviation; otherwise, the evaluation result is confirmed as excessive deviation.

[0014] To achieve the above objectives, the present invention also provides a high-speed wire harness welding alignment system based on multimodal vision, comprising: An environment confirmation module is used to confirm the receipt of a wire harness welding alignment command and to confirm the wire harness welding alignment environment based on the wire harness welding alignment command. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. A multimodal perception module is used to acquire images of the welding terminal based on the visual information acquisition unit to obtain a multimodal dataset; The multimodal dataset is preprocessed to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set. The pose calculation module is used to obtain an ideal model, obtain a multimodal fusion objective function based on the ideal model, a two-dimensional feature point set, a three-dimensional feature point set, and a key height feature point set, calculate the optimal rigid body transformation based on the multimodal fusion objective function, and calculate the pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose. Acquire a delay dataset, calculate translation delay compensation based on the delay dataset and motion control unit, and perform wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain a welded wire harness; The execution verification module is used to perform quality analysis on the welding wire harness and obtain evaluation results. The evaluation results include excessive deviation or no deviation. If the evaluation result is no deviation, the welding wire harness is confirmed as a precision welding wire harness. Based on the precision welding wire harness, high-speed wire harness welding alignment of the welding terminals is realized.

[0015] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: Memory, storing at least one instruction; The processor executes the instructions stored in the memory to implement the high-speed wire harness welding alignment method based on multimodal vision described above.

[0016] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned high-speed wire harness welding alignment method based on multimodal vision.

[0017] To address the problems described in the background art, this invention confirms the receipt of wire harness welding alignment commands and, based on these commands, confirms the wire harness welding alignment environment. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. Therefore, this invention, in the high-speed wire harness welding alignment process, fully considers the small size of the terminals, processing deformation, dynamic errors caused by high-speed movement, and the complex requirements of sub-millimeter precision that traditional single-vision methods cannot meet. Thus, by confirming the alignment environment through multimodal vision, the organic integration of multi-source information fusion and closed-loop control of the system is ensured, laying the foundation for subsequent high-precision... The alignment provides a reliable foundation, thereby improving the overall stability and consistency of high-speed welding. Based on the visual information acquisition unit, images of the welding terminals are acquired to obtain a multimodal dataset. This invention employs multimodal visual acquisition (including 2D images, 3D coordinates, and height contours, among other multi-channel data), overcoming the limitations of a single camera or sensor in complex lighting, reflection, and occlusion scenarios. This provides a rich and complementary information foundation for feature extraction. The multimodal dataset is preprocessed to obtain preprocessed 2D images, preprocessed 3D coordinate sets, and preprocessed height contours. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D images, preprocessed 3D coordinate sets, and preprocessed height contours. Extracting features yields a two-dimensional feature point set, a three-dimensional feature point set, and a key height feature point set. This invention, through targeted preprocessing and multi-level feature extraction, captures planar contours, spatial positions, and microscopic height changes, achieving a full-dimensional representation of the geometry and posture of the welding terminal. This significantly improves the robustness and completeness of the feature points. An ideal model is obtained, and a multimodal fusion objective function is derived based on this ideal model, the two-dimensional feature point set, the three-dimensional feature point set, and the key height feature point set. The optimal rigid body transformation is calculated based on this multimodal fusion objective function, and the pose deviation is calculated based on the optimal rigid body transformation and the pre-constructed ideal pose. This invention introduces a multimodal fusion objective function, unifying 2D, 3D, and height features into the optimal rigid body transformation. The volume transformation solution framework achieves global optimal pose estimation under the constraint of complementary multi-source information, effectively suppressing the interference of single-modal noise or local distortion on accuracy, thereby significantly improving the accuracy and stability of pose deviation calculation; it acquires a delay dataset, calculates translation delay compensation based on the delay dataset and motion control unit, and performs wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain the welded wire harness. It can be seen that this invention addresses the system delays (visual processing delay, communication delay, mechanical response delay, etc.) unique to high-speed motion scenarios by using delay compensation for feedforward correction, achieving spatiotemporal synchronization of vision-motion-execution, thereby effectively eliminating the common "cannot catch up" or "overshoot" phenomena under high-speed dynamics;The welding harness is subjected to quality analysis to obtain evaluation results, which include excessive deviation or no deviation. If the evaluation result is no deviation, the welding harness is confirmed as a precision welding harness. Based on the precision welding harness, high-speed welding alignment of the welding terminals is achieved. It is evident that this invention introduces a closed-loop quality evaluation and judgment mechanism in the final stage, forming a complete adaptive alignment closed loop of "acquisition-fusion-estimation-compensation-execution-verification". This not only ensures high precision in single welding but also provides continuous optimization basis for subsequent batch production. Therefore, this invention can significantly improve the accuracy, speed, stability, and automation level of high-speed harness welding alignment. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a high-speed wire harness welding alignment method based on multimodal vision provided in an embodiment of the present invention. Figure 2 This is a functional block diagram of a high-speed wire harness welding alignment system based on multimodal vision provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the high-speed wire harness welding alignment method based on multimodal vision, according to an embodiment of the present invention.

[0019] Explanation of reference numerals in the attached figures: 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] This application provides a high-speed wire harness welding alignment method based on multimodal vision. The execution subject of this high-speed wire harness welding alignment method based on multimodal vision includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the high-speed wire harness welding alignment method based on multimodal vision can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0023] Reference Figure 1 The diagram shown is a flowchart illustrating a high-speed wire harness welding alignment method based on multimodal vision according to an embodiment of the present invention. In this embodiment, the high-speed wire harness welding alignment method based on multimodal vision includes: S1. Confirm receipt of wire harness welding alignment command, and confirm wire harness welding alignment environment based on the wire harness welding alignment command. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit.

[0024] It should be explained that the wire harness welding alignment command refers to the command issued by the personnel who want to achieve high-speed wire harness welding alignment; the wire harness welding alignment environment refers to the necessary environment for achieving high-speed wire harness welding alignment; and the wire harness welding alignment system refers to a system capable of achieving high-speed wire harness welding alignment. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. For specific applications of these units, please refer to subsequent embodiments. The welding terminal refers to the wiring terminal that needs to be connected to the wire harness for high-speed wire harness welding alignment. The purpose of this invention is to improve the accuracy and welding efficiency of high-speed wire harness welding alignment.

[0025] For example, Zhang is a welder at a car repair shop. In order to improve the accuracy and efficiency of high-speed wire harness welding alignment, Zhang issued a wire harness welding alignment command and confirmed the wire harness welding alignment environment.

[0026] S2. Based on the visual information acquisition unit, images of the welding terminals are acquired to obtain a multimodal dataset.

[0027] Furthermore, the step of acquiring images of the welding terminal based on the visual information acquisition unit to obtain a multimodal dataset includes: Obtain the welding station; when the welding terminal arrives at the welding station, obtain a visual information acquisition command. The visual information acquisition unit receives the visual information acquisition command, wherein the visual information acquisition unit includes a 2D image acquisition device, a 3D visual acquisition device, and a line laser contour acquisition device, and controls the visual information acquisition unit to perform the following operations based on the visual information acquisition command: The welding terminal is acquired using the 2D image acquisition device to obtain a two-dimensional image of the terminal. The 3D vision acquisition device acquires images of the welding terminal to obtain an original depth image. Based on the pre-built 3D reconstruction algorithm, cloud coordinates are constructed from the original depth image to obtain a 3D point cloud coordinate set. The line laser profile acquisition device performs line laser scanning along a preset welding path to obtain a one-dimensional height profile of the welding terminal. By summarizing the two-dimensional images, three-dimensional point cloud coordinate sets, and one-dimensional height contours of the terminals, a multimodal dataset is obtained.

[0028] It should be understood that the welding station refers to the area where wire harness welding is performed. The method of obtaining a visual information acquisition command when the welding terminal arrives at the welding station means that when the welding terminal arrives at the welding station, it indicates that the welding terminal has completed the pre-transfer process and can be used for corresponding information acquisition. The visual information acquisition command refers to the command issued by the wire harness welding alignment system to control the visual information acquisition unit to acquire information. The visual information acquisition unit refers to a functional module that integrates multiple acquisition devices and is capable of acquiring visual information in multiple modalities. The visual information acquisition unit includes a 2D image acquisition device, a 3D visual acquisition device, and a line laser contour acquisition device. The method of performing two-dimensional image acquisition of the welding terminal based on the 2D image acquisition device refers to using the 2D image acquisition device to perform two-dimensional image acquisition of the welding terminal at the welding station. The two-dimensional image of the terminal refers to the two-dimensional image of the welding terminal acquired by the 2D image acquisition device. The 2D image acquisition device is a device capable of performing two-dimensional image acquisition; optionally, a high-resolution camera can be used as the 2D image acquisition device. The method for acquiring images of the welding terminal based on the 3D vision acquisition device refers to acquiring a depth image of the welding terminal using the 3D vision acquisition device. The original depth image is an image that can characterize the relative distance between each point on the welding terminal and the sensor, and each pixel of the original depth image records the relative distance between a certain point on the welding terminal and the sensor. The 3D vision acquisition device is a device that can perform spatial ranging and imaging of the welding terminal. Optionally, a 3D camera can be used as the 3D vision acquisition device. The method for constructing cloud coordinates of the original depth image based on a pre-built 3D reconstruction algorithm refers to establishing a spatial rectangular coordinate system with the 3D vision acquisition device as the origin, and constructing the spatial rectangular coordinates of each point on the surface of the welding terminal based on the relative distances recorded on the original depth image. The 3D point cloud coordinate set is the set of spatial rectangular coordinates corresponding to each point on the surface of the welding terminal. The method of scanning the welding terminal with a line laser along a preset welding path based on the line laser profile acquisition device refers to the line laser profile acquisition device projecting a line laser onto the surface of the welding terminal according to a welding path matched with the welding operation and simultaneously acquiring the reflected signal to obtain the spatial position information of the welding terminal distributed along the welding path. The welding path refers to the preset path on the surface of the welding terminal for welding.The line laser profile acquisition device refers to a device capable of emitting a line laser and obtaining spatial position by receiving laser reflection signals. Optionally, a laser sensor can be used as the line laser profile acquisition device. The one-dimensional height profile refers to the measured structured data, specifically represented as data pairs arranged in spatial order (along the welding path). Each data pair contains two values: the linear position of the preset welding path and the height value of the welding terminal surface relative to a reference plane (a cross-section of the welding terminal) at that position. For example, three measurement points were measured along the welding path, resulting in a one-dimensional height profile of [(0, 2.01), (5.0, 2.15), (10.0, 2.08)]. The multimodal data refers to the combination of a two-dimensional image, a three-dimensional point cloud coordinate set, and a one-dimensional height profile.

[0029] S3. Perform data preprocessing on the multimodal dataset to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour. Based on the multimodal feature extraction unit, perform feature extraction on the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set.

[0030] It should be explained that the data preprocessing of the multimodal dataset to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour includes: A unified time standard is obtained based on a pre-built system clock. The two-dimensional image, three-dimensional point cloud coordinate set, and one-dimensional height profile of the terminal are time-calibrated based on the unified time standard to obtain a calibrated two-dimensional image, a calibrated three-dimensional point cloud coordinate set, and a calibrated one-dimensional height profile. The calibrated 2D image is denoised based on a pre-built image filtering algorithm to obtain a denoised 2D image. The contrast of the denoised 2D image is then enhanced to obtain a pre-processed 2D image. For each calibration 3D point cloud coordinate in the calibration 3D point cloud coordinate set, the following operation is performed: Based on the calibrated 3D point cloud coordinates, obtain multiple surrounding 3D point cloud coordinates, calculate multiple Euclidean distances based on the multiple surrounding 3D point cloud coordinates, and calculate the mean and standard deviation of the distances based on the multiple Euclidean distances. Based on the mean and standard deviation of the distance, the mean distance and the mean standard deviation of the overall distance are calculated. The mean distance and the standard deviation of each calibration 3D point cloud coordinate in the calibration 3D point cloud coordinate set are compared with the mean distance and the mean standard deviation of the overall distance. If the mean distance is greater than the mean distance and the standard deviation is greater than the mean standard deviation of the overall distance, the calibration 3D point cloud coordinates are removed to obtain a preprocessed 3D coordinate set. The calibrated one-dimensional height profile is smoothed based on a pre-built smoothing algorithm to obtain a pre-processed height profile.

[0031] Furthermore, the method for obtaining a unified time standard based on a pre-built system clock refers to establishing a unified global time reference using a high-precision clock source (such as a GPS synchronization clock) within the wire harness welding alignment system. The system clock refers to the high-precision clock within the wire harness welding alignment system. The method for time calibration of the terminal's two-dimensional image, three-dimensional point cloud coordinate set, and one-dimensional height profile based on the unified time standard refers to comparing or interpolating the timestamps of different modal data to align the three sets of data to the same time on the time axis, thereby eliminating data asynchrony caused by acquisition trigger delay or transmission delay. The calibrated two-dimensional image, calibrated three-dimensional point cloud coordinate set, and calibrated one-dimensional height profile refer to the terminal's two-dimensional image, three-dimensional point cloud coordinate set, and one-dimensional height profile after time calibration, respectively. The method for denoising the calibrated two-dimensional image based on a pre-built image filtering algorithm refers to using an image filtering algorithm to remove random noise from the image. Optionally, a Gaussian filtering algorithm can be used as the image filtering algorithm. The denoised two-dimensional image refers to the calibrated two-dimensional image after denoising. The method for enhancing the contrast of the denoised 2D image refers to using adaptive histogram equalization to make key features such as terminal edges and pad areas in the denoised 2D image clearer and more prominent. The preprocessed 2D image refers to the denoised 2D image after contrast enhancement. The method for obtaining multiple surrounding 3D point cloud coordinates based on the calibrated 3D point cloud coordinates refers to first using the calibrated 3D point cloud coordinates as the center, and then using a spatial index (k-d tree) algorithm to search for the two nearest surrounding calibrated 3D point cloud coordinates (the calibrated 3D point cloud coordinates and the two nearest surrounding calibrated 3D point cloud coordinates are not on the same straight line). Then, a plane is determined based on the calibrated 3D point cloud coordinates and the two nearest surrounding calibrated 3D point cloud coordinates. Based on the ideal model (i.e., the standard CAD model of the welding terminal that is subsequently established), multiple model planes representing the real surface of the welding terminal are obtained. The planes are overlapped and compared with the model planes. If they overlap with any model plane, the spatial index (k-d tree) algorithm is used to continue searching for the nearest fixed number (e.g., 20) of calibrated 3D point cloud coordinates on the plane, and these coordinates are used as the surrounding 3D point cloud coordinates. If they do not overlap with any model plane, the spatial index (k-d tree) algorithm is used again to search for the two nearest surrounding calibrated 3D point cloud coordinates, and the above steps of determining the plane, overlapping and comparing, and subsequent steps are repeated. The plurality of surrounding 3D point cloud coordinates refer to a fixed number of calibrated 3D point cloud coordinates that are closest to the calibrated 3D point cloud coordinates. Calculating the plurality of Euclidean distances based on the plurality of surrounding 3D point cloud coordinates means performing spacing calculations on each coordinate point in the plurality of surrounding 3D point cloud coordinates based on the Euclidean distance calculation formula. The plurality of Euclidean distances refer to the Euclidean distances between the calibrated 3D point cloud coordinates and the plurality of surrounding 3D point cloud coordinates.The method for calculating the mean and standard deviation of distances based on the multiple Euclidean distances refers to calculating the mean of distances based on the multiple Euclidean distances, and then calculating the standard deviation based on the mean distance and the multiple Euclidean distances according to the standard deviation formula. The mean distance and standard deviation refer to the mean and standard deviation of the multiple Euclidean distances, respectively. The method for calculating the overall mean distance and overall standard deviation based on the mean distance and standard deviation refers to calculating the mean of the mean distance and standard deviation corresponding to all surrounding 3D point cloud coordinates in the calibration 3D point cloud coordinate set. The overall mean distance and overall standard deviation refer to the mean of the mean distance and standard deviation corresponding to all surrounding 3D point cloud coordinates in the calibration 3D point cloud coordinate set, respectively. If the mean distance is greater than the overall mean distance and the standard deviation is greater than the overall standard deviation, it indicates that the calibration 3D point cloud coordinates deviate significantly and need to be removed. The preprocessed 3D coordinate set refers to the calibration 3D point cloud coordinate set after removing abnormal calibration 3D point cloud coordinates. The method for smoothing the calibrated one-dimensional height profile based on a pre-built smoothing algorithm refers to using a smoothing algorithm to eliminate high-frequency noise (significantly deviated data pairs in the one-dimensional height profile) introduced during laser scanning due to vibration, minor surface reflections, or sensor jitter, while preserving the true height change trend along the welding path. Optionally, polynomial fitting smoothing or moving average filtering can be used as the smoothing algorithm. The pre-processed height profile refers to the calibrated one-dimensional height profile after the above smoothing operation.

[0032] It should be understood that the step of extracting features from the preprocessed 2D image, preprocessed 3D coordinate set, and preprocessed height contour based on the multimodal feature extraction unit to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set includes: Edge detection is performed on the preprocessed 2D image based on a multimodal feature extraction unit and a pre-constructed two-dimensional feature detection algorithm to obtain two-dimensional contour edges. Feature points are then extracted from the two-dimensional contour edges to obtain two-dimensional feature points. Multiple principal curvatures are calculated based on a pre-constructed surface fitting algorithm and a pre-processed 3D coordinate set. The multiple principal curvatures are then filtered based on a preset curvature threshold, and a set of 3D feature points is obtained based on the pre-processed 3D coordinates corresponding to the filtered principal curvatures. The pre-built inflection point detection algorithm is used to detect inflection points in the preprocessed height contour to obtain a set of key height feature points.

[0033] It should be explained that the method of edge detection of the preprocessed 2D image based on the multimodal feature extraction unit and the pre-constructed two-dimensional feature detection algorithm refers to the multimodal feature extraction unit performing edge extraction on the preprocessed 2D image based on the two-dimensional feature detection algorithm to obtain the two-dimensional contour edges of the welding terminal pads, pins, notches, etc. Optionally, Canny edge detection or Sobel operator can be used as the two-dimensional feature detection algorithm. The two-dimensional contour edge refers to the contour of the welding terminal pads, pins, notches, etc. detected based on the two-dimensional feature detection algorithm. The method of feature point extraction of the two-dimensional contour edge refers to extracting significant corners, intersections, or curvature abrupt changes on the two-dimensional contour edge as feature points based on the Harris corner detection algorithm. The two-dimensional feature point set refers to the set of feature points (corners, intersections, or curvature abrupt changes) extracted on the two-dimensional contour edge. The method for calculating multiple principal curvatures based on a pre-constructed surface fitting algorithm and a pre-processed 3D coordinate set refers to using a surface fitting algorithm to perform local surface fitting on the pre-processed 3D coordinate set to obtain a fitted surface characterizing its surface morphology. Then, based on the differential geometric properties of the fitted surface, the principal curvatures of each corresponding point on the surface (pre-processed 3D coordinates in the pre-processed 3D coordinate set) are calculated. Optionally, a least squares plane fitting algorithm can be used as the surface fitting algorithm. The method for filtering the multiple principal curvatures based on a preset curvature threshold refers to using the maximum value and extreme value of the principal curvature at each point. The arithmetic mean of the minimum values ​​is calculated to obtain the average curvature. Then, the average curvature corresponding to each point is compared with a preset curvature threshold. Abnormal data with average curvature exceeding the curvature threshold are eliminated, and average curvatures that do not exceed the curvature threshold are selected. The preprocessed three-dimensional coordinates corresponding to the average curvatures that do not exceed the curvature threshold are used as three-dimensional feature points. The curvature threshold refers to a preset principal curvature critical value used to determine whether the curvature is abnormal. The curvature threshold can be obtained by measuring the average curvature of each point on the surface of a batch of confirmed qualified welded terminal samples and selecting the largest average curvature as the curvature threshold. The method of inflection point detection of the preprocessed height profile based on the preconstructed inflection point detection algorithm refers to using the Douglas-Peucker line segment simplification algorithm to identify the turning points (inflection points) in the height profile curve where the slope changes significantly. The turning points usually correspond to height change positions on the welding path (such as steps, pad edges, pin roots, etc.). These points are used as key height feature points. The set of key height feature points refers to the set of turning points corresponding to height change positions on the welding path.

[0034] S4. Obtain an ideal model, and based on the ideal model, the two-dimensional feature point set, the three-dimensional feature point set, and the key height feature point set, obtain a multimodal fusion objective function, calculate the optimal rigid body transformation based on the multimodal fusion objective function, and calculate the pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose.

[0035] Furthermore, the step of obtaining a multimodal fusion objective function based on the ideal model, the two-dimensional feature point set, the three-dimensional feature point set, and the key height feature point set, and calculating the optimal rigid body transformation based on the multimodal fusion objective function, includes: Based on the ideal model, an ideal three-dimensional coordinate set is obtained. Based on the two-dimensional feature point set and the ideal three-dimensional coordinate set, feature point matching is performed to obtain a two-dimensional to three-dimensional feature point set, wherein the two-dimensional to three-dimensional feature point set includes M two-dimensional to three-dimensional feature points. Based on the ideal model, an ideal model point set and an ideal height value set are obtained. A multimodal fusion objective function is constructed based on the aforementioned 2D-to-3D feature point set, ideal model point set, ideal height value set, 2D feature point set, 3D feature point set, and key height feature point set. The 3D feature point set includes L 3D feature points, and the key height feature point set includes P key height feature points. The multimodal fusion objective function is shown below: in, Indicates the target score. Represents the rotation matrix. Represents the translation vector. Represents the pixel scaling factor. Represents the camera projection model. This represents the camera intrinsic parameter matrix. Represents the first digit in the set of 2D to 3D feature points. Two-dimensional to three-dimensional feature points Represents the first feature point in the two-dimensional feature point set. Two-dimensional feature points, Represents the 3D feature point set. Three-dimensional feature points, Let j represent the j-th ideal model point in the ideal model point set. Represents component functions, Represents the first key height feature point in the set. Key height feature points, Represents the set of ideal height values. An ideal height value, , and Indicates the weighting coefficient. , and Indicates subscript, Represents norm operations; The rotation matrix and translation vector in the multimodal fusion objective function are iterated until the objective score is less than a preset convergence threshold. Then, the rotation matrix and translation vector are integrated to obtain the optimal rigid body transformation, wherein the optimal rigid body transformation includes the optimal rotation matrix and the optimal translation vector.

[0036] It should be understood that the method for obtaining the ideal three-dimensional coordinate set based on the ideal model refers to extracting the three-dimensional coordinates (in millimeters, in the world coordinate system) of all key geometric points from a pre-established standard CAD model of the welding terminal. The ideal three-dimensional coordinate set refers to the set of three-dimensional coordinates of all key geometric points extracted from the standard model, which is used for subsequent matching. The method for performing feature point matching based on the two-dimensional feature point set and the ideal three-dimensional coordinate set to obtain the two-dimensional to three-dimensional feature point set refers to using the PnP (Perspective-n-Point) algorithm to match the currently extracted two-dimensional feature point set (image plane coordinates) with the corresponding three-dimensional points after the ideal model is projected onto the image plane coordinates, solving the correspondence from image coordinates to three-dimensional model coordinates, thereby associating two-dimensional feature points representing the same point but with different dimensions with the ideal three-dimensional coordinates to obtain M two-dimensional to three-dimensional feature points (i.e., each point corresponds to a set of image two-dimensional coordinates and a set of matching three-dimensional reference coordinates). The two-dimensional to three-dimensional feature point set refers to the set of three-dimensional feature points that, after being matched and associated by the PnP algorithm, are mapped from two-dimensional feature points to the world coordinate system and correspond one-to-one with the ideal three-dimensional coordinates in the standard CAD model of the welding terminal. The method for obtaining the ideal model point set and ideal height value set based on the ideal model refers to extracting significant geometric feature points (such as corner points, center points, etc.) for three-dimensional feature matching from the standard CAD ideal model of the same welding terminal to form L ideal model points. At the same time, sampling is performed along the welding path in the standard CAD ideal model at a set interval (e.g., every 0.5 mm), and the height value of each sampling point in the Z-axis direction in the world coordinate system is calculated (the reference plane for the height value acquisition is consistent with the reference plane for the one-dimensional height contour acquisition mentioned above), resulting in P ideal height values. Then, the ICP precise matching algorithm is used to match the three-dimensional feature points in the three-dimensional feature point set with the ideal three-dimensional coordinates in the ideal model point set, and to match the height feature points in the key height feature point set with the ideal height values ​​in the ideal height value set. The ideal model point set refers to the set of three-dimensional coordinates of L significant geometric feature points (such as corner points, center points, etc.) extracted from the standard CAD ideal model of the welding terminal and used for precise spatial matching with the three-dimensional feature point set via ICP. The ideal height value set refers to the set of reference height values ​​of P key height feature points extracted from the same standard CAD ideal model of the welding terminal and used for matching with the key height feature point set. The method for constructing the multimodal fusion objective function is to sum the error terms of the three modes by weighting them: the first term is the two-dimensional reprojection error (measuring the deviation between the transformed three-dimensional point projected onto the image plane and the actual observed two-dimensional feature point), the second term is the three-dimensional point-to-point Euclidean distance error (directly constraining the accuracy of rigid body transformation in space), and the third term is the height component error (specifically constraining the accuracy in the Z direction to prevent height drift caused by two-dimensional matching).The target score is the sum of all error terms. As can be seen from the above, the smaller the value, the higher the pose transformation accuracy. The weight coefficient can be dynamically adjusted according to the importance of the actual scenario (for example, it can be appropriately increased in high-speed production). (To enhance high precision). The rotation matrix refers to the rotational transformation used to describe the measured coordinate system relative to the ideal model coordinate system, belonging to the rotation component of rigid body transformation. The translation vector is used to describe the translational transformation of the measured coordinate system relative to the ideal model coordinate system, belonging to the translation component of rigid body transformation. The pixel scaling factor refers to the normalized scaling factor between the image pixel coordinates and the physical dimensions of the world coordinate system, used to unify the dimensions of pixel domain error and physical domain error. The camera projection model is a pinhole camera projection function used to project the three-dimensional world coordinates after rigid body transformation onto the two-dimensional image pixel plane. The camera intrinsic parameter matrix refers to a 3×3 triangular matrix containing inherent parameters such as camera focal length, image principal point, and distortion correction, which are the input parameters of the camera projection model. The camera focal length, image principal point, and distortion correction can be measured using known techniques such as Zhang Zhengyou calibration method, checkerboard calibration method, or OpenCV camera calibration module, which will not be elaborated here. The component function refers to a function used to extract height values ​​along the Z-axis (the Z-axis is the direction perpendicular to the reference plane) from a set of input key height feature points. For example, the component function for extracting three-dimensional coordinates is as follows: However, the key height feature point set extraction form for data pairs is as follows: ,in, Indicates the linear position of the welding path. The height values ​​are represented as follows: For a set of height feature points [(0, 2.01), (5.0, 2.15), (10.0, 2.08)], the height values ​​obtained after extraction using the component function are 2.01, 2.15, and 2.08. The method for iteratively optimizing the rotation matrix R and translation vector T can be solved using the Levenberg-Marquardt (LM) algorithm until the target score converges to a preset convergence threshold. This convergence threshold refers to the preset threshold for determining the convergence of the multimodal fusion objective function (e.g., 0.05 mm). 2 The objective function is the square of each error, and as shown in the above calculation formula, the objective function is the square of each error. Furthermore, regardless of the 2D reprojection error (after pixel scaling), the 3D point-to-point Euclidean distance error, or the height component error, the unit is millimeters (mm). Therefore, the units of the objective score and the convergence threshold are both square millimeters (mm). 2 The optimal rotation matrix and optimal translation vector refer to the three-dimensional orthogonal rotation matrix and three-dimensional translation vector obtained by iterative optimization of the multimodal fusion objective function until convergence, respectively. The optimal rigid body transformation refers to the set of the optimal rotation matrix and the optimal translation vector.

[0037] For example, if the ideal model is a standard automotive wiring harness terminal CAD, the extracted ideal 3D coordinate set contains approximately 200 reference points; the current 2D feature point set has 120 points, and through PnP matching, M=80 reliable 2D to 3D feature points are obtained, resulting in a 3D feature point set L=70 points, and a critical height feature point P=12 points. Weighting coefficients are set to 1.0, 0.8, and 1.2 respectively. After approximately 15 iterations of optimization using the LM algorithm, the target score is 0.042mm. 2 <0.08 mm 2 The optimal rotation matrix (3×3 orthogonal matrix) and the optimal translation vector = [0.012, -0.008, 0.015] mm are obtained.

[0038] It should be explained that the calculation of pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose includes: Based on the ideal pose, obtain the ideal rotation matrix and the ideal translation vector; The rotation deviation is calculated based on the optimal rotation matrix and the ideal rotation matrix, and the displacement deviation is calculated based on the ideal translation vector and the optimal translation vector. The positional deviation is obtained by summing the rotational and displacement deviations.

[0039] Furthermore, the method for obtaining the ideal rotation matrix and ideal translation vector based on the ideal pose refers to calculating the pose that the welding terminal should be in under ideal conditions (i.e., the target relative pose of the welding gun to the terminal), and the ideal pose is obtained by CAD simulation. The ideal pose is directly decomposed into an ideal rotation matrix and an ideal translation vector. The ideal rotation matrix and ideal translation vector refer to a 3×3 orthogonal matrix representing the ideal orientation and a vector representing the ideal position offset, respectively, obtained from the decomposition of the ideal pose. The method for calculating the rotation deviation based on the optimal rotation matrix and the ideal rotation matrix refers to calculating the angle difference using the Frobenius norm of the rotation matrix difference. The method for calculating the displacement deviation based on the ideal translation vector and the optimal translation vector is to directly calculate the Euclidean norm of the difference between the two vectors. The rotation deviation refers to the angle difference characterizing the orientation of the measured terminal relative to the ideal pose, and the displacement deviation refers to the vector difference characterizing the spatial position of the measured terminal relative to the ideal pose. The pose deviation refers to the set of rotation deviation and displacement deviation.

[0040] For example, if the translation vector obtained by the optimal rigid body transformation is [2.1, -1.4, 0.8] mm and the ideal translation vector is [0,0,0] mm, then the displacement deviation is approximately 2.65 mm; the rotation deviation is approximately 0.9°, and the pose deviation can be expressed as {[2.1, -1.4, 0.8], 0.9°}.

[0041] S5. Obtain the delay dataset, calculate the translation delay compensation number based on the delay dataset and the motion control unit, and perform wire harness welding based on the welding execution unit, the translation delay compensation number and the pose deviation to obtain the welded wire harness.

[0042] It should be understood that the acquisition of the delay dataset, and the calculation of the translation delay compensation number based on the delay dataset and the motion control unit, includes: A delay dataset is obtained based on the motion control unit, wherein the delay dataset includes visual processing delay, communication delay and robot response delay, and the total delay time is calculated based on the delay dataset; The terminal transmission line speed and terminal transmission acceleration are obtained based on the motion control unit. The translational delay compensation number is calculated based on the total delay time, terminal transmission line speed, and terminal transmission acceleration, using the following formula: in, This represents the translation delay compensation number. Indicates the total delay time. Indicates transmission acceleration. This indicates the terminal transmission line speed.

[0043] It should be explained that the method for obtaining the delay dataset based on the motion control unit refers to recording and statistically analyzing the visual processing delay (including delays in image preprocessing, feature extraction, PnP matching, etc.), communication delay (the round-trip transmission delay of sending the pose deviation result to the motion control unit via Ethernet or serial port buses), and robot response delay (the mechanical delay of the servo driver responding and starting movement after the motion control unit receives the pose command) using a high-precision timer built into the motion control unit. The motion control unit refers to a functional module capable of controlling the conveying of welding terminals and acquiring the linear velocity and acceleration of the terminal conveying in real time. Optionally, the motion control unit can be constructed using encoders, conveyor belts, etc. The delay dataset refers to the collection of visual processing delay, communication delay, and robot response delay. The method for calculating the total delay time based on the delay dataset is to directly add the above three delays; the total delay time is the sum of the visual processing delay, communication delay, and robot response delay. The method for obtaining the terminal conveying linear velocity and terminal conveying acceleration based on the motion control unit refers to using the motion control unit to acquire the real-time speed and acceleration of the conveyor belt. The terminal conveying linear speed and terminal conveying acceleration refer to the linear speed and instantaneous acceleration of the welding terminal when it is conveyed to the welding station by the conveyor belt, and the translation delay compensation number refers to the additional forward distance of the welding terminal relative to the visual acquisition during the total delay time.

[0044] For example, if the measured visual processing delay is 45ms, the communication delay is 8ms, and the robot response delay is 12ms, then the total delay time is 0.065s. The current transmission line speed is v = 1200 mm / s, and the acceleration is a = 500 mm / s². 2 (Acceleration segment), then the translation delay compensation = 1200 × 0.065 + 0.5 × 500 × (0.065) 2 ≈78 + 1.06≈79.06 mm, meaning that the welding position needs to be compensated 79 mm in advance.

[0045] Furthermore, the process of welding the wire harness based on the welding execution unit, the translation delay compensation number, and the pose deviation to obtain the welded wire harness includes: The compensated displacement is calculated based on the translation delay compensation number and the displacement deviation in the pose deviation. The initial welding position is obtained based on the pre-constructed welding gun, and the initial welding position is adjusted based on the compensated displacement and rotation deviation to obtain the adjusted welding position. The welding execution unit controls the welding gun to weld the welding terminal at the adjusted welding position to obtain a welding wire harness.

[0046] It should be understood that the method of calculating the compensated displacement based on the translational delay compensation number and the displacement deviation in the pose deviation refers to mapping the translational delay compensation number to a spatial rectangular coordinate system to obtain a spatial vector of the translational delay compensation number (the direction of the conveyor belt direction in the spatial rectangular coordinate system), and then adding the spatial vector of the translational delay compensation number to the displacement deviation to obtain the compensated displacement vector. The compensated displacement refers to the vector obtained after the above displacement compensation. The method of obtaining the initial welding position based on the pre-constructed welding gun refers to reading the preset initial position of the welding gun relative to the ideal terminal pose (usually a waiting pose at a certain safe height above the center of the weld point) from the teaching point of the welding execution unit or CAD offline programming. The method of adjusting the initial welding position based on the compensated displacement and rotational deviation refers to first applying the rotational deviation in the optimal rigid body transformation (or the rotational part in the pose deviation) to perform a rotational transformation (rotation around the local coordinate system of the terminal) on the initial welding position, and then performing a translational adjustment along the compensated displacement vector to obtain the final adjusted welding position. The adjusted welding position refers to the adjusted initial welding position. The method of controlling the welding gun to weld the welding terminal at the adjusted welding position based on the welding execution unit refers to moving the welding robot to the adjusted position according to the planned path (straight line or circular interpolation), triggering the welding power supply to perform spot welding or laser welding and other processes to complete the welding connection between the wire harness and the terminal. The welding wire harness refers to the wire harness after welding with the welding terminal.

[0047] S6. Perform quality analysis on the welding wire harness to obtain evaluation results, wherein the evaluation results include excessive deviation or no deviation. If the evaluation result is no deviation, the welding wire harness is confirmed as a precision welding wire harness. Based on the precision welding wire harness, high-speed wire harness welding alignment of the welding terminals is achieved.

[0048] It should be explained that the quality analysis of the welded wire harness to obtain the evaluation results includes: Based on the visual information acquisition unit, images of the welding wire harness are acquired to obtain images after welding. The image after welding is segmented to obtain the welding point region. The pixel center coordinates are obtained based on the welding point region. The physical position deviation is calculated based on the pixel center coordinates and the preset ideal position. The area of ​​the welding region is calculated based on the welding point region. The physical position deviation and the area of ​​the welding area are compared with the preset position deviation threshold and the preset area of ​​the welding area, respectively. If the physical position deviation is less than the position deviation threshold and the area of ​​the welding area is within the area of ​​the welding area, the evaluation result is confirmed as no deviation; otherwise, the evaluation result is confirmed as excessive deviation.

[0049] Furthermore, the method of acquiring images of the welding wire harness based on the visual information acquisition unit to obtain a post-weld image refers to the visual information acquisition unit immediately capturing an image of the weld point area of ​​the currently welded wire harness after the welding execution unit completes the welding action. The post-weld image refers to a high-resolution color image of the welding wire harness acquired by the visual information acquisition unit. The step of segmenting the post-weld image to obtain the weld point area includes: performing grayscale processing on the post-weld image to obtain a grayscale image; obtaining a preset grayscale histogram of the welding area; obtaining grayscale intervals of the welding area based on the grayscale histogram of the welding area; segmenting the grayscale image based on the grayscale intervals of the welding area to obtain multiple initial welding areas; and determining the initial welding area with the largest area among the multiple initial welding areas as the weld point area.The method for grayscale processing of the post-weld image refers to converting the multi-channel color values ​​(e.g., RGB) of each pixel in the image into a single-channel grayscale value (range 0~255) using an image grayscale weighted average method. The grayscale image refers to the post-weld image after grayscale conversion, and each pixel in the grayscale image corresponds to a grayscale value (range 0~255). The method for obtaining the preset grayscale histogram of the welding area refers to pre-selecting multiple qualified standard welded wire harness samples and manually identifying the welded areas (the welded areas are darker in color compared to unwelded areas). (Significant regional color differences) Grayscale detection was performed on the welded areas of these samples (using the same method as the grayscale conversion operation described above) to obtain a grayscale histogram characterizing the grayscale value distribution of the welded area. This welded area grayscale histogram is a histogram statistically obtained from multiple qualified standard welded wire harness samples, used to characterize the pixel grayscale value distribution characteristics of the standard weld point area. It uses grayscale values ​​as the abscissa and the frequency of pixel occurrence corresponding to the grayscale value as the ordinate. The method for obtaining the grayscale interval of the welded area based on the welded area grayscale histogram refers to statistically analyzing the grayscale values ​​based on the welded area grayscale histogram. Gray values ​​whose frequency of occurrence exceeds a preset frequency (e.g., 50 times) are used to identify the grayscale range of the welding area. This welding grayscale range refers to the range of gray values ​​obtained from the above statistics used to identify whether an area is a welding area. For example, the preset frequency is set to 50 times. In the preset welding area grayscale histogram, pixels with grayscale values ​​80, 81, 82...160 all appear more than 50 times, while pixels with grayscale values ​​less than 80 or greater than 160 appear less than 50 times. Therefore, the welding grayscale range is [80, 160]. The method for segmenting a grayscale image based on the grayscale range to obtain multiple initial welding regions refers to using the welding grayscale range as a threshold for judgment, identifying pixel regions with grayscale values ​​within this range as initial welding regions. These multiple initial welding regions refer to multiple unconnected pixel regions identified as having grayscale values ​​within the welding grayscale range. The method for determining the initial welding region with the largest area among these multiple initial welding regions as the welding point region refers to determining the initial welding region with the largest total number of pixels as the welding point region (the remaining regions may be small weld points generated by sputtering during welding). The welding point region refers to the image region containing the welding point segmented from the original post-weld image. The method for obtaining the pixel center coordinates based on the welding point region refers to solving for the geometric center of the segmented welding point region using the image's zeroth and first moments. The pixel center coordinates refer to the coordinates (in the pixel coordinate system) of the geometric center of the welding point region.The method for calculating the physical position deviation based on the pixel center coordinates and the preset ideal position refers to first calculating the pixel distance between the pixel center coordinates and the ideal position, and then converting the pixel distance into the actual physical distance using a pixel-physical scaling factor. The pixel-physical scaling factor refers to a preset conversion relationship between pixels and actual physical distances, for example, 0.02mm = 1 pixel. The preset ideal position refers to the target coordinates of the welding point in the image pixel coordinate system. The physical position deviation refers to the actual physical offset distance of the welding point position relative to the ideal welding point position. The method for calculating the welding area based on the welding point region refers to first counting the total number of pixels in the welding point region, and then combining the pixel-physical scaling factor to convert the pixel area into the actual physical area. The welding area refers to the actual physical area of ​​the welding region. The physical position deviation and the welding area are compared with a preset position deviation threshold and a preset welding area range, respectively. If the physical position deviation is less than the position deviation threshold and the welding area is within the welding area range, it indicates that the actual position offset of the welding point meets the accuracy requirements and the size of the welding area meets the process standards, and the evaluation result is "no deviation". Otherwise, it indicates that the actual position offset of the welding point exceeds the allowable accuracy (welding deviation), the welding area is too large (over-welding by the welding gun, welding gun position too close), or the welding area is too small (missed weld). The position deviation threshold and the welding area range refer to the preset maximum allowable value of the physical position offset of the welding point and the preset qualified welding area range, respectively. The evaluation result refers to the judgment result used to determine whether the welding meets the requirements, including excessive deviation or no deviation. If the evaluation result is "no deviation", it means that there is no deviation in the welding of the welding harness, and the welding harness is confirmed as a precision welding harness. The precision welding harness refers to the welding harness that is judged to have no welding deviation. If the evaluation result is "excessive deviation", it indicates that the welding deviation is too large, and there may be missed weld, welding deviation, or over-welding.

[0050] To address the problems described in the background art, this invention confirms the receipt of wire harness welding alignment commands and, based on these commands, confirms the wire harness welding alignment environment. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. Therefore, this invention, in the high-speed wire harness welding alignment process, fully considers the small size of the terminals, processing deformation, dynamic errors caused by high-speed movement, and the complex requirements of sub-millimeter precision that traditional single-vision methods cannot meet. Thus, by confirming the alignment environment through multimodal vision, the organic integration of multi-source information fusion and closed-loop control of the system is ensured, laying the foundation for subsequent high-precision... The alignment provides a reliable foundation, thereby improving the overall stability and consistency of high-speed welding. Based on the visual information acquisition unit, images of the welding terminals are acquired to obtain a multimodal dataset. This invention employs multimodal visual acquisition (including 2D images, 3D coordinates, and height contours, among other multi-channel data), overcoming the limitations of a single camera or sensor in complex lighting, reflection, and occlusion scenarios. This provides a rich and complementary information foundation for feature extraction. The multimodal dataset is preprocessed to obtain preprocessed 2D images, preprocessed 3D coordinate sets, and preprocessed height contours. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D images, preprocessed 3D coordinate sets, and preprocessed height contours. Extracting features yields a two-dimensional feature point set, a three-dimensional feature point set, and a key height feature point set. This invention, through targeted preprocessing and multi-level feature extraction, captures planar contours, spatial positions, and microscopic height changes, achieving a full-dimensional representation of the geometry and posture of the welding terminal. This significantly improves the robustness and completeness of the feature points. An ideal model is obtained, and a multimodal fusion objective function is derived based on this ideal model, the two-dimensional feature point set, the three-dimensional feature point set, and the key height feature point set. The optimal rigid body transformation is calculated based on this multimodal fusion objective function, and the pose deviation is calculated based on the optimal rigid body transformation and the pre-constructed ideal pose. This invention introduces a multimodal fusion objective function, unifying 2D, 3D, and height features into the optimal rigid body transformation. The volume transformation solution framework achieves global optimal pose estimation under the constraint of complementary multi-source information, effectively suppressing the interference of single-modal noise or local distortion on accuracy, thereby significantly improving the accuracy and stability of pose deviation calculation; it acquires a delay dataset, calculates translation delay compensation based on the delay dataset and motion control unit, and performs wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain the welded wire harness. It can be seen that this invention addresses the system delays (visual processing delay, communication delay, mechanical response delay, etc.) unique to high-speed motion scenarios by using delay compensation for feedforward correction, achieving spatiotemporal synchronization of vision-motion-execution, thereby effectively eliminating the common "cannot catch up" or "overshoot" phenomena under high-speed dynamics;The welding harness is subjected to quality analysis to obtain evaluation results, which include excessive deviation or no deviation. If the evaluation result is no deviation, the welding harness is confirmed as a precision welding harness. Based on the precision welding harness, high-speed welding alignment of the welding terminals is achieved. It is evident that this invention introduces a closed-loop quality evaluation and judgment mechanism in the final stage, forming a complete adaptive alignment closed loop of "acquisition-fusion-estimation-compensation-execution-verification". This not only ensures high precision in single welding but also provides continuous optimization basis for subsequent batch production. Therefore, this invention can significantly improve the accuracy, speed, stability, and automation level of high-speed harness welding alignment.

[0051] like Figure 2 The diagram shown is a functional block diagram of a high-speed wire harness welding alignment system based on multimodal vision provided in an embodiment of the present invention.

[0052] The high-speed wire harness welding alignment system 100 based on multimodal vision described in this invention can be installed in an electronic device. Depending on the functions implemented, the high-speed wire harness welding alignment system 100 based on multimodal vision may include an environment verification module 101, a multimodal perception module 102, a pose calculation module 103, and an execution verification module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and is stored in the memory of the electronic device.

[0053] The environment confirmation module 101 is used to confirm the receipt of the wire harness welding alignment command and confirm the wire harness welding alignment environment based on the wire harness welding alignment command. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. The multimodal perception module 102 is used to acquire images of the welding terminal based on the visual information acquisition unit to obtain a multimodal dataset; The multimodal dataset is preprocessed to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set. The pose calculation module 103 is used to obtain an ideal model, obtain a multimodal fusion objective function based on the ideal model, a two-dimensional feature point set, a three-dimensional feature point set and a key height feature point set, calculate the optimal rigid body transformation based on the multimodal fusion objective function, and calculate the pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose. Acquire a delay dataset, calculate translation delay compensation based on the delay dataset and motion control unit, and perform wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain a welded wire harness; The execution verification module 104 is used to perform quality analysis on the welding wire harness and obtain an evaluation result. The evaluation result includes excessive deviation or no deviation. If the evaluation result is no deviation, the welding wire harness is confirmed as a precision welding wire harness. Based on the precision welding wire harness, high-speed wire harness welding alignment of the welding terminals is realized.

[0054] In detail, the modules in the high-speed wire harness welding alignment system 100 based on multimodal vision described in this embodiment of the invention employ the same methods as described above during use. Figure 1 The method used is the same as the high-speed wire harness welding alignment method based on multimodal vision described above, and can produce the same technical effect, so it will not be repeated here.

[0055] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a high-speed wire harness welding alignment method based on multimodal vision, according to an embodiment of the present invention.

[0056] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a high-speed wire harness welding alignment method program based on multimodal vision.

[0057] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a high-speed wire harness welding alignment method program based on multimodal vision, but also to temporarily store data that has been output or will be output.

[0058] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a high-speed wire harness welding alignment method program based on multimodal vision) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0059] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0060] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0061] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0062] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0063] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0064] The high-speed wire harness welding alignment method program based on multimodal vision stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following: The system confirms receipt of the wire harness welding alignment command and confirms the wire harness welding alignment environment based on the command. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. The visual information acquisition unit acquires images of the welding terminal to obtain a multimodal dataset. The multimodal dataset is preprocessed to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set. Obtain an ideal model, and based on the ideal model, a two-dimensional feature point set, a three-dimensional feature point set, and a key height feature point set, obtain a multimodal fusion objective function, calculate the optimal rigid body transformation based on the multimodal fusion objective function, and calculate the pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose. Acquire a delay dataset, calculate translation delay compensation based on the delay dataset and motion control unit, and perform wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain a welded wire harness; The welding harness is subjected to quality analysis to obtain evaluation results, which include excessive deviation or no deviation. If the evaluation result is no deviation, the welding harness is identified as a precision welding harness. Based on the precision welding harness, high-speed welding alignment of the welding terminals is achieved.

[0065] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0066] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0067] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: The system confirms receipt of the wire harness welding alignment command and confirms the wire harness welding alignment environment based on the command. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. The visual information acquisition unit acquires images of the welding terminal to obtain a multimodal dataset. The multimodal dataset is preprocessed to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set. Obtain an ideal model, and based on the ideal model, a two-dimensional feature point set, a three-dimensional feature point set, and a key height feature point set, obtain a multimodal fusion objective function, calculate the optimal rigid body transformation based on the multimodal fusion objective function, and calculate the pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose. Acquire a delay dataset, calculate translation delay compensation based on the delay dataset and motion control unit, and perform wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain a welded wire harness; The welding harness is subjected to quality analysis to obtain evaluation results, which include excessive deviation or no deviation. If the evaluation result is no deviation, the welding harness is identified as a precision welding harness. Based on the precision welding harness, high-speed welding alignment of the welding terminals is achieved.

[0068] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0069] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0071] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A high-speed wire harness welding alignment method based on multimodal vision, characterized in that, The method includes: The system confirms receipt of the wire harness welding alignment command and confirms the wire harness welding alignment environment based on the command. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. The visual information acquisition unit acquires images of the welding terminal to obtain a multimodal dataset. The multimodal dataset is preprocessed to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set. Obtain an ideal model, and based on the ideal model, a two-dimensional feature point set, a three-dimensional feature point set, and a key height feature point set, obtain a multimodal fusion objective function, calculate the optimal rigid body transformation based on the multimodal fusion objective function, and calculate the pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose. Acquire a delay dataset, calculate translation delay compensation based on the delay dataset and motion control unit, and perform wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain a welded wire harness; The welding harness is subjected to quality analysis to obtain evaluation results, which include excessive deviation or no deviation. If the evaluation result is no deviation, the welding harness is identified as a precision welding harness. Based on the precision welding harness, high-speed welding alignment of the welding terminals is achieved.

2. The high-speed wire harness welding alignment method based on multimodal vision as described in claim 1, characterized in that, The process of acquiring images of the welding terminal based on the visual information acquisition unit to obtain a multimodal dataset includes: Obtain the welding station; when the welding terminal arrives at the welding station, obtain a visual information acquisition command. The visual information acquisition unit receives the visual information acquisition command, wherein the visual information acquisition unit includes a 2D image acquisition device, a 3D visual acquisition device, and a line laser contour acquisition device, and controls the visual information acquisition unit to perform the following operations based on the visual information acquisition command: The welding terminal is acquired using the 2D image acquisition device to obtain a two-dimensional image of the terminal. The 3D vision acquisition device acquires images of the welding terminal to obtain an original depth image. Based on the pre-built 3D reconstruction algorithm, cloud coordinates are constructed from the original depth image to obtain a 3D point cloud coordinate set. The line laser profile acquisition device performs line laser scanning along a preset welding path to obtain a one-dimensional height profile of the welding terminal. By summarizing the two-dimensional images, three-dimensional point cloud coordinate sets, and one-dimensional height contours of the terminals, a multimodal dataset is obtained.

3. The high-speed wire harness welding alignment method based on multimodal vision as described in claim 2, characterized in that, The step of preprocessing the multimodal dataset to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour includes: A unified time standard is obtained based on a pre-built system clock. The two-dimensional image, three-dimensional point cloud coordinate set, and one-dimensional height profile of the terminal are time-calibrated based on the unified time standard to obtain a calibrated two-dimensional image, a calibrated three-dimensional point cloud coordinate set, and a calibrated one-dimensional height profile. The calibrated 2D image is denoised based on a pre-built image filtering algorithm to obtain a denoised 2D image. The contrast of the denoised 2D image is then enhanced to obtain a pre-processed 2D image. For each calibration 3D point cloud coordinate in the calibration 3D point cloud coordinate set, the following operation is performed: Based on the calibrated 3D point cloud coordinates, obtain multiple surrounding 3D point cloud coordinates, calculate multiple Euclidean distances based on the multiple surrounding 3D point cloud coordinates, and calculate the mean and standard deviation of the distances based on the multiple Euclidean distances. Based on the mean and standard deviation of the distance, the mean distance and the mean standard deviation of the overall distance are calculated. The mean distance and the standard deviation of each calibration 3D point cloud coordinate in the calibration 3D point cloud coordinate set are compared with the mean distance and the mean standard deviation of the overall distance. If the mean distance is greater than the mean distance and the standard deviation is greater than the mean standard deviation of the overall distance, the calibration 3D point cloud coordinates are removed to obtain a preprocessed 3D coordinate set. The calibrated one-dimensional height profile is smoothed based on a pre-built smoothing algorithm to obtain a pre-processed height profile.

4. The high-speed wire harness welding alignment method based on multimodal vision as described in claim 3, characterized in that, The multimodal feature extraction unit extracts features from the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set, including: Edge detection is performed on the preprocessed 2D image based on a multimodal feature extraction unit and a pre-constructed two-dimensional feature detection algorithm to obtain two-dimensional contour edges. Feature points are then extracted from the two-dimensional contour edges to obtain two-dimensional feature points. Multiple principal curvatures are calculated based on a pre-constructed surface fitting algorithm and a pre-processed 3D coordinate set. The multiple principal curvatures are then filtered based on a preset curvature threshold, and a set of 3D feature points is obtained based on the pre-processed 3D coordinates corresponding to the filtered principal curvatures. The pre-built inflection point detection algorithm is used to detect inflection points in the preprocessed height contour to obtain a set of key height feature points.

5. The high-speed wire harness welding alignment method based on multimodal vision as described in claim 4, characterized in that, The process of obtaining a multimodal fusion objective function based on the ideal model, the two-dimensional feature point set, the three-dimensional feature point set, and the key height feature point set, and calculating the optimal rigid body transformation based on the multimodal fusion objective function, includes: Based on the ideal model, an ideal three-dimensional coordinate set is obtained. Based on the two-dimensional feature point set and the ideal three-dimensional coordinate set, feature point matching is performed to obtain a two-dimensional to three-dimensional feature point set, wherein the two-dimensional to three-dimensional feature point set includes M two-dimensional to three-dimensional feature points. Based on the ideal model, an ideal model point set and an ideal height value set are obtained. A multimodal fusion objective function is constructed based on the aforementioned 2D-to-3D feature point set, ideal model point set, ideal height value set, 2D feature point set, 3D feature point set, and key height feature point set. The 3D feature point set includes L 3D feature points, and the key height feature point set includes P key height feature points. The multimodal fusion objective function is shown below: in, Indicates the target score. Represents the rotation matrix. Represents the translation vector. Represents the pixel scaling factor. Represents the camera projection model. Represents the camera intrinsic parameter matrix. Represents the first digit in the set of 2D to 3D feature points. Two-dimensional to three-dimensional feature points Represents the first feature point in the two-dimensional feature point set. Two-dimensional feature points, Represents the set of three-dimensional feature points. Three-dimensional feature points, Let j represent the j-th ideal model point in the ideal model point set. Represents component functions, Represents the first key height feature point in the set. Key height feature points, Represents the set of ideal height values. An ideal height value, , and Indicates the weighting coefficient. , and Indicates subscript, Represents norm operations; The rotation matrix and translation vector in the multimodal fusion objective function are iterated until the objective score is less than a preset convergence threshold. Then, the rotation matrix and translation vector are integrated to obtain the optimal rigid body transformation, wherein the optimal rigid body transformation includes the optimal rotation matrix and the optimal translation vector.

6. The high-speed wire harness welding alignment method based on multimodal vision as described in claim 5, characterized in that, The calculation of pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose includes: Based on the ideal pose, obtain the ideal rotation matrix and the ideal translation vector; The rotation deviation is calculated based on the optimal rotation matrix and the ideal rotation matrix, and the displacement deviation is calculated based on the ideal translation vector and the optimal translation vector. The positional deviation is obtained by summing the rotational and displacement deviations.

7. The high-speed wire harness welding alignment method based on multimodal vision as described in claim 6, characterized in that, The process of acquiring the delay dataset and calculating the translation delay compensation number based on the delay dataset and the motion control unit includes: A delay dataset is obtained based on the motion control unit, wherein the delay dataset includes visual processing delay, communication delay and robot response delay, and the total delay time is calculated based on the delay dataset; The terminal transmission line speed and terminal transmission acceleration are obtained based on the motion control unit. The translational delay compensation number is calculated based on the total delay time, terminal transmission line speed, and terminal transmission acceleration, using the following formula: in, This represents the translation delay compensation number. Indicates the total delay time. Indicates transmission acceleration. This indicates the terminal transmission line speed.

8. The high-speed wire harness welding alignment method based on multimodal vision as described in claim 7, characterized in that, The process of welding a wire harness based on the welding execution unit, translation delay compensation number, and pose deviation to obtain a welded wire harness includes: The compensated displacement is calculated based on the translation delay compensation number and the displacement deviation in the pose deviation. The initial welding position is obtained based on the pre-constructed welding gun, and the initial welding position is adjusted based on the compensated displacement and rotation deviation to obtain the adjusted welding position. The welding execution unit controls the welding gun to weld the welding terminal at the adjusted welding position to obtain a welding wire harness.

9. The high-speed wire harness welding alignment method based on multimodal vision as described in claim 8, characterized in that, The quality analysis of the welding wire harness to obtain the evaluation results includes: Based on the visual information acquisition unit, images of the welding wire harness are acquired to obtain images after welding. The image after welding is segmented to obtain the welding point region. The pixel center coordinates are obtained based on the welding point region. The physical position deviation is calculated based on the pixel center coordinates and the preset ideal position. The area of ​​the welding region is calculated based on the welding point region. The physical position deviation and the area of ​​the welding area are compared with the preset position deviation threshold and the preset area of ​​the welding area, respectively. If the physical position deviation is less than the position deviation threshold and the area of ​​the welding area is within the area of ​​the welding area, the evaluation result is confirmed as no deviation; otherwise, the evaluation result is confirmed as excessive deviation.

10. A high-speed wire harness welding alignment system based on multimodal vision, characterized in that, The device includes: An environment confirmation module is used to confirm the receipt of a wire harness welding alignment command and to confirm the wire harness welding alignment environment based on the wire harness welding alignment command. The wire harness welding alignment environment includes a wire harness welding alignment system and welding terminals. The wire harness welding alignment system includes a visual information acquisition unit, a multimodal feature extraction unit, a motion control unit, and a welding execution unit. A multimodal perception module is used to acquire images of the welding terminal based on the visual information acquisition unit to obtain a multimodal dataset; The multimodal dataset is preprocessed to obtain a preprocessed 2D image, a preprocessed 3D coordinate set, and a preprocessed height contour. Based on the multimodal feature extraction unit, features are extracted from the preprocessed 2D image, the preprocessed 3D coordinate set, and the preprocessed height contour to obtain a 2D feature point set, a 3D feature point set, and a key height feature point set. The pose calculation module is used to obtain an ideal model, obtain a multimodal fusion objective function based on the ideal model, a two-dimensional feature point set, a three-dimensional feature point set, and a key height feature point set, calculate the optimal rigid body transformation based on the multimodal fusion objective function, and calculate the pose deviation based on the optimal rigid body transformation and the pre-constructed ideal pose. Acquire a delay dataset, calculate translation delay compensation based on the delay dataset and motion control unit, and perform wire harness welding based on the welding execution unit, translation delay compensation, and pose deviation to obtain a welded wire harness; The execution verification module is used to perform quality analysis on the welding wire harness and obtain evaluation results. The evaluation results include excessive deviation or no deviation. If the evaluation result is no deviation, the welding wire harness is confirmed as a precision welding wire harness. Based on the precision welding wire harness, high-speed wire harness welding alignment of the welding terminals is realized.

Images