Welding path planning method based on multi-modal fusion and deep learning

Abstract

The invention provides a welding path planning method based on multi-modal fusion and deep learning, and relates to the technical field of welding automation, the method comprises the following steps: synchronously acquiring three-dimensional coordinate information of a welding workpiece through a laser ranging module and a CCD visual sensor, and fusing laser height parameters and visual plane feature points; an initial path is generated based on the dynamic welding adaptation coefficient and the welding environment complexity, and laser errors, visual matching deviation and environment interference intensity are quantified; laser and visual data are dynamically weighted by using multi-modal feature fusion weights, spatial-temporal features are extracted through a deep learning model, and a welding track is optimized; and the six-axis mechanical arm is controlled to execute welding operation, weld quality is monitored in real time, and path parameters are adjusted in a closed-loop mode. According to the method, the problem of insufficient precision of a single sensor is solved through a multi-modal data complementation mechanism, sudden obstacles and temperature fluctuation are dealt with in combination with dynamic parameter calculation and real-time path correction, and the local path optimization capability is improved by using the convolutional neural network and a self-attention mechanism.

Description

Technical Field

[0001] The present invention relates to the field of welding automation technology, and in particular to a welding path planning method based on multimodal fusion and deep learning. Background Art

[0002] In traditional welding automation technology, path planning mainly relies on a single sensor (such as laser ranging or visual recognition) to obtain environmental information. However, the limitations of a single sensor are significant. For example, laser ranging is susceptible to noise on complex geometric surfaces or under dynamic interference, while visual sensors are prone to feature point mismatches in scenes with uneven lighting, reflections, or occlusions. The singleness of this data source leads to insufficient path planning accuracy, making it difficult to meet the high-precision and high-stability process requirements, especially in the welding of precision electronic components or the processing of complex industrial workpieces. In addition, existing technologies mostly use static path planning algorithms (such as traditional iterative nearest point algorithms), which cannot respond in real time to environmental changes during the welding process (such as sudden obstacles, temperature fluctuations, or material deformation), resulting in the accumulation of path deviations and ultimately affecting welding quality.

[0003] Existing welding systems often lack effective multimodal data fusion mechanisms. Data from different sensors is combined only through simple weighting or fixed rules, making it difficult to dynamically adapt to environmental changes. For example, in high-temperature environments, thermal expansion in the weld area can cause visual feature points to shift. Traditional methods are unable to dynamically adjust data weights based on real-time errors, resulting in an increased deviation between the fused path and actual requirements. Furthermore, path optimization algorithms often rely on preset rules or empirical parameters, lacking autonomous learning capabilities and struggling to cope with diverse welding tasks (such as those involving different materials, workpiece shapes, or process parameters). This results in weak system generalization capabilities and requires frequent manual intervention and adjustments.

[0004] Furthermore, traditional welding quality monitoring is typically based on a single indicator (such as temperature or weld width), lacking comprehensive evaluation and closed-loop feedback from multi-dimensional data. For example, insufficient penetration or current fluctuations during welding can lead to internal defects, but existing systems struggle to identify and adjust path parameters in a timely manner. Even if some systems introduce feedback mechanisms, refined dynamic control cannot be achieved due to data processing delays or a single adjustment strategy (such as fixed threshold triggering). In the long term, the lack of historical data accumulation and self-learning capabilities prevents continuous optimization of system performance, making it difficult to adapt to process upgrades or new scenarios. These issues collectively restrict the widespread application of welding automation technology in high-precision manufacturing. Summary of the Invention

[0005] In order to solve the technical problems in the prior art, the present invention provides a welding path planning method based on multimodal fusion and deep learning.

[0006] The technical solutions provided by the present invention are as follows: The present invention provides a welding path planning method based on multimodal fusion and deep learning, comprising: S1. Synchronously collect the three-dimensional coordinate information of the welding workpiece through the laser ranging module and the CCD vision sensor, wherein the laser ranging module obtains the height parameters of the workpiece surface, and the CCD vision sensor extracts the planar feature points of the welding area through image processing; S2. Preliminary planning of the welding path based on the dynamic welding adaptability coefficient (DWAC) and the welding environment complexity (WECC). The DWAC is used to characterize the adaptability to dynamic changes during welding, and the WECC is used to quantify the geometric complexity and interference factors of the welding area. S3. Use multimodal feature fusion weights to fuse laser ranging data and visual feature data to generate a high-precision three-dimensional welding path; S4. Locally optimize the path using a deep learning model, where the model is based on a convolutional neural network and an attention mechanism, with the fused multimodal data as input and the optimized welding trajectory as output; S5. Control the six-axis robotic arm to perform welding operations, monitor the welding quality in real time, and dynamically adjust the path parameters based on the feedback data.

[0007] Furthermore, the calculation formula of the dynamic welding adaptation coefficient DWAC is: ; in, Indicates the standard deviation of laser ranging error in millimeters; Indicates the calibration accuracy of the laser sensor, which is defined as the maximum allowable error value of the factory calibration; Indicates the standard deviation of the visual feature point matching error, in pixels; Indicates the resolution of the CCD camera in pixels per millimeter; Indicates the current welding speed in millimeters per second; 、 Weight coefficient calibrated by welding test, unitless.

[0008] Furthermore, the calculation formula of the welding environment complexity WECC is: ; in, Indicates the number of obstacles in the welding area, in pieces; Indicates the area of the welding area in square millimeters; 、 Indicates the maximum and minimum temperature fluctuations during the welding process, in degrees Celsius; Indicates the ambient temperature in degrees Celsius; 、 The coefficient is determined experimentally and has no unit.

[0009] Furthermore, the S3 further includes: The calculation of the multimodal feature fusion weight is achieved by weighted Kalman filtering, where: The weight of laser data is dynamically adjusted by DWAC, and the weight calculation formula is: ; The weight of the visual data is determined by the inverse of the feature point matching error, and the weight calculation formula is: ; in, is the weight of the laser data, is the weight of the visual data.

[0010] Furthermore, the S4 further includes: Input layer: receives fused multimodal data, with a dimension of ,in is the number of feature points, is the length of the time series; Convolution layer: 3 layers of 3D convolution kernels are used to extract spatial-temporal features. The convolution kernel size is ; Attention layer: The feature weights of key welding areas are enhanced through the self-attention mechanism, with 4 attention heads; Output layer: Generates the optimized welding path coordinate sequence with dimension .

[0011] Furthermore, the S5 further includes: The quality monitoring collects weld width, penetration and temperature data in real time through multi-dimensional sensors and calculates the welding quality index (WQI). When the WQI is lower than the threshold, path replanning is triggered. The calculation formula of the WQI is: ; in, Indicates the weld width in millimeters; Indicates the weld penetration depth in millimeters; Indicates the actual welding temperature in degrees Celsius; Indicates the theoretical soldering temperature in degrees Celsius.

[0012] Furthermore, the S2 further includes: The preliminary planning adopts an improved iterative closest point algorithm to generate an initial welding path by minimizing the Euclidean distance error between the target point cloud and the reference point cloud, where the Euclidean distance error threshold is 0.1 mm.

[0013] Furthermore, the S4 further includes: The deep learning model is trained using an inverse matching strategy, and adversarial training is performed by exchanging the input order of the reference image and the target image. The training cycle is 200 rounds and the batch size is 32.

[0014] Furthermore, the S3 further includes: The calibration accuracy of the laser ranging module is ±0.03 mm, the resolution of the CCD camera is 2448×2048 pixels, and the sampling period is 0.33 ms.

[0015] Furthermore, the S5 further includes: The dynamic adjustment path parameters include welding speed, robot end attitude angle and welding current, and the adjustment range is determined by the following formula: ; in, It represents the adjustment amplitude of the path parameters, dimensionless; It represents the scaling factor of experimental calibration and has no unit.

[0016] The beneficial effects brought about by the technical solution provided by the present invention include at least: (1) In the present invention, the problem of insufficient accuracy of single sensor data in traditional welding is effectively solved by combining the complementary advantages of laser ranging module and CCD vision sensor through multimodal data fusion technology. Laser ranging provides high-precision vertical distance measurement, and vision sensor captures plane feature points. The two realize data fusion through a dynamic weight distribution mechanism. The system automatically adjusts the weight of laser and vision data according to real-time environmental conditions. For example, in scenes with strong light interference or reflective surfaces, the weight of vision data is reduced, and the path planning is instead based on laser data. This dynamic fusion mechanism significantly improves the robustness of the welding path, ensuring that high-precision three-dimensional paths can still be generated under complex geometric surfaces, dynamic interference or uneven lighting conditions, and avoiding the path deviation caused by the failure of a single sensor in traditional methods.

[0017] (2) In the present invention, a path planning strategy based on the dynamic welding adaptation coefficient and the complexity of the welding environment solves the problem that the static algorithm cannot respond to environmental changes in real time. By introducing the dynamic welding adaptation coefficient to quantify the standardized influence of laser and visual errors, and dynamically adjusting the path parameters in combination with the welding speed, the system can adapt to the process requirements under different welding speeds. At the same time, the welding environment complexity parameter comprehensively evaluates the obstacle density and temperature fluctuations, triggering a real-time path correction mechanism. For example, when a sudden obstacle or abnormal temperature fluctuation is detected, the system uses a fast exploration random tree algorithm to generate a local detour path and optimizes the trajectory smoothness through a Bezier curve. This dynamic response mechanism ensures the continuity and stability of the welding process and avoids path interruption or quality degradation caused by sudden environmental changes in traditional static planning.

[0018] (3) In the present invention, the introduction of closed-loop feedback and long-term self-learning mechanisms breaks through the limitations of traditional welding systems that lack continuous optimization capabilities. By using multi-dimensional sensors to monitor weld width, penetration depth, and temperature data in real time and calculate the welding quality index, the system can accurately identify welding defects and trigger dynamic adjustments. The adjustment range of the path parameters is determined by the weighted dynamic welding adaptation coefficient and environmental complexity to achieve refined control. In addition, historical welding data is stored and used for offline model training. The convolution kernel weights and attention mechanism parameters are iteratively optimized through deep learning models, enabling the system to adapt to new materials, new processes, or complex scenarios. This closed-loop mechanism from single tasks to long-term learning not only improves the stability of welding quality, but also reduces the frequency of manual intervention, providing sustainable technical support for automation upgrades in intelligent manufacturing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic flow chart of a welding path planning method based on multimodal fusion and deep learning provided by an embodiment of the present invention; Figure 2 A schematic diagram of the data preprocessing and noise filtering process of a welding path planning method based on multimodal fusion and deep learning provided by an embodiment of the present invention; Figure 3 A schematic diagram of a real-time path correction process of a welding path planning method based on multimodal fusion and deep learning provided by an embodiment of the present invention; Figure 4A schematic diagram of the long-term performance monitoring and model self-learning process of a welding path planning method based on multimodal fusion and deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0022] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0023] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0024] In the embodiment of the present invention, sometimes the subscript is as follows It may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0025] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0026] Reference Manual Figure 1 , shows a flow chart of a welding path planning method based on multimodal fusion and deep learning provided by an embodiment of the present invention.

[0027] An embodiment of the present invention provides a welding path planning method based on multimodal fusion and deep learning. The processing flow may include the following steps: S1. The three-dimensional coordinate information of the welding workpiece is synchronously collected through the laser ranging module and the CCD vision sensor. The laser ranging module obtains the height parameters of the workpiece surface, and the CCD vision sensor extracts the planar feature points of the welding area through image processing.

[0028] Furthermore, during the initial stages of welding path planning, the three-dimensional coordinates of the welded workpiece are simultaneously collected using a laser ranging module and a CCD vision sensor. The laser ranging module accurately measures the workpiece surface height parameters, while the CCD vision sensor uses image processing technology to extract planar feature points in the weld area. This synchronized collection of multimodal data ensures the integrity and consistency of the three-dimensional spatial information, providing a reliable foundation for subsequent path planning. This complementary mechanism effectively reduces the impact of errors caused by a single sensor, especially in complex environments (such as reflective surfaces or dynamic interference).

[0029] It should be noted that if Figure 2 As shown in the figure, after the laser ranging module LDM and CCD vision sensor complete data acquisition, the system preprocesses the raw data to eliminate noise interference. This process is achieved through the following steps: S101, Laser Data Denoising: A sliding window filter algorithm is used to smooth laser ranging data and remove abnormal fluctuations caused by environmental vibration or surface reflections. For example, in high-speed welding scenarios, laser data may generate high-frequency noise due to robot arm vibration. The filtering algorithm effectively suppresses this noise by calculating the mean or median of the data within the window.

[0030] S102, Visual Data Enhancement: Enhance image contrast and remove isolated noise points through histogram equalization and morphological opening. For example, in low-light conditions, histogram equalization can increase image brightness, while opening (erosion followed by dilation) can eliminate small noise points, ensuring accurate feature point extraction.

[0031] S103, Spatiotemporal Alignment Verification: Verify the consistency of the timestamps of the laser and visual data. If the deviation exceeds a threshold (e.g., 0.5ms), data re-collection is triggered. For example, if the timestamp of the laser data lags behind the visual data, the system discards the current frame data and resynchronizes the collection to avoid 3D coordinate errors caused by spatiotemporal misalignment.

[0032] S2. Preliminary planning of the welding path is performed based on the dynamic welding adaptability coefficient DWAC and the welding environment complexity WECC. The DWAC is used to characterize the adaptability to dynamic changes in the welding process, and the WECC is used to quantify the geometric complexity and interference factors of the welding area.

[0033] Furthermore, the system performs preliminary welding path planning based on the dynamic welding adaptability coefficient (DWAC) and the welding environment complexity (WECC). DWAC quantifies the welding system's adaptability to complex environments by integrating laser ranging error, visual matching error, and dynamic changes in welding speed. WECC assesses the geometric complexity and interference intensity of the weld area by considering obstacle density and temperature fluctuations. The introduction of these two parameters enables path planning to dynamically respond to environmental changes. For example, it automatically adjusts path parameters in the presence of high obstacle density or drastic temperature fluctuations, ensuring that the initial path meets both geometric accuracy and process adaptability.

[0034] It should be noted that if Figure 3 As shown in the figure, after generating the initial welding path, the system dynamically corrects the path through real-time environmental monitoring: S201, Sudden Obstacle Detection: Utilizes CCD vision sensors to capture dynamic changes in the welding area in real time, combined with optical flow to identify moving obstacles. For example, when a temporary tool on the production line enters the welding area, optical flow detects the moving object through pixel displacement analysis and marks it as a potential obstacle.

[0035] S202, Local Path Replanning: Utilizes the Rapidly Exploring Random Tree (RRT) algorithm to generate local detour paths within obstacle avoidance constraints. RRT rapidly explores feasible paths through random sampling and collision detection, ensuring path adjustments are completed within milliseconds.

[0036] S203, Path Smoothing Optimization: A Bezier curve is used to fit the corrected path to ensure the continuity and smoothness of the robot's motion. For example, the corrected path may contain sharp corners. The Bezier curve generates a smooth trajectory using control points to avoid sudden stops or jitters in the robot.

[0037] S3. Use multimodal feature fusion weights to fuse laser ranging data and visual feature data to generate a high-precision three-dimensional welding path.

[0038] Furthermore, using multimodal feature fusion weights, laser ranging data and visual feature data are dynamically weighted and fused to generate a high-precision 3D welding path. The laser data weight is dynamically adjusted based on the DWAC, while the visual data weight is determined by the inverse of the feature point matching error. This weight distribution mechanism enables the system to automatically optimize the data fusion strategy under varying environmental conditions. For example, when the visual data exhibits significant errors due to lighting variations, the system reduces the visual weight and relies on the laser data to guide path generation, thus avoiding path deviations caused by the failure of a single sensor.

[0039] S4. Locally optimize the path through a deep learning model, where the model is built based on a convolutional neural network and an attention mechanism, with the input being the fused multimodal data and the output being the optimized welding trajectory.

[0040] Furthermore, a deep learning model is used to locally optimize the path. Built on a convolutional neural network and a self-attention mechanism, the model takes fused multimodal data as input and outputs an optimized welding trajectory. The convolutional layer extracts spatial-temporal features to capture dynamic changes in the weld area, while the self-attention mechanism enhances the feature weights of critical areas (such as dense welds or complex contours), enabling the model to focus on local regions that significantly influence path planning. This optimization not only improves the geometric accuracy of the path but also, through adaptive learning capabilities, addresses the diverse demands of different materials and welding conditions.

[0041] S5. Control the six-axis robotic arm to perform welding operations, monitor the welding quality in real time, and dynamically adjust the path parameters based on the feedback data.

[0042] Furthermore, when controlling the six-axis robotic arm to perform welding operations, the system monitors weld width, penetration, and temperature in real time and calculates a welding quality index (WQI). When the WQI falls below a preset threshold, path replanning is triggered, dynamically adjusting the welding speed, robotic arm attitude angle, and welding current. The adjustment range is determined by the weighted sum of the DWAC and WECC, ensuring rapid response to changes in complex environments while reducing unnecessary adjustments in stable environments. This closed-loop feedback mechanism not only ensures the stability of welding quality but also provides a foundation for long-term performance optimization through continuous data accumulation. For example, historical data can be used to train models in subsequent tasks to improve prediction accuracy.

[0043] It should be noted that if Figure 4 As shown in the figure, after the closed-loop control is completed, the system achieves long-term performance optimization through historical data accumulation and model iteration: S501. Welding Quality Database Construction: This database stores parameters (such as WQI, DWAC, and WECC) and result data (such as weld topography) for each welding task. For example, the database records welding parameters and quality indicators for different materials (such as aluminum alloy and stainless steel), providing a data foundation for subsequent analysis.

[0044] S502, Offline Model Training: The deep learning model is retrained monthly based on historical data to optimize convolution kernel weights and attention mechanism parameters. For example, for frequently occurring welding defects (such as porosity and lack of fusion), the model's learning capabilities are enhanced by adding new training samples.

[0045] S503, Parameter Adaptive Push: Synchronize the optimized model parameters to the online system and verify the new model's performance improvement in real-time tasks. For example, after the new model passes testing in a virtual simulation environment, it is automatically deployed to the production line to ensure seamless upgrades.

[0046] During the initial stages of welding path planning, a laser distance module (LDM) and a charge-coupled device (CCD) vision sensor simultaneously collect the three-dimensional coordinates of the welded workpiece. The LDM emits a high-precision laser beam and receives the reflected signal, measuring the workpiece's surface height parameters in real time. Its calibration accuracy reaches ±0.03 mm, ensuring reliable vertical measurements. The CCD vision sensor uses high-resolution image acquisition (2448×2048 pixels) combined with edge detection and feature point matching algorithms to extract planar feature points in the weld area, such as weld point coordinates, contour edges, and surface texture. The simultaneous acquisition of laser distance measurement and vision data is achieved through timestamp alignment technology, ensuring the spatiotemporal consistency of the three-dimensional coordinate information and preventing data misalignment due to acquisition delays.

[0047] It should be noted that the calibration accuracy of the laser ranging module and the resolution of the CCD camera are key parameters for system performance. The calibration accuracy of the laser sensor directly affects the accuracy of height measurement, while the resolution of the CCD determines the density and accuracy of planar feature point extraction. Furthermore, the laser ranging module's sampling period is 0.33 milliseconds, which meets the real-time requirements of high-speed welding scenarios. This multimodal data acquisition method not only covers the three-dimensional spatial information of the welding area but also enhances the robustness of the system through complementary mechanisms. For example, in conditions of strong light interference or reflective surfaces, visual data may be distorted. In these cases, the system relies primarily on laser data to maintain the stability of path planning.

[0048] It's understandable that the introduction of the Dynamic Welding Adaptation Coefficient (DWAC) and the Welding Environment Complexity Coefficient (WECC) is one of the core innovations of this invention. DWAC quantifies the system's adaptability to complex welding environments by integrating laser ranging error, visual matching error, and dynamic changes in welding speed. Its calculation formula is: ; in, Indicates the standard deviation of laser ranging error in millimeters; Indicates the calibration accuracy of the laser sensor, which is defined as the maximum allowable error value of the factory calibration; Indicates the standard deviation of the visual feature point matching error, in pixels; Indicates the resolution of the CCD camera in pixels per millimeter; Indicates the current welding speed in millimeters per second; 、 Weight coefficient calibrated by welding test, unitless.

[0049] In one possible implementation, the welding environment complexity (WECC) is quantified by the density of obstacles and the degree of temperature fluctuation in the welding area. The calculation formula is: ; in, Indicates the number of obstacles in the welding area, in pieces; Indicates the area of the welding area in square millimeters; 、 Indicates the maximum and minimum temperature fluctuations during the welding process, in degrees Celsius; Indicates the ambient temperature in degrees Celsius; 、 The coefficient is determined experimentally and has no unit.

[0050] Specifically, the weight of multimodal feature fusion is calculated using the Weighted Kalman Filter technology. The weight of the laser data is dynamically adjusted by DWAC, and the calculation formula is: ; The weight of the visual data is determined by the inverse of the feature point matching error, and the weight calculation formula is: ; By dynamically adjusting the weights of the two types of data, the system can achieve optimal data fusion in complex environments. For example, when the visual matching error increases, the system automatically reduces the visual weight and relies on laser data to guide path planning, thus avoiding path deviations caused by image distortion.

[0051] It should be noted that the design of the deep learning model combines the 3D Convolutional Neural Network (3D-CNN) and the Self-Attention Mechanism. The input layer receives the fused multimodal data with a dimension of ,in is the number of feature points, The convolution layer uses three layers of 3×3×3 convolution kernels to extract the spatiotemporal features of the welding area, such as the dynamic changes in geometric shapes and interference patterns. The self-attention layer uses four sets of attention heads to enhance the feature weights of key areas (such as dense welds or complex contours), thereby optimizing the key areas of path planning. The output layer generates an optimized welding path coordinate sequence with a dimension of , ensuring the continuity and accuracy of the path.

[0052] As you can understand, the deep learning model is trained using a reverse matching strategy. By swapping the input order of the reference and target images and combining it with adversarial training, the model's robustness to viewpoint changes is improved. The training cycle is set to 200 epochs with a batch size of 32 to balance model convergence speed and computing resource consumption. This training method enables the model to adapt to welding tasks under different viewpoints, such as oblique shooting or partial occlusion.

[0053] In one possible implementation, preliminary welding path planning utilizes an improved iterative closest point (ICP) algorithm. This algorithm generates an initial welding path by minimizing the Euclidean distance error between the target point cloud (real-time 3D data) and the reference point cloud (pre-set ideal path model). The Euclidean distance error threshold is set to 0.1 mm to ensure path planning accuracy while avoiding degradation of real-time performance due to over-optimization.

[0054] Specifically, quality monitoring uses multi-dimensional sensors to collect real-time data on weld width, penetration, and temperature, and calculates the Welding Quality Index (WQI). The calculation formula is: ; in, Indicates the weld width in millimeters; Indicates the weld penetration depth in millimeters; Indicates the actual welding temperature in degrees Celsius; Indicates the theoretical welding temperature in degrees Celsius. When the WQI falls below the preset threshold, the system automatically triggers path replanning, dynamically adjusting the welding speed, end-of-arm attitude angle, and welding current.

[0055] It should be noted that the dynamic adjustment range of the path parameters is determined by the following formula: ; in, It represents the adjustment amplitude of the path parameters, dimensionless; This is a unitless scaling factor derived from experimental calibration. For example, in highly complex environments (those with dense obstacles or drastic temperature fluctuations), the system responds quickly to environmental changes by increasing the adjustment amplitude. In stable environments, however, the adjustment amplitude is reduced to avoid path jitter.

[0056] As can be understood, the closed-loop control mechanism of the present invention achieves autonomous optimization of the welding process through a "perception-decision-execution-feedback" cycle. Multidimensional sensor data is not only used for real-time quality assessment but also fed back to the path planning module, forming a closed loop of dynamic adjustments. For example, if insufficient weld penetration is detected, the system will reduce the welding speed and replan the path to extend the local welding time, thereby improving the penetration rate.

[0057] In one possible implementation, the system's modular design supports flexible expansion. The laser ranging module can be replaced with other high-precision sensors (such as an ultrasonic rangefinder), and the CCD vision sensor can be upgraded to a high-speed global shutter camera to accommodate dynamic welding scenarios. The deep learning model's structure can also be adjusted, for example by increasing the number of attention heads to handle more complex welding tasks or by introducing a long short-term memory (LSTM) network to enhance time series data processing capabilities.

[0058] Specifically, the application scope of this invention is not limited to precision welding of electronic circuit boards but can also be expanded to high-precision fields such as automotive manufacturing and aerospace. For example, in the welding of new energy vehicle battery modules, the system can dynamically adjust the path to avoid thermal runaway. In aircraft engine blade repair, high-precision path planning can ensure that the mechanical properties of the weld meet stringent standards.

[0059] It should be noted that compared with traditional welding methods, this invention offers three advantages: first, deep fusion of multimodal data, overcoming the limitations of a single sensor; second, adaptive calculation of dynamic parameters, enhancing responsiveness to complex environments; and third, the combination of deep learning and classical algorithms, balancing path planning accuracy and efficiency. Future optimization directions include the introduction of reinforcement learning to enhance autonomous decision-making capabilities, or integrating digital twin technology to optimize actual parameters through virtual simulation.

[0060] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: (1) In the present invention, the problem of insufficient accuracy of single sensor data in traditional welding is effectively solved by combining the complementary advantages of laser ranging module and CCD vision sensor through multimodal data fusion technology. Laser ranging provides high-precision vertical distance measurement, and vision sensor captures plane feature points. The two realize data fusion through a dynamic weight distribution mechanism. The system automatically adjusts the weight of laser and vision data according to real-time environmental conditions. For example, in scenes with strong light interference or reflective surfaces, the weight of vision data is reduced, and the path planning is instead based on laser data. This dynamic fusion mechanism significantly improves the robustness of the welding path, ensuring that high-precision three-dimensional paths can still be generated under complex geometric surfaces, dynamic interference or uneven lighting conditions, and avoiding the path deviation caused by the failure of a single sensor in traditional methods.

[0061] (2) In the present invention, a path planning strategy based on the dynamic welding adaptation coefficient and the complexity of the welding environment solves the problem that the static algorithm cannot respond to environmental changes in real time. By introducing the dynamic welding adaptation coefficient to quantify the standardized influence of laser and visual errors, and dynamically adjusting the path parameters in combination with the welding speed, the system can adapt to the process requirements under different welding speeds. At the same time, the welding environment complexity parameter comprehensively evaluates the obstacle density and temperature fluctuations, triggering a real-time path correction mechanism. For example, when a sudden obstacle or abnormal temperature fluctuation is detected, the system uses a fast exploration random tree algorithm to generate a local detour path and optimizes the trajectory smoothness through a Bezier curve. This dynamic response mechanism ensures the continuity and stability of the welding process and avoids path interruption or quality degradation caused by sudden environmental changes in traditional static planning.

[0062] (3) In the present invention, the introduction of closed-loop feedback and long-term self-learning mechanisms breaks through the limitations of traditional welding systems that lack continuous optimization capabilities. By using multi-dimensional sensors to monitor weld width, penetration depth, and temperature data in real time and calculate the welding quality index, the system can accurately identify welding defects and trigger dynamic adjustments. The adjustment range of the path parameters is determined by the weighted dynamic welding adaptation coefficient and environmental complexity to achieve refined control. In addition, historical welding data is stored and used for offline model training. The convolution kernel weights and attention mechanism parameters are iteratively optimized through deep learning models, enabling the system to adapt to new materials, new processes, or complex scenarios. This closed-loop mechanism from single tasks to long-term learning not only improves the stability of welding quality, but also reduces the frequency of manual intervention, providing sustainable technical support for automation upgrades in intelligent manufacturing scenarios.

[0063] The above content is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0064] There are a few points to note: (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0065] (2) For the sake of clarity, the thickness of layers or regions in the drawings used to describe the embodiments of the present invention are exaggerated or reduced, that is, these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly on" or "under" the other element or intervening elements may be present.

[0066] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to form new embodiments.

[0067] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A welding path planning method based on multimodal fusion and deep learning, characterized in that: include: S1. Synchronously collect the three-dimensional coordinate information of the welding workpiece through the laser ranging module and the CCD vision sensor, wherein the laser ranging module obtains the height parameters of the workpiece surface, and the CCD vision sensor extracts the planar feature points of the welding area through image processing; S2. Preliminary planning of the welding path based on the dynamic welding adaptability coefficient (DWAC) and the welding environment complexity (WECC). The DWAC is used to characterize the adaptability to dynamic changes during welding, and the WECC is used to quantify the geometric complexity and interference factors of the welding area. S3. Use multimodal feature fusion weights to fuse laser ranging data and visual feature data to generate a high-precision three-dimensional welding path; S4. Locally optimize the path using a deep learning model, where the model is based on a convolutional neural network and an attention mechanism, with the fused multimodal data as input and the optimized welding trajectory as output; S5. Control the six-axis robotic arm to perform welding operations, monitor the welding quality in real time, and dynamically adjust the path parameters based on the feedback data.

2. A welding path planning method based on multimodal fusion and deep learning according to claim 1, characterized in that: include: The calculation formula of the dynamic welding adaptation coefficient DWAC is: ； in, Indicates the standard deviation of laser ranging error in millimeters; Indicates the calibration accuracy of the laser sensor, which is defined as the maximum allowable error value of the factory calibration; Indicates the standard deviation of the visual feature point matching error, in pixels; Indicates the resolution of the CCD camera in pixels per millimeter; Indicates the current welding speed in millimeters per second; 、 Weight coefficient calibrated by welding test, unitless.

3. A welding path planning method based on multimodal fusion and deep learning according to claim 1, characterized in that: include: The calculation formula of the welding environment complexity WECC is: ； in, Indicates the number of obstacles in the welding area, in pieces; Indicates the area of the welding area in square millimeters; 、 Indicates the maximum and minimum temperature fluctuations during the welding process, in degrees Celsius; Indicates the ambient temperature in degrees Celsius; 、 The coefficient is determined experimentally and has no unit.

4. The welding path planning method based on multimodal fusion and deep learning according to claim 1, characterized in that: Said S3 further comprises: The calculation of the multimodal feature fusion weight is achieved by weighted Kalman filtering, where: The weight of laser data is dynamically adjusted by DWAC, and the weight calculation formula is: ； The weight of the visual data is determined by the inverse of the feature point matching error, and the weight calculation formula is: ； in, is the weight of the laser data, is the weight of the visual data.

5. The welding path planning method based on multimodal fusion and deep learning according to claim 1, characterized in that: Said S4 further comprises: Input layer: receives fused multimodal data, with a dimension of ,in is the number of feature points, is the length of the time series; Convolution layer: 3 layers of 3D convolution kernels are used to extract spatial-temporal features. The convolution kernel size is ; Attention layer: The feature weights of key welding areas are enhanced through the self-attention mechanism, with 4 attention heads; Output layer: Generates the optimized welding path coordinate sequence with dimension .

6. The welding path planning method based on multimodal fusion and deep learning according to claim 1, characterized in that: Said S5 further comprises: The quality monitoring collects weld width, penetration and temperature data in real time through multi-dimensional sensors and calculates the welding quality index (WQI). When the WQI is lower than the threshold, path replanning is triggered. The calculation formula of the WQI is: ； in, Indicates the weld width in millimeters; Indicates the weld penetration depth in millimeters; Indicates the actual welding temperature in degrees Celsius; Indicates the theoretical soldering temperature in degrees Celsius.

7. The welding path planning method based on multimodal fusion and deep learning according to claim 1, characterized in that: Said S2 further comprises: The preliminary planning adopts an improved iterative closest point algorithm to generate an initial welding path by minimizing the Euclidean distance error between the target point cloud and the reference point cloud, where the Euclidean distance error threshold is 0.1 mm.

8. The welding path planning method based on multimodal fusion and deep learning according to claim 5, characterized in that: Said S4 further comprises: The deep learning model is trained using an inverse matching strategy, and adversarial training is performed by exchanging the input order of the reference image and the target image. The training cycle is 200 rounds and the batch size is 32.

9. The welding path planning method based on multimodal fusion and deep learning according to claim 4, characterized in that: Said S3 further comprises: The calibration accuracy of the laser ranging module is ±0.03 mm, the resolution of the CCD camera is 2448×2048 pixels, and the sampling period is 0.33 ms.

10. The welding path planning method based on multimodal fusion and deep learning according to claim 6, characterized in that: Said S5 further comprises: The dynamic adjustment path parameters include welding speed, robot end attitude angle and welding current, and the adjustment range is determined by the following formula: ； in, It represents the adjustment amplitude of the path parameters, dimensionless; It represents the scaling factor of experimental calibration and has no unit.

Images