Robot teaching-free track generation and real-time correction method based on multi-modal vision

Abstract

The invention relates to the technical field of computer vision, and discloses a robot teaching-free track generation and real-time correction method based on multi-modal vision, which comprises the following steps: synchronously acquiring data, and preprocessing; fusing multi-modal data, and extracting geometric, surface and temperature characteristics of the welding seam; simulating the visual field of a sensor and judging whether a welding seam area is severely shielded by a welding gun or a clamp; when the welding seam is shielded, reasoning and estimating the geometrical shape of the shielded part by using observation data of an unshielded area; generating and executing a section of local safe welding track according to the complete geometric estimation of the welding seam; when the sheltered area is temporarily visible, comparing the actual and reasoning results, and correcting a subsequent global trajectory; and the procedures are executed circularly till welding is completed, and the temperature is monitored in the whole process to adjust the technology. According to the method, an incremental trajectory generation and visual relocation mechanism based on occlusion perception and spatio-temporal reasoning is introduced, so that autonomous trajectory generation and real-time adjustment can be realized in a complex environment with limited visual information.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method for generating and correcting robot trajectories without teaching based on multimodal vision in real time. Background Technology

[0002] In the field of high-end equipment manufacturing, there exists a class of precision metal components with complex internal cavity structures, such as the irregularly shaped alloy cavities of engines. The manufacturing or repair of these components often requires multi-layered, multi-pass lap welding operations to be completed within extremely confined spaces. Due to the intricate internal structure of the cavity, the winding welding path, and the fact that the volume of the welding torch, wire feed mechanism, and necessary fixtures carried by the robot's end effector is not negligible relative to the workspace, these mechanisms themselves frequently and inevitably obstruct the direct line of sight to the weld area during the welding process. This obstruction is not static or global, but rather occurs dynamically with the robot's movement, creating continuous and unavoidable blind spots in localized areas.

[0003] Under conditions of dynamic local visual occlusion, traditional automation solutions that rely on acquiring complete and continuous global weld information for path planning face fundamental challenges. When the sensor's field of view is obstructed by its own mechanism, the system cannot directly acquire key geometric information such as the 3D morphology and bevel features of the obstructed section of the weld. If planning is delayed until the area is fully within the field of view, the robot needs to pause frequently, disrupting the continuity of the welding process and adversely affecting thermal cycle control. If inferences are made blindly based on historical information or theoretical models before occlusion, it cannot cope with real-time changes in the weld position and shape caused by actual workpiece machining errors, assembly deviations, and thermal deformation during welding, and the generated trajectory may not match the actual working conditions. Therefore, the following solutions are proposed to address these issues. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for robot trajectory generation and real-time correction based on multimodal vision, comprising the following steps: Simultaneously collect multimodal sensor data from the robot's end effector and robot body state data, and perform spatiotemporal registration and preprocessing on the multimodal sensor data; Based on the preprocessed multimodal data, local multimodal fusion features of the workpiece working area within the current field of view are extracted in real time; Based on the robot's end-effector pose, end-effector model, and known workpiece model information, the system detects the visual occlusion status in the current sensor field of view in real time and determines the current effective perception area. If it is determined that the current state is in a state of severe visual occlusion, the incremental state estimation process is initiated: using the established spatiotemporal dynamic prior model of weld geometry, and based on the local multimodal fusion features extracted in the non-occluded area at the current and historical times, the geometric state of the operation path in the occluded area is estimated through a probabilistic reasoning framework. Based on the complete geometric state of the operation path, including the estimated state of the occluded area, and combined with robot motion constraints, a local operation trajectory segment starting from the current position and ending within the safe area is planned and executed in real time. During the execution of local operation trajectory segments, occlusion detection is continuously performed. When the previously occluded operation area is detected to enter the effective perception area, a rapid relocation correction process is triggered: accurately extract the real geometric features of the area that has just become visible, calculate the state offset between it and the historical inference estimate, and construct a spatiotemporal propagation correction field based on the offset to globally correct and adjust the geometric state estimate of subsequent unoperated paths. Repeat the above steps until the task is completed.

[0005] Preferably, the multimodal sensor data includes at least point cloud data acquired by a structured light 3D vision sensor, color image data acquired by an industrial camera, and temperature distribution data acquired by an infrared thermal imager; the robot body state data includes at least the six-dimensional pose of the end effector.

[0006] Preferably, real-time extraction of local multimodal fusion features specifically includes: processing 3D point cloud data, color image data and temperature distribution data separately through parallel deep neural network branches, extracting geometric features, appearance features and thermal features of the working area, and fusing these features into a unified local multimodal fusion feature vector.

[0007] Preferably, the determination of the current effective sensing area and the severe visual occlusion state is as follows: the effective sensing area is obtained by simulating the sensor field of view and performing Boolean operation with the workpiece model; when the proportion of the area belonging to the theoretical working path within the effective sensing area is lower than a certain threshold, it is determined to be a severe visual occlusion state, and the parameterized interval of the occluded path segment is recorded.

[0008] Preferably, the spatiotemporal dynamics prior model constrains the geometric state of the operation path to maintain smooth changes between spatially adjacent locations and temporally adjacent moments; the probabilistic inference framework adopts a particle filtering algorithm, where each particle represents a set of assumptions about the geometric state of the occluded region, and the particle weight update depends on the likelihood probability between its corresponding complete path state and the actual observed features of the unoccluded region.

[0009] Preferably, in the weight update step of the particle filter algorithm, the calculation of particle weights does not depend on the direct observation of the occluded region, but is based on the consistency between the complete path geometry assumed by the particle and the multimodal fusion features extracted from the actual observation in the unoccluded region at the current moment for evaluation and update.

[0010] Preferably, the local operation trajectory segment is planned in real time, and its planning endpoint is restricted to the boundary of the current effective sensing area or the area confirmed as safe by state estimation, and the planning step size is fixed.

[0011] Preferably, the condition for triggering the rapid relocation correction process is that the proportion of the previously occluded working area entering the current effective sensing area exceeds a certain threshold.

[0012] Preferably, a spatiotemporal propagation correction field is constructed and a global correction is performed. Specifically, the intensity of the correction field is a function of the state offset and decreases exponentially with the increase of the arc length distance from the relocation trigger point and the spatial Euclidean distance. The correction field is used to perform weighted correction on the geometric state estimates of all subsequent unworked path points, and the unexecuted trajectory parts of the robot are replanned accordingly.

[0013] Preferably, during the cyclic execution process, the temperature field during the operation is continuously monitored using infrared thermal imager data. When the temperature or temperature rise rate is detected to exceed the preset safety threshold, a speed adjustment command is automatically introduced in the trajectory generation or correction stage to achieve process monitoring.

[0014] The present invention has the following beneficial effects: (1) By introducing an incremental trajectory generation and visual relocalization mechanism based on occlusion perception and spatiotemporal reasoning, this invention can achieve autonomous trajectory generation and real-time adjustment in complex environments with limited visual information. The system does not rely on the complete visual information of the occluded area, but uses local spatiotemporal data collected by multimodal sensors to infer the geometric features of the weld in real time through an incremental Bayesian state estimation model and generate an executable safe trajectory segment. This design improves the system's ability to work continuously under conditions of partial information loss and enhances its adaptability to extremely limited spatial environments. At the same time, through a rapid relocalization and correction mechanism triggered by a brief visual window, reasoning deviations can be corrected in time to maintain the global consistency of the trajectory, thereby improving the robustness and reliability of the overall operation. (2) This invention enhances the safety and real-time performance of trajectory generation through real-time multimodal data fusion and local trajectory planning; the system continuously collects and processes data during robot movement, dynamically determines the perceptible area through occlusion detection, and generates a safe trajectory segment based on the current state, taking into account joint constraints and collision constraints; when environmental changes cause visual occlusion to be temporarily removed, the system quickly triggers the relocation algorithm, uses the observation data to correct the previous inference results, and updates the subsequent trajectory; this real-time feedback and adjustment mechanism reduces the risk of execution errors caused by insufficient information or environmental changes, ensures the stability and safety of the operation process, and improves the system's responsiveness in dynamic environments; (3) This invention helps to improve the control accuracy and quality consistency of the welding process through multimodal data fusion and heat input monitoring; the system integrates information from structured light 3D vision, high dynamic range camera and infrared thermal imager to comprehensively extract the geometric, texture and temperature features of the weld for trajectory generation and correction; in incremental inference, multimodal observation provides richer state constraints and improves the accuracy of estimation; at the same time, infrared thermal imager data is used to monitor welding heat input, and when an abnormal temperature is detected, the system automatically adjusts the trajectory parameters to control the thermal effect, thereby maintaining stable process conditions; this helps to reduce welding defects, promote the formation of uniform weld structure and improve overall quality. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1, please refer to Figure 1 This invention relates to a method for robot trajectory generation and real-time correction without teaching based on multimodal vision, comprising the following steps: Simultaneously collect multimodal sensor data from the robot's end effector and robot body state data, and perform spatiotemporal registration and preprocessing on the multimodal sensor data; Based on the preprocessed multimodal data, local multimodal fusion features of the workpiece working area within the current field of view are extracted in real time; Based on the robot's end-effector pose, end-effector model, and known workpiece model information, the system detects the visual occlusion status in the current sensor field of view in real time and determines the current effective perception area. If it is determined that the current state is in a state of severe visual occlusion, the incremental state estimation process is initiated: using the established spatiotemporal dynamic prior model of weld geometry, and based on the local multimodal fusion features extracted in the non-occluded area at the current and historical times, the geometric state of the operation path in the occluded area is estimated through a probabilistic reasoning framework. Based on the complete geometric state of the operation path, including the estimated state of the occluded area, and combined with robot motion constraints, a local operation trajectory segment starting from the current position and ending within the safe area is planned and executed in real time. During the execution of local operation trajectory segments, occlusion detection is continuously performed. When the previously occluded operation area is detected to enter the effective perception area, a rapid relocation correction process is triggered: accurately extract the real geometric features of the area that has just become visible, calculate the state offset between it and the historical inference estimate, and construct a spatiotemporal propagation correction field based on the offset to globally correct and adjust the geometric state estimate of subsequent unoperated paths. Repeat the above steps until the task is completed.

[0019] Multimodal sensor data includes at least point cloud data acquired by a structured light 3D vision sensor, color image data acquired by an industrial camera, and temperature distribution data acquired by an infrared thermal imager; robot body state data includes at least the six-dimensional pose of the end effector.

[0020] Real-time extraction of local multimodal fusion features specifically includes: processing 3D point cloud data, color image data and temperature distribution data separately through parallel deep neural network branches, extracting geometric features, appearance features and thermal features of the working area, and fusing these features into a unified local multimodal fusion feature vector.

[0021] The determination of the current effective sensing area and the severe visual occlusion status is as follows: the effective sensing area is obtained by simulating the sensor field of view and performing Boolean operation with the workpiece model; when the proportion of the area belonging to the theoretical working path within the effective sensing area is lower than a certain threshold, it is determined to be a severe visual occlusion status, and the parameterized interval of the occluded path segment is recorded.

[0022] The spatiotemporal dynamics prior model constrains the geometric state of the operation path to maintain smooth changes between spatially adjacent locations and temporally adjacent moments; the probabilistic inference framework adopts a particle filtering algorithm, where each particle represents a set of assumptions about the geometric state of the occluded region, and the particle weight update depends on the likelihood probability between its corresponding complete path state and the actual observed features of the unoccluded region.

[0023] In the weight update step of the particle filter algorithm, the calculation of particle weights does not depend on the direct observation of the occluded region, but is based on the consistency between the complete path geometry assumed by the particle and the multimodal fusion features extracted from the actual observation in the unoccluded region at the current moment for evaluation and update.

[0024] The local operation trajectory segment is planned in real time, and its planning endpoint is restricted to the boundary of the current effective sensing area or the area confirmed as safe by state estimation, and the planning step size is fixed.

[0025] The condition that triggers the fast relocation correction process is that the proportion of previously occluded working areas entering the current effective sensing area exceeds a certain threshold.

[0026] A spatiotemporal propagation correction field is constructed and a global correction is performed. Specifically, the intensity of the correction field is a function of the state offset and decays exponentially with the increase of the arc length distance from the relocation trigger point and the spatial Euclidean distance. The correction field is used to perform weighted correction on the geometric state estimates of all subsequent unworked path points, and the unexecuted trajectory parts of the robot are replanned accordingly.

[0027] During the cyclic execution, the temperature field during the operation is continuously monitored using infrared thermal imager data. When the temperature or temperature rise rate is detected to exceed the preset safety threshold, a speed adjustment command is automatically introduced in the trajectory generation or correction stage to achieve process monitoring.

[0028] One specific application of this embodiment is: Step S100: Synchronous Acquisition and Preprocessing of Multimodal Data A multimodal sensor module is rigidly mounted on the robot's end effector (welding torch). This module includes a structured light 3D vision sensor, a high dynamic range (HDR) industrial camera, and a short baseline infrared thermal imager. Upon robot controller startup, the aforementioned sensors are synchronously triggered. The structured light sensor emits a specific coded pattern onto the workpiece surface, acquiring the current field-of-view point cloud. HDR camera captures RGB images of the weld area. Infrared thermal imager acquires temperature distribution matrix of weld seam and its adjacent area. Simultaneously, the six-dimensional pose of the end effector is read in real time from the robot controller. and joint angle data; establish a unified spatiotemporal coordinate system, and , , Through the pre-calibrated external parameter matrix Registered to the robot's base coordinate system; Perform voxel grid downsampling and statistical outlier removal filtering; The application uses fast segmentation based on prior colors of the weld area to extract binary masks of possible weld areas. .

[0029] Step S200: Real-time extraction of local weld features based on multimodal fusion The preprocessed multimodal data is input into a lightweight feature extraction network (such as a cropped PointNet parallel branch of CNN); point cloud Input the PointNet branch and output the 3D geometric feature vector of the weld section within the current view. It includes information such as the cross-section centerline, bevel angle, and depth; At the same time, With mask The superimposed vectors are input into a CNN branch, and the output vectors are surface texture and melt pool morphology feature vectors. Infrared temperature data After normalization, its gradient magnitude map is calculated, and the statistical characteristics of temperature distribution (mean, variance, and gradient direction consistency) of the region centered on the weld are extracted. ; Ultimately, Concatenate to form the local multimodal fusion feature vector at the current time step .

[0030] Step S300: Visual occlusion detection and perceptible area determination The system is based on the current robot end-effector pose. Given the known 3D model of the welding torch fixture and the sensor's field of view, the sensor's field of view is simulated and calculated in 3D space using the ray projection method. ;Will Perform Boolean operations with the known workpiece CAD model (containing only a rough outline of the part to be welded and the theoretical weld location) obtained from the robot controller; define the effective sensing area: In the formula, for The effective sensing area of ​​the time sensor; for The field of view (three-dimensional spatial region) of the time sensor. The working surface area obtained from the CAD model; For morphological dilation operators; This is a fixed distance for the expansion operation, used to compensate for errors between the model and the actual object; Will Mapping back to image coordinates yields an effective perceptual mask. ;like The proportion of pixels belonging to the theoretical weld seam area is lower than the preset threshold. If the percentage is 40%, then the weld is determined to be severely obscured in the current field of view, and the parametric range of the obscured weld segment on the CAD model is recorded. ,in This is the arc length parameter along the theoretical weld path.

[0031] Step S400: Geometric inference of the shading zone weld based on incremental Bayesian estimation When the system determines that it has entered a severe occlusion state (step S300 output is true), this step is initiated; the system maintains a weld geometry state estimation model; this model discretizes the entire weld into... One arc length position , The state variables at each location are key geometric parameters of the weld section, such as center point offset. (Perpendicular to the weld direction), depth ,width , recorded as ; Establishing a spatiotemporal dynamic prior: Assuming that the weld geometry changes smoothly along the arc length and over time, a joint spatiotemporal prior model is constructed: In the formula, It is a conditional probability function; for All moments A set of state vectors for each position; All moments A set of state vectors for each position; for Multimodal fusion feature vector at time step; Proportional to; It is a natural exponential function; This is a spatial smoothness weighting coefficient that controls the consistency of state changes between adjacent locations; This is the time smoothness weighting coefficient, which controls the consistency of state changes between adjacent time points; Let L be the Euclidean norm (L2 norm) of the vector. for Time of the first The state vector at each position; for Time of the first The state vector at each position; this prior constraint ensures that the geometric state at adjacent positions and adjacent times should not change abruptly; Constructing multimodal observation likelihood: for the current time Unshielded weld area (i.e.) ), its geometric state Features that can be directly extracted from step S200 The middle decoder obtains the result (through a trained shallow decoder network); the observation likelihood is defined as: In the formula, It is the probability density function of a multivariate normal distribution; for Geometrically related sub-vectors in the time-varying multimodal fusion features; for The set of geometric states of the weld area that can be directly observed at any time; A known function that maps the geometric state to the feature space (implemented by the trained decoder); The observation noise covariance matrix describes the uncertainty of the observation data; Incremental Bayesian update: for occluded areas status It itself has no direct observation; the system uses the Rao-Blackwellized particle filter framework for processing; specifically: Particle: Each particle A set of assumptions representing the geometric state of the occluded region ; Prediction: Based on the spacetime dynamics prior, the state of each particle is propagated: ; Update: Weight updates rely not only on direct observations of the occluded region (which does not exist), but also on the observation consistency of the unoccluded region; calculate the complete weld geometry represented by each particle. (Including occluded and unoccluded areas) Generate the features of the currently observed unoccluded area. The likelihood probability is used as the weight of the particle. : In the formula, For the first The weight value of each particle; This is the index of the particles in the particle filter; For the first The geometric state of the unobstructed region corresponding to each particle; Estimation: Final geometric state estimation of the occluded region The following is obtained by weighted averaging the states of all particles: ; In the formula, for The final estimate of the geometric state of the occluded region at any given time; For the first Each particle represents the geometric state of the occluded region; The key to this step is to use reliable observations of the unobstructed area, bridged by a spatiotemporal dynamics model, to inversely constrain and infer the geometric state of the obstructed area, thus achieving incremental state estimation under partial information loss.

[0032] Step S500: Generation and execution of safe trajectory segments Complete weld geometry estimation based on the output of step S400 (Including the inferred occlusion area), as well as the current robot's joint constraints and collision constraints with the workpiece (based on the CAD model), to perform local trajectory planning; The planning starting point is the current position of the robot's end effector. ; The planning endpoint is set as follows: advance one fixed planning step from the current position along the theoretical weld direction. (e.g., 20mm), but not exceeding the current effective sensing area. Set back a safe distance from the boundary. (e.g., 3mm) position; this ensures that the endpoint of the generated trajectory segment is located within an area that is "confirmed to be safe" or "inferred to be safe even though it is occluded"; Between the starting point and the ending point, To guide this process, a three-dimensional spatial trajectory is generated that keeps the welding torch in optimal posture (e.g., perpendicular to the local tangent of the weld and at a specific angle to the workpiece surface) and speed. After the trajectory is converted into a joint space trajectory, it is sent to the robot controller for execution. The system also records whether the local trajectory is generated based on direct observation (step S200) or inference estimation (step S400), and associates it with a confidence level. The confidence level is determined by the effective number of particles or the weight variance of the particle filter in step S400.

[0033] Step S600, Brief Visual Window Trigger and Fast Visual Relocation The local trajectory generated in step S500 of robot execution During the process, the occlusion detection in step S300 is continuously performed; once a previously occluded weld area is detected... Any part entering the currently effective sensing area And its visible ratio exceeds the threshold. (e.g., 60%), the system immediately triggers a rapid visual repositioning process; Data capture: At the moment of triggering, cache the multimodal data at that instant. , , and robot pose ; Precise feature extraction and matching: for High-precision feature extraction was performed on the weld seam area that was transformed into a visible region to obtain its true geometric features. At the same time, retrieve the previous records from the system logs. The geometric state estimate of the region obtained through inference in step S400 at the time when the region is first determined to be occluded. ; Calculate state offset and global correction: Define state offset In the formula, This offset represents the current moment that triggers visual relocalization; it reflects the cumulative bias caused by model errors, inference uncertainties, etc.; the system employs a correction field algorithm based on spatiotemporal propagation to adjust the locally observed offset. Propagation to the subsequent unwelded portion (arc length) of the entire weld seam ,in It is the arc length corresponding to the current relocation point); correction field Defined as: In the formula, For the position of arc length The corrected field vector at that location; The arc length parameter is the target position; This refers to the arc length parameter corresponding to the current relocation point; arc length The coordinate vector of the corresponding weld location in three-dimensional space; arc length The coordinate vector of the corresponding weld location in three-dimensional space; The attenuation coefficient in the arc length domain controls the attenuation rate of the correction effect along the arc length direction of the weld. This is the spatial domain attenuation coefficient, which controls the attenuation rate of the correction effect in three-dimensional space; This formula means that the closer an unwelded point is to the relocation point (in space and time), the greater the correction force it receives; Then update the geometry estimates for these unwelded points: ; In the formula, For the updated version Geometric state vectors at each location; For the first time before the update Geometric state vectors at each location; In the first Arc length at each position The corrected field vector calculated at that location; Real-time trajectory adjustment: based on updated global geometry. The system rapidly replans the global trajectory that the robot has not yet executed and that is affected by the correction field. The replanning starts from the current actual position of the robot and is constrained by the corrected geometric state to generate a new trajectory segment that smoothly transitions to the new target. This segment immediately and seamlessly replaces the corresponding part in the original plan and is sent to the robot controller for execution. This relocation and correction mechanism realizes the verification of the previous inference results and the online consistency optimization of the global trajectory in a very short time (such as within a control cycle) using a short visual window.

[0034] Step S700, cyclic execution and process monitoring Repeat steps S200 to S600 to form a closed loop of perception / occlusion determination-inference / local generation-execution-relocation correction until the welding task is completed; throughout the process, infrared thermal imager data It is used not only for feature extraction (step S200), but also continuously for monitoring welding heat input; the system presets a temperature upper limit. and temperature rise rate threshold If the temperature at a certain point exceeds Or the local temperature rise rate exceeds In trajectory generation (step S500) or correction (step S600), the system will automatically introduce a brief pause or deceleration command into the trajectory segment corresponding to the point to indirectly adjust the process parameters and ensure welding quality. All collected data (multimodal data, robot status, generated trajectory, inference confidence, relocation events and corrections) are recorded synchronously for subsequent analysis and model optimization.

[0035] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for robot trajectory generation and real-time correction without teaching based on multimodal vision, characterized in that, Includes the following steps: Simultaneously collect multimodal sensor data from the robot's end effector and robot body state data, and perform spatiotemporal registration and preprocessing on the multimodal sensor data; Based on the preprocessed multimodal data, local multimodal fusion features of the workpiece working area within the current field of view are extracted in real time; Based on the robot's end-effector pose, end-effector model, and known workpiece model information, the system detects the visual occlusion status in the current sensor field of view in real time and determines the current effective perception area. If it is determined that the current state is in a state of severe visual occlusion, the incremental state estimation process is initiated: using the established spatiotemporal dynamic prior model of weld geometry, and based on the local multimodal fusion features extracted in the non-occluded area at the current and historical times, the geometric state of the operation path in the occluded area is estimated through a probabilistic reasoning framework. Based on the complete geometric state of the operation path, including the estimated state of the occluded area, and combined with robot motion constraints, a local operation trajectory segment starting from the current position and ending within the safe area is planned and executed in real time. During the execution of the local operation trajectory segment, occlusion detection is continuously performed. When the previously occluded operation area is detected to enter the effective perception area, a fast relocation correction process is triggered: the real geometric features of the newly visible area are accurately extracted, the state offset between the area and the historical inference estimate is calculated, and a spatiotemporal propagation correction field is constructed based on the offset to globally correct and adjust the geometric state estimate of subsequent unoperated paths. Repeat the above steps until the task is completed.

2. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 1, characterized in that: The multimodal sensor data includes at least point cloud data acquired by a structured light 3D vision sensor, color image data acquired by an industrial camera, and temperature distribution data acquired by an infrared thermal imager; the robot body state data includes at least the six-dimensional pose of the end effector.

3. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 1, characterized in that: The real-time extraction of local multimodal fusion features specifically includes: processing three-dimensional point cloud data, color image data and temperature distribution data through parallel deep neural network branches, extracting geometric features, appearance features and thermal features of the working area, and fusing these features into a unified local multimodal fusion feature vector.

4. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 1, characterized in that: The determination of the current effective sensing area and the severe visual occlusion state is specifically as follows: the effective sensing area is obtained by simulating the sensor field of view and performing Boolean operation with the workpiece model; when the proportion of the area belonging to the theoretical working path within the effective sensing area is lower than a certain threshold, it is determined to be a severe visual occlusion state, and the parameterized interval of the occluded path segment is recorded.

5. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 1, characterized in that: The spatiotemporal dynamics prior model constrains the geometric state of the operation path to maintain smooth changes between spatially adjacent positions and temporally adjacent moments; the probabilistic inference framework adopts a particle filtering algorithm, where each particle represents a set of assumptions about the geometric state of the occluded region, and the particle weight update depends on the likelihood probability between its corresponding complete path state and the actual observed features of the unoccluded region.

6. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 5, characterized in that: In the weight update step of the particle filter algorithm, the calculation of particle weights does not depend on direct observation of the occluded region, but is based on the consistency between the complete path geometry assumed by the particle and the multimodal fusion features extracted from the actual observation in the unoccluded region at the current moment for evaluation and update.

7. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 1, characterized in that: The real-time planned local operation trajectory segment has its planning endpoint restricted within the boundary of the current effective sensing area or within an area confirmed as safe by state estimation, and the planning step size is fixed.

8. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 1, characterized in that: The condition for triggering the rapid relocation correction process is that the proportion of previously occluded working areas entering the current effective sensing area exceeds a certain threshold.

9. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 1, characterized in that: The construction of the spatiotemporal propagation correction field and the global correction are specifically as follows: the intensity of the correction field is a function of the state offset, and it decays exponentially with the increase of the arc length distance from the relocation trigger point and the spatial Euclidean distance; the correction field is used to perform weighted correction on the geometric state estimates of all subsequent unworked path points, and the unexecuted trajectory parts of the robot are replanned accordingly.

10. The method for robot trajectory generation and real-time correction based on multimodal vision according to claim 1, characterized in that: During the cyclic execution process, the temperature field during the operation is continuously monitored using infrared thermal imager data. When the temperature or temperature rise rate is detected to exceed the preset safety threshold, a speed adjustment command is automatically introduced in the trajectory generation or correction stage to achieve process monitoring.

Images