Self-adaptive spraying mechanical arm based on multi-modal perception and control method thereof

Abstract

The invention provides a self-adaptive spraying mechanical arm based on multi-mode perception and a control method of the self-adaptive spraying mechanical arm. The method comprises the steps that multi-source perception data of a to-be-repaired area is obtained through a dual-mode vision system and a laser scanning device, a two-dimensional repairing area segmentation map is generated, and space registration and local geometric modeling are achieved in combination with three-dimensional point cloud; on the basis of local geometric features extracted by the model, retrieving matched process parameters from the knowledge graph, generating a digital spraying instruction set containing tracks, postures and dynamic process parameters, and performing simulation verification in a digital twin environment; the mechanical arm state and the virtual model are synchronized in real time in the execution process, the paint film thickness is dynamically monitored, and compensation adjustment is triggered; and after spraying is completed, the coating quality is detected, and a result is fed back to the knowledge graph to update the parameter mapping relation. According to the method, closed-loop control from sensing, decision making, execution, evaluation to learning is achieved, and the spraying uniformity, the self-adaptability and the intelligent level under the complex working condition are improved.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent spraying robot system integration and adaptive control, specifically to an adaptive spraying robot arm control method based on multimodal perception and a robot arm using the control method. Background Technology

[0002] In the current field of industrial automated repair, spraying repair of complex surface defects often relies on human experience or robotic arm control systems based on single sensors. This generally suffers from low target recognition accuracy, poor modeling integrity, and weak adaptability to process parameters. Traditional methods typically employ independent 2D vision or 3D laser scanning equipment for defect detection, making it difficult to simultaneously acquire accurate defect locations and detailed surface morphology information. This limits the ability to perceive irregular, discontinuous, or low-contrast areas, affecting the accuracy and coverage integrity of subsequent repair paths. Although some systems have attempted to introduce multi-source data fusion strategies, the lack of effective cross-modal association mechanisms makes it difficult to bridge the semantic gap between heterogeneous data, failing to generate a unified and high-fidelity spatial representation and limiting the ability to understand the environment under complex working conditions. Furthermore, existing modeling methods often neglect spatial consistency registration between 2D texture features and 3D structures when converting perception results into local geometric models, resulting in insufficient model accuracy and difficulty in supporting refined path planning requirements. At the process decision level, most systems still use fixed parameter templates or simple conditional judgments to set spraying trajectories, attitudes, and process parameters, lacking dynamic response capabilities to local geometric features and failing to achieve adaptive adjustments for different defect types and substrate conditions. Meanwhile, the actual spraying process generally lacks a pre-simulation verification stage, resulting in high trial-and-error costs, significant material waste, and an inability to predict coating uniformity and edge coverage. In the control execution phase, traditional systems are mostly open-loop architectures, lacking real-time feedback and linkage between the robotic arm's motion and the control system. Physical execution deviations easily accumulate, affecting overall operational stability. After spraying, quality assessment relies heavily on offline detection methods, resulting in long feedback cycles, making it difficult to promptly correct process parameters and drive continuous system optimization. Therefore, the existing technological system has not yet formed a closed-loop intelligent control mechanism from environmental perception to process execution and quality feedback, restricting the widespread application and performance improvement of automated spraying systems in complex and dynamic industrial scenarios. Summary of the Invention

[0003] Based on this, in order to solve the technical problems of existing spraying robotic arms having insufficient ability to perceive the surface morphology of the target under complex working conditions, relying on human experience for spraying parameter adjustment, and poor coating uniformity due to low adaptive control accuracy, an adaptive spraying robotic arm control method based on multimodal perception and a robotic arm using the control method are proposed.

[0004] This invention protects, in one aspect, an adaptive spraying robot arm control method based on multimodal perception, comprising the following steps: S110. Multi-source perception data of the area to be repaired is obtained by integrating a dual vision system and a laser scanning device. S120. The multi-source sensing data is fused using a deep learning network to generate a unified representation containing information on defect location and surface morphology. S130. Spatial registration is performed between the fused two-dimensional image data and the high-precision three-dimensional point cloud data, and a geometric model of the local area to be repaired is constructed based on the registration result. S140. Based on the local geometric features of each path point in the geometric model, retrieve matching process parameter combinations from the preset knowledge graph to generate a digital spraying instruction set containing trajectory, attitude and dynamic process parameters. S150. Before spraying, the digital spraying instruction set is input into the digital twin simulation environment to simulate the entire process, verify the feasibility of the instructions, and output the predicted coverage result. S160. During the real-time execution of the robotic arm, its motion state parameters are continuously collected and updated synchronously with the digital twin; S170. After the spraying is completed, the actual coating quality is inspected, and the inspection results are fed back to the knowledge graph.

[0005] Further, step S110 acquires multi-source perception data of the area to be repaired by integrating a dual vision system and a laser scanning device, and further includes: using a visible light camera to acquire color boundary information of the area to be repaired and the position image of the masking tape; using a short-wave infrared camera to penetrate the surface paint layer to acquire the real edge of the masking tape and the contour image of the area to be repaired; dynamically fusing the position image of the visible light camera and the contour image of the short-wave infrared camera to output a two-dimensional repair area segmentation map.

[0006] Furthermore, a dual-stream encoder structure is used to process visible light and short-wave infrared images respectively. A convolutional backbone network is used to extract features from the images to generate a multi-level feature map sequence. A cross-modal attention gating unit is introduced to calculate the cross-modal attention weight matrix. The cross-modal attention weight matrix is ​​used to perform dynamic weighted fusion and output the fused dual-modal features.

[0007] Furthermore, in the gating unit, global average pooling is performed on the multi-level feature map sequences to generate channel context vectors. After nonlinear transformation through a fully connected layer, cross-weighting is performed to enhance key features in the channel dimension. A bilinear spatial attention map is constructed to calculate cross-modal correlations between pixel positions and generate a spatial weight matrix to suppress false activation regions caused by illumination reflection or masking film reflection. The weighted features are fused and output to the UNet decoder path, and spatial details are restored by combining the skip connection features of the corresponding level on the encoder side.

[0008] Furthermore, after generating the two-dimensional patched area segmentation map, pixel-level modal confidence mapping is performed on the segmentation results. The visible light or short-wave infrared modal sources that each pixel relies on during the inference process are recorded, and the modal confidence map corresponding to the segmentation map is output. The modal confidence map is used for the spatial registration process of the three-dimensional point cloud obtained by laser scanning and the segmentation map. The registration error term is weighted according to the confidence value of each pixel position. Higher weights are given to high confidence regions, and neighborhood smoothing constraints are introduced to low confidence regions to optimize the registration accuracy.

[0009] Furthermore, step S130 also includes: inputting the registered 3D point cloud data into a semantic segmentation network to label each point in the point cloud with geometric semantic labels for boundaries, protrusions, or concave corners.

[0010] Further, in step S140, for each path point on each path segment, the geometric feature vector of its corresponding local point cloud neighborhood is extracted, including the principal curvature tensor, normal rate of change, local surface type, and boundary distance field. The geometric feature vector is input as a query key into the embedding space matching module of the knowledge graph. The similarity score between the geometric feature vector and each process mode node in the knowledge graph is calculated using a graph attention network. The recommendation parameters of adjacent knowledge nodes are weighted and aggregated by combining the semantic label confidence and uncertainty weight.

[0011] Furthermore, in step S150, when the digital spraying instruction set is input into the digital twin simulation environment for full-process simulation, the coating formation process under different process parameters is pre-simulated offline using the finite element method, and the results are constructed into a response surface database, which is embedded in the simulation engine for online prediction; the virtual spraying process is visualized and simulated to reproduce the reflectivity, diffuse reflection and gloss characteristics of the paint surface, and a predicted visually consistent image is generated; forward warning is given for the instantaneous singular configuration of the robotic arm, and it is determined whether dynamic obstacle avoidance or path fine-tuning is triggered.

[0012] Furthermore, in step S160, the position, speed, and torque data of each joint of the robotic arm are received through the time synchronization middleware. Event timestamp alignment and state interpolation compensation are performed on the multi-frequency heterogeneous feedback data stream to drive the digital twin to achieve virtual-real synchronization. The compensation instruction injection adopts a segmented locking and sliding window rebinding strategy to update only the process parameters of the path segments that have not yet been executed. The digital twin internally maintains a deviation traceability map to record the trigger position, cause classification, and adjustment amount of each compensation.

[0013] Further, in step S170, the actual coating quality test results are input into the quality scoring model to generate a five-dimensional structured quality index vector containing uniformity, orange peel index, sagging risk, missed spray area ratio and gloss deviation; a fuzzy inference rule engine is used to perform nonlinear mapping on each index; the original geometric feature vector, environmental parameters, actual process parameter sequence and the final quality score Q of this operation are constructed into a quadruple training sample.

[0014] Furthermore, the training samples of the quadruples are input into the graph neural network fine-tuning mechanism to locally update the node vector representation of the spraying process knowledge graph; the knowledge graph is stored in a graph database, where nodes represent geometric process quality association patterns and edge weights represent empirical confidence; for low-quality samples with a quality score Q below a preset threshold, a decision tree model is used to trace the root cause path of the deviation and generate correction suggestions for the knowledge graph.

[0015] In another aspect, this invention also claims protection for an adaptive spraying robot arm based on multimodal perception, which performs operations according to the control method described in any of the foregoing claims.

[0016] The core of the adaptive spraying robotic arm and its control method of this invention lies in acquiring high-precision multi-source perception data of the area to be repaired by fusing a dual vision system and a laser scanning device. This is combined with a cross-modal attention mechanism to effectively fuse heterogeneous data, generating a unified representation containing defect location and surface morphology information, significantly improving the accuracy and completeness of target area identification under complex working conditions. Furthermore, by spatially registering two-dimensional images with three-dimensional point clouds and constructing a geometric model of the local repair area, refined modeling of irregular surfaces is achieved, providing a reliable spatial basis for subsequent path planning. Based on this, a preset knowledge graph is used to intelligently retrieve the optimal combination of process parameters according to the local geometric features of each path point, generating a model containing trajectory, attitude, and dynamic process parameters. The digital spraying instruction set enhances the intelligence and adaptability of process decision-making. Before spraying, the entire process is simulated and verified in a digital twin simulation environment, predicting the coating coverage effect in advance and optimizing the instruction set, effectively avoiding trial and error costs and resource waste in actual operation. During the execution of the robotic arm, motion state parameters are continuously collected and updated synchronously with the digital twin, realizing closed-loop linkage between the physical entity and the virtual model, improving the real-time performance and stability of the control process. After spraying, the multi-source sensing module is restarted to detect the actual coating quality and feeds the results back to the knowledge graph to dynamically update the parameter mapping relationship, forming a complete closed-loop optimization mechanism of "perception-decision-execution-evaluation-learning", which improves the consistency of spraying quality and the process adaptability. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This application provides a flowchart of an intelligent spraying control method based on digital twins. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0019] Please see Figure 1 This application provides an adaptive spraying robot arm control method based on multimodal perception. This method integrates multi-source sensing data, constructs a digital twin simulation environment, and implements closed-loop quality feedback to achieve intelligent control throughout the entire process, from perception to decision-making, execution, and optimization. Specifically, it includes the following steps: S110. Multi-source perception data of the area to be repaired is obtained by integrating a dual vision system and a laser scanning device. S120. A deep learning network with a cross-modal attention mechanism is used to fuse the multi-source sensing data to generate a unified representation containing information on defect location and surface morphology. S130. Spatial registration is performed between the fused two-dimensional image data and the high-precision three-dimensional point cloud data, and a geometric model of the area to be repaired is constructed based on the registration result. S140. Based on the local geometric features of each path point in the geometric model, retrieve matching process parameter combinations from the preset knowledge graph to generate a digital spraying instruction set containing trajectory, attitude and dynamic process parameters. S150. Before spraying, the digital spraying instruction set is input into the digital twin simulation environment to simulate the entire process, verify the feasibility of the instructions, and output the predicted coverage result. S160. During the real-time execution of the robotic arm, its motion state parameters are continuously collected and updated synchronously with the digital twin; S170. After the spraying is completed, the actual coating quality is tested again, and the test results are fed back to the knowledge graph to update the parameter mapping relationship.

[0020] The adaptive spraying robot control method based on multimodal perception provided in this application effectively solves the problems of inaccurate perception, rigid planning, and open-loop execution in existing spraying systems under complex environments. First, through the collaborative work of a dual vision system and a laser scanning device, combined with a deep learning network using a cross-modal attention mechanism, a high-precision unified representation of the defect location and surface morphology of the repair area is achieved, significantly improving the robustness of perception under reflective, dark, or complex textured conditions. Second, through spatial registration and geometric modeling of two-dimensional images and three-dimensional point clouds, an accurate three-dimensional model of the local area to be repaired is constructed, providing a reliable data foundation for subsequent refined path planning. Furthermore, based on the local features of each path point in this geometric model, the optimal combination of process parameters is dynamically retrieved from a preset knowledge graph, generating a digital instruction set containing trajectory, posture, and dynamic process parameters. This effectively avoids uneven spraying on curved surfaces and boundary buildup problems during spraying. Furthermore, by introducing a digital twin simulation environment to simulate and verify the entire spraying process before execution, potential risks can be identified in advance, ensuring the feasibility of instructions. During execution, the status of the robotic arm and the digital twin are continuously synchronized, achieving real-time monitoring through virtual-real linkage. Finally, after spraying, the actual coating quality is inspected, and the results are fed back to the knowledge graph to update parameter mapping relationships, forming a closed-loop optimization mechanism of execution, evaluation, and learning. This not only improves the success rate and consistency of single spraying but also enhances the ability to self-evolve. Long-term use can continuously accumulate process experience and improve the overall level of intelligence.

[0021] Step S110 acquires multi-source perception data of the area to be repaired by integrating a dual vision system and a laser scanning device. This step achieves highly robust perception of the area to be repaired under complex lighting and curved surface conditions, overcoming the limitations of traditional single vision modality in recognizing reflective surfaces, dark paint surfaces, or masking tape edges.

[0022] Preferably, in specific implementations, the dual vision system includes a visible light camera and a short-wave infrared camera, which can work together to collect complementary information: the visible light camera is used to capture the color boundary information of the area to be repaired and the position image of the masking tape (such as masking tape), which is suitable for accurate differentiation of surface texture and color under normal lighting; the short-wave infrared camera has the ability to penetrate the surface paint film, and can obtain the true edge of the masking tape and the fine geometric contour image of the area to be repaired, especially in scenarios with high light reflection, dark light-absorbing materials or slightly contaminated surfaces, it has stronger stability.

[0023] The acquired visible light and short-wave infrared images are then simultaneously input into the CrossModal Attention UNet network. This network employs a dual-stream encoder structure to process the two inputs separately and achieves feature-level dynamic fusion through a cross-modal attention mechanism. To ensure spatial consistency, the visible light and short-wave infrared cameras achieve initial alignment at the hardware level through a common optical path design or precise extrinsic parameter calibration. In the software preprocessing stage, geometric distortion correction and sub-pixel-level registration are further performed to control the deviation of corresponding points within 0.5 pixels in the image plane, thereby ensuring the accuracy of subsequent fusion.

[0024] Preferably, the CrossModal Attention UNet network's dual-stream encoder uses a ResNet18 convolutional backbone with shared weights, forcing feature extraction from both modalities at the same semantic level and avoiding representation shifts caused by parameter differences. Multi-level feature map sequences are output sequentially from four scales, C2 to C5, with each level connected to a cross-modal attention gating unit (CMAG) to achieve multi-scale dynamic information interaction. In the CMAG, features from one modality serve as the query, and features from the other as the key and value. After linear projection, the cross-modal attention weight matrix is ​​calculated, and a bilinear pooling mechanism is introduced to enhance the modeling of correlations between spatial locations, effectively suppressing erroneous response regions caused by surface reflections or reflections from the edges of occlusion films.

[0025] To further enhance the information selection capability of the channel dimension, a preferred approach is to perform global average pooling on the two feature paths within the CMAG to generate channel context vectors. After nonlinear transformation by a fully connected layer, a cross-weighted operation is performed to highlight the key responses of complementary channels. Simultaneously, a bilinear attention map is constructed in the spatial dimension to measure cross-modal correlation between pixels, forming a spatial weight matrix for weighted aggregation of heterogeneous information. The fused features are then fed into the UNet decoder path, where skip connections at corresponding levels on the encoder side are used to recover spatial details, ultimately outputting a high-precision two-dimensional patched region segmentation map.

[0026] To enhance the interpretability of model decisions and improve adaptability to downstream tasks, pixel-level modal confidence mapping is performed simultaneously with the generation of the 2D patch segmentation map. This records the primary modal source (i.e., visible light camera or short-wave infrared camera) relied upon by each pixel during inference. The modal confidence map is generated based on the spatial distribution statistics of learnable gating coefficients α embedded in the CMAG: at each spatial location, the gradient variance (reflecting texture sharpness) and infrared transmission intensity (reflecting penetration capability) within a 3×3 neighborhood are extracted and input into a lightweight MLP network to predict α∈[0,1], which is used to dynamically adjust the contribution weight of the Query modality. This gating mechanism is optimized end-to-end during backpropagation. High α values ​​indicate a greater reliance on visible light information, while low α values ​​favor infrared modes; their spatial distribution forms the basis of the modal confidence map.

[0027] This modal confidence map serves as an auxiliary output in subsequent 3D reconstruction processes and is also used in ICP (Iterative Closest Point) registration: it backprojects the 2D segmentation mask onto the camera's view frustum, filtering out an initial candidate set of 3D point clouds for the area to be repaired; during registration, it weights the registration error term based on the confidence value of the corresponding pixel location, assigning higher weights to high-confidence areas to achieve accurate alignment, while introducing neighborhood smoothing constraints to prevent incorrect matching in low-confidence areas. Furthermore, after registration, this modal confidence map is input as an additional channel to the PointNet++ semantic segmentation network, where it is concatenated with the coordinates, normals, colors, and local curvature features of the point cloud to form a multi-dimensional input. In the Set Abstraction layer, it guides the propagation of geometric semantic labels such as "boundary," "convexity," or "concave corner" through confidence-weighted aggregation, achieving refined semantic inference under uncertainty modeling.

[0028] In another embodiment of the present invention, the CrossModal Attention UNet network is trained using a hybrid supervision strategy. The loss function includes a main segmentation loss and auxiliary constraint terms: the main loss includes a weighted cross-entropy loss, which takes into account both class imbalance and edge-fitting accuracy; supplemented by modality consistency constraints, which force the deep feature representations of the two modalities in semantically consistent regions to be as close as possible, improving the model's robustness to single-modality loss or degradation. The training process adopts a progressive multi-scale strategy, first initializing the network parameters on low-resolution images, then gradually increasing the input resolution and fine-tuning, ultimately achieving high-precision segmentation performance at full resolution. The entire perception module thus constructs a complete link from original image acquisition to semantically enhanced 3D point cloud output, providing a high-quality multi-source perception foundation for subsequent adaptive spraying planning.

[0029] Step S120 utilizes a deep learning network with a cross-modal attention mechanism to fuse the multi-source perception data, generating a unified representation containing defect location and surface morphology information. Based on the multi-source perception data acquired in step S110, the two-dimensional semantic segmentation results from the dual-vision system are further deeply fused with the high-precision three-dimensional point cloud data obtained from the laser scanning device to construct a unified representation model that combines spatial positioning accuracy with rich geometric details. This unified representation not only accurately marks the boundary range and defect distribution of the area to be repaired but also fully preserves key morphological features such as surface curvature, normal variation, and local convexity / concaveness, providing a cognitive basis for subsequent path planning, indicating "where" and / or "what shape" of the area.

[0030] In specific implementation, the two-dimensional patching region segmentation map and its corresponding modal confidence map output by the aforementioned CrossModal Attention UNet, as a high-level semantic representation of the image modality, are back-projected onto the camera coordinate system and aligned pixel-to-point-cloud level with the three-dimensional point cloud spatially registered by the ICP algorithm. During this process, guided by the effective pixel region of the segmentation mask, a subset of the point cloud within the corresponding view frustum is extracted to form a preliminary candidate set of defect locations. Simultaneously, combining the per-pixel reliability weights provided by the modal confidence map, a weighted sampling strategy is introduced in the point cloud projection mapping stage to prioritize the retention of three-dimensional structural information in high-confidence regions and reduce noise interference from low-quality image regions.

[0031] Subsequently, a cross-modal fusion architecture based on graph neural networks is adopted to align and fuse 2D semantic features and 3D geometric features in the latent space. First, a lightweight CNN branch extracts local texture and contextual semantic features from the segmentation map to generate 2D embedding vectors with class discrimination. Simultaneously, a PointNet++ network is used to extract geometric features such as spatial coordinates, normals, curvature, and density of local point cloud patches to generate corresponding 3D embedding vectors. The two types of embedding vectors interact in a shared attention fusion module: using 2D semantic features as queries and 3D geometric features as keys and values, cross-domain attention weights are calculated, enabling each 3D point to adaptively absorb image semantic information related to its projection position.

[0032] This fusion process fully considers the data sparsity and resolution differences of different modalities, and introduces a deformable attention mechanism to allow the query position to dynamically shift within a certain range to find the best matching response, thereby alleviating the misalignment problem caused by projection errors or point cloud sparsity. Finally, the fused feature vector is expanded point by point on the point cloud sequence to form a set of joint feature representations rich in semantic labels and surface morphology descriptions, namely, a unified representation of "defect location-surface morphology". This representation is stored in the form of an enhanced point cloud, with each point carrying multi-dimensional attribute fields such as position coordinates, normal vector, curvature, region type (e.g., boundary, plane, concave corner, convex surface), and original modality confidence, forming a unified input source for subsequent geometric modeling and process decisions.

[0033] The entire fusion processing flow runs in real time in an accelerated environment, supporting efficient inference in batch processing mode and ensuring that the end-to-end latency from raw sensing data to the unified representation is controlled within 200 milliseconds, meeting the response speed requirements of industrial sites. The resulting unified representation overcomes the lack of depth accuracy in pure vision methods and compensates for the semantic deficiencies of pure point cloud methods, achieving a comprehensive, accurate, and computable description of complex patch areas, laying a solid data foundation for spatial modeling and path generation.

[0034] Step S130 spatially registers the fused 2D image data with high-precision 3D point cloud data, and constructs a local (or global, where global or local is not limited and both are applicable) geometric model of the area to be repaired based on the registration result. Building upon the unified representation containing defect location and surface morphology information generated in step S120, precise spatial alignment and geometric modeling operations are further performed. The aim is to establish a local 3D geometric model with consistent coordinates, complete structure, and enhanced semantics, serving as the baseline digital basis for subsequent adaptive spraying path planning.

[0035] This spatial registration process is based on the extrinsic parameter calibration of the camera and laser scanning device initially aligned in step S110, combined with the semantic guidance features output in step S120, and employs an iterative optimization strategy to improve registration accuracy. Specifically, firstly, using known camera intrinsic parameters and binocular or multi-view geometric relationships, key pixels (such as segmentation boundaries and texture abrupt change points) in the fused 2D image data are back-projected into 3D spatial rays; simultaneously, edge point sets and significant surface feature points of the corresponding regions are extracted from the high-precision 3D point cloud. By minimizing the distance error between the back-projected rays and the nearest neighbor point cloud points, an objective function is constructed, and an improved ICP algorithm is used for non-rigid registration optimization. A weighting factor is introduced to adjust the matching contribution of different regions—high-confidence semantic regions are given greater weight, ensuring that the alignment accuracy of key repair boundaries is controlled within ±0.2mm.

[0036] To further eliminate local distortions caused by viewpoint occlusion or sparse point clouds, regularization constraints based on normal consistency and curvature continuity are introduced to suppress non-physical deformations during optimization. After registration, semantic labels on all 2D images (such as "area to be repaired", "occluded area", and "boundary transition zone") are accurately mapped to their corresponding 3D point cloud vertices, forming a semantically enhanced point cloud dataset with accurate spatial coordinates.

[0037] Further optimization was performed, and based on this, the geometric modeling process for the local repair area was initiated. First, based on the point cloud of the defect area marked after registration, a closed triangular mesh model was generated using the Poisson surface reconstruction algorithm. This algorithm effectively fills the tiny holes in the point cloud through implicit function fitting, while maintaining a smooth transition of the original geometric details. To accommodate the modeling requirements of complex curved surfaces (such as door edges and bumper arcs), the sampling density and octree level depth were dynamically adjusted during the reconstruction process to control the computational load while ensuring model resolution.

[0038] Subsequently, topology optimization and local simplification were performed on the generated mesh model to remove isolated noise patches and compress the number of redundant triangles. This controlled the number of faces within a reasonable range without affecting geometric fidelity, facilitating subsequent real-time rendering and simulation. Ultimately, the geometric model not only accurately reproduced the 3D contour and surface undulation of the area to be repaired but also inherited semantic attribute fields from previous steps, including region type labels, normal directions, curvature distribution maps, and boundary confidence information. This formed a lightweight, structured, and resolvable digital twin foundation for local repair areas, geared towards process decision-making.

[0039] The entire modeling process is executed asynchronously in the background service module, supporting multi-task parallel processing. It can be exported to common 3D formats such as STL, OBJ, or ASMT-DMU via standardized interfaces for direct use by downstream modules. The resulting geometric model achieves the transformation from raw perceptual data to a usable digital entity, providing high-fidelity, low-latency geometric input support for process parameter retrieval and digital instruction generation based on local geometric features in the Buzhou S140, ensuring that the spraying path can truly conform to the actual object shape and adaptively generate.

[0040] Step S140 retrieves matching process parameter combinations from a preset knowledge graph based on the local geometric features of each path point in the geometric model, generating a digital spraying instruction set containing trajectory, attitude, and dynamic process parameters. Building upon the geometric model of the local repair area constructed in step S130, the process enters the intelligent decision-making stage. This aims to achieve refined and adaptive binding between the spraying path and process parameters, overcoming the technical bottleneck of traditional fixed-parameter spraying which easily produces defects such as unevenness, sagging, or edge accumulation on complex curved surfaces.

[0041] In practice, the triangular mesh in the geometric model is first discretized and sampled to generate an ordered sequence of spatial path points, forming the basic spraying trajectory. For each path point on each path segment, the geometric feature vector within its corresponding local point cloud neighborhood is further extracted. This feature vector comprehensively reflects the three-dimensional morphological characteristics of that location. The extracted geometric feature vector includes the principal curvature tensor (describing the degree of curvature of the surface in two orthogonal directions), the rate of change of normal (characterizing the smoothness of the surface), the local surface type obtained by fitting and classifying using the RANSAC algorithm (such as plane, sphere, cylinder, saddle surface), and the boundary distance field (representing the shortest Euclidean distance from the current point to the boundary of the repair area). These features together constitute the multi-dimensional input key for subsequent process queries.

[0042] The aforementioned geometric feature vectors are input as query keys into the preset spraying process knowledge graph embedding spatial matching module. This knowledge graph is organized in a graph structure, with nodes representing typical geometric patterns (such as "high curvature convex surface", "groove with aspect ratio > 3", "transition slope"), and edges connecting similar patterns and carrying recommended process parameter sets and their confidence weights. The matching module uses a graph attention network to calculate the semantic similarity score between the current feature vector and each process pattern node, and combines the label confidence from the PointNet++ semantic segmentation output, as well as the uncertainty weights provided by the modality confidence graph generated in step S110, to weight and aggregate the recommended parameters of adjacent knowledge nodes to obtain the optimal initial values ​​of process parameters suitable for the current local morphology.

[0043] To achieve smooth evolution of process parameters along a continuous path and avoid abrupt changes that could lead to jumps in paint film thickness or unstable atomization, the spraying process parameter space is modeled as a five-dimensional non-Euclidean manifold, with dimensions representing spray gun movement speed, distance from the workpiece surface, paint flow rate, atomization pressure, and spray overlap rate. Within this manifold space, a geodesic interpolation algorithm is used to interpolate process parameters between adjacent control points, ensuring that parameter transitions conform to physical realizability. The interpolation kernel function is modulated by local curvature gradients: when the path traverses regions of abrupt curvature change (such as edge intersections), the interpolation step size is automatically compressed, and a piecewise linearity preservation strategy is switched to prevent process parameter oscillations and ensure the stability of the spraying process.

[0044] Further optimized, a context-aware constraint engine is introduced during parameter binding to dynamically validate the recommended parameters output by the knowledge graph. This engine maintains a configurable rule base, including hard constraints such as "paint flow rate in concave corner areas must not exceed 0.8 L / min" and "minimum spray distance must not be less than 15 cm." Simultaneously, it combines the spray gun's current movement direction, acceleration status, and wet film feedback information from adjacent painted areas (provided by a paint film sensor) to perform soft constraint optimization, such as reducing the flow rate in advance to prevent paint buildup when approaching a shaded area. All validated parameters are encapsulated into process data objects conforming to ISO 10303-235 standards, ensuring interoperability with other intelligent manufacturing systems.

[0045] The generated digital spraying instruction set is organized using a hierarchical spatiotemporal coding structure: the bottom layer is the SE(3) path trajectory in the joint space or task space of the robotic arm, which defines the running route of the spray gun in three-dimensional space; the middle layer is the spray gun posture sequence represented by unit quaternions, which ensures that the nozzle is always perpendicular to the local surface normal or tilted at a preset angle; the top layer is the time sequence of the aforementioned five-dimensional process parameters. The three are strictly aligned through a shared timestamp axis and are synchronously parsed by the controller during execution.

[0046] Preferably, the instruction set also embeds a dynamic rebinding trigger flag, which is used to reactivate the knowledge graph query mechanism in a specified path segment when the actual coverage deviation detected by the response digital twin system exceeds a set threshold during subsequent execution, and adjust the process parameters of the unexecuted part in real time to achieve online closed-loop correction.

[0047] Preferably, the entire instruction generation process is run on a high-performance edge computing unit, achieving millisecond-level response and ensuring an end-to-end latency of less than 300ms from geometric analysis to instruction output. The resulting digital spraying instruction set not only integrates trajectory, attitude, and process, but also provides key capabilities for highly consistent and high-quality automated spraying operations.

[0048] Step S150 involves inputting the digital spraying instruction set into a digital twin simulation environment for full-process simulation before spraying, verifying the feasibility of the instructions, and outputting predicted coverage results. After completing the digital spraying instruction set, which includes trajectory, attitude, and dynamic process parameters, generated in step S140, it is imported into the digital twin simulation environment built on the host computer system for full-link virtual verification. This step enables a preliminary assessment of the safety, accessibility, and quality consistency of the spraying process, avoiding actual operation failures or equipment damage caused by path conflicts, attitude instability, or process mismatches.

[0049] When performing full-process simulation in a digital twin simulation environment, a pre-built material behavior response surface database is invoked. This database uses the finite element method to perform offline simulation modeling of the coating formation process under various typical working conditions, covering physical mechanisms such as paint mist diffusion dynamics, droplet impact and spreading behavior, wet film accumulation law, and surface wetting characteristics under different combinations of paint viscosity, ambient temperature and humidity, spray distance, flow rate, and moving speed. The mapping relationship between key parameters and the final paint film thickness distribution is extracted through numerous simulation experiments and stored in the database in the form of response surface functions. When the actual instruction set is input, the simulation engine uses real-time table lookup and interpolation based on the process parameters at each path point to predict the paint film deposition amount at that location. It also accumulates and calculates the overall coverage uniformity and layer thickness consistency along the entire trajectory, generating a predicted coverage result map.

[0050] Further optimized, physically based rendering and ray tracing techniques are employed to visualize and simulate the virtual spraying process. A material property field driven by predicted paint film thickness, including reflectivity, roughness, normal perturbation, and clear coat refractive index, is applied to the surface of the 3D geometric model. Combined with an ambient light probe and an HDR background, the diffuse reflection, specular highlights, and gloss transition characteristics of the repaired area under different viewing angles and lighting conditions are accurately reproduced. The resulting predicted visual consistency image can be used to identify potential color differences, orange peel texture, or uneven edge gloss in advance, assisting operators or AI quality inspection modules in making predictive decisions.

[0051] At the kinematic level, the simulation engine integrates continuous collision detection and real-time Jacobian matrix analysis modules to comprehensively verify the safety and stability of the robotic arm's movements. The module detects potential penetration risks between the spray gun, robotic arm links, and the workpiece to be repaired or other fixed structures at sub-millisecond time steps, especially for large-angle swing paths near complex curved surfaces. Simultaneously, it analyzes the Jacobian matrix of each joint of the robotic arm in real time, monitoring its condition number trends. If the condition number approaches a singular configuration (i.e., the determinant approaches zero), a forward warning mechanism is immediately triggered. Combined with a preset joint torque limit model, it determines whether the current command might cause motor overload or control jitter. If a risk exists, a local path fine-tuning algorithm is automatically activated. While maintaining the spray trajectory's adherence to the original geometry, the intermediate attitude angle is adjusted to avoid singular configurations or potential interference areas.

[0052] The entire simulation process runs on a GPU-accelerated architecture, supporting multi-threaded parallel processing to ensure that the simulation time from instruction input to result output is controlled within 1.5 seconds, meeting the rapid response requirements of industrial sites. After the simulation, an evaluation report is generated, including "motion feasibility score," "expected paint film thickness deviation range," "visual consistency index," and "risk alarm list." The instruction set is only allowed to be unlocked and sent to the physical control system after all key indicators have passed the threshold verification. It is evident that step S150 achieves accurate evaluation and control of the spraying task, significantly improving the first-time success rate and overall system reliability.

[0053] S160: During the real-time execution of the robotic arm, its motion state parameters are continuously collected and updated synchronously with the digital twin. After completing the simulation verification in step S150 and confirming the feasibility of the instructions, the digital spraying instruction set is officially issued to the physical robotic arm control system to start the actual spraying operation. Simultaneously, the system enters the closed-loop execution phase. Through high-frequency, multi-source heterogeneous data acquisition and real-time synchronization mechanisms, it ensures that the physical spraying process and the digital twin in the host computer maintain a highly consistent state mapping, providing a data foundation for subsequent online monitoring and dynamic compensation.

[0054] Preferably, during the execution of the robotic arm, real-time feedback data streams from multiple sensor nodes are continuously received via an industrial communication network: the robotic arm controller uploads the position, speed, and torque information of each joint at a sampling frequency of 1 kHz; the miniature IMU (Inertial Measurement Unit) installed at the end of the spray gun outputs the attitude quaternion and angular velocity data of the spray gun at a frequency of 500 Hz to accurately track changes in nozzle pointing; at the same time, a non-contact paint film thickness sensor based on the β-ray backscattering principle is integrated on the side of the spray gun, outputting a measurement value every 200 ms, with a spatial sampling density of 3 measurement points per square centimeter, to achieve high-resolution online monitoring of the wet film thickness of the sprayed area.

[0055] Because the aforementioned data sources have different sampling frequencies and timing characteristics, a time synchronization middleware is introduced for unified management. This middleware achieves global clock alignment based on the IEEE 1588 Precision Time Protocol, timestamps each data record, and uses a combination of linear interpolation and Kalman filtering to compensate for missing or delayed data frames, ensuring precise alignment of data from different channels on the timeline. The processed multimodal state sequence is injected into the digital twin in real time, driving the synchronous evolution of its internal robotic arm dynamics model, spray gun trajectory, and coating accumulation model, achieving synchronization between virtual and real states. This allows the virtual environment to realistically reproduce every subtle movement of the physical system.

[0056] When the paint film thickness sensor detects that the actual thickness of a local area deviates from the target value by more than a certain amount, such as 15-20 μm, the look-ahead compensation mechanism is immediately activated. This mechanism calls the built-in reverse process optimization solver, using the measured thickness of the currently sprayed section as the boundary condition, to reconstruct the process parameter optimization objective function for the remaining unexecuted paths. Optimization variables include the paint flow rate, atomization pressure, and residence time of subsequent spray guns. Constraints cover equipment physical limits (e.g., maximum paint flow ≤ 1.2 L / min), process requirements (e.g., minimum interlayer drying interval ≥ 30 s), and edge transition smoothness indicators (gradient change rate ≤ 5 μm / mm), ensuring that the adjustment scheme is both effective and safe.

[0057] Further optimization involves employing a sequential quadratic programming algorithm, which can complete nonlinear optimization calculations within 50ms, generating a set of corrected process parameter sequences, and sending updated instructions to the PLC execution unit via a secure communication channel. To ensure the stability of the execution process and the integrity of historical data, the injection of compensation instructions adopts a segmented locking and sliding window rebinding strategy: the parameters of the path segments that have been sprayed remain unchanged, and the process parameters are replaced only for the path intervals that have not yet been executed, thus avoiding interference with the areas where paint film has already formed.

[0058] The digital twin maintains a structured deviation tracing graph to record complete contextual information for each compensation event, including the spatial coordinates of the trigger location, cause classification (e.g., material viscosity fluctuations, minor nozzle blockage, trajectory deviation, etc.), original deviation amount, adjustment parameters, and final compensation effect. This graph not only supports post-event quality traceability but also serves as important input for subsequent feedback learning modules, used to analyze long-term system performance trends and optimize process recommendation logic in the knowledge graph.

[0059] Through the above process, step S160 achieves dynamic coupling between the physical execution process and the virtual model, making the digital twin not only a static mirror, but also an intelligent agent with real-time perception, state inference and decision support capabilities, providing online control support for the entire spraying process.

[0060] S170. After spraying is completed, the actual coating quality is detected again using multi-source sensing, and the detection results are fed back to the knowledge graph to update the parameter mapping relationship. After the robotic arm completes the execution of the entire spraying command, it enters the most critical link of closed-loop control, namely quality verification and experience accumulation. Preferably, this step can, for example, reuse the multi-source sensing process in step S110 to quickly scan and comprehensively evaluate the sprayed area, realizing the leap from "execution end" to "continuous evolution", enabling the entire method to achieve self-learning and process optimization capabilities.

[0061] In practice, the dual vision system consisting of visible light and near-infrared cameras is first reactivated to acquire multi-angle, multi-spectral images of the area to be repaired. Simultaneously, a laser scanning device is used to obtain the microscopic morphological changes of the coating surface. After distortion correction and parallax correction, the acquired images are input into a lightweight dual-stream convolutional network: the texture branch, based on an improved ResNet18 structure and incorporating locally binary mode-enhanced convolutional kernels, is used to extract surface defect features such as orange peel texture and graininess; the reflection branch separates specular and diffuse reflection components through phase analysis of the frequency domain signal, evaluating gloss uniformity and sagging tendency. The fused features are then fed into a cross-modal attention fusion unit to generate a spatially adaptive defect response map, and combined with multi-scale edge detection logic, accurately locates the unsprayed boundary and paint film fracture zone.

[0062] The detection results are then input into a pre-defined quality scoring model to generate a structured quality index vector containing five sub-indicators: uniformity (0-1), orange peel index (0-3), sagging risk (0-1), missed spray area ratio (%), and gloss deviation (ΔGU). The scoring process uses a fuzzy inference rule engine for nonlinear mapping. The rule set is encoded by expert experience, such as "if the sagging length > 5 mm and the reflection gradient > 0.8 GU / mm, then the sagging risk = 0.95", or "when the gloss gradient change in the edge transition area exceeds the threshold, the gloss deviation score decreases". The scores of each sub-item are normalized and then weighted and fused to form the final comprehensive quality score Q∈[0,1], and a confidence label is attached to record the consistency between the strength of the image evidence and the model response on which each judgment is based.

[0063] Preferably, to achieve knowledge evolution, the key data from the entire process of this task are constructed into a four-tuple training sample for subsequent model iterations. This sample includes: [Input] the original geometric feature vector (including principal curvature, aspect ratio, and boundary curvature gradient), [Context] environmental parameters (ambient temperature and humidity, paint batch number), [Decision] the actual sequence of process parameters (speed, distance, flow rate, and atomization pressure changes over time), and [Output] the final quality score Q and its sub-item scores. This four-tuple forms a causal chain of "input conditions, execution strategy, and output result," which is the core data unit for learning.

[0064] The quadruple sample, along with a unique job fingerprint, is written into the distributed time-series database. The job fingerprint is generated by combining a timestamp, workpiece ID, and spray gun number, ensuring that each record is traceable and comparable. The database, as a long-term memory storage center, supports efficient retrieval and batch access by vehicle type, region type, process combination, and other dimensions, providing a data foundation for periodic knowledge updates.

[0065] An incremental graph neural network fine-tuning task is periodically initiated to dynamically update the node representations in the spraying process knowledge graph. This knowledge graph is stored in a graph database as triples, where nodes represent association patterns between geometric patterns, process parameters, and quality performance, and edge weights reflect the confidence level of historical experience. Historical sample sets are called, and a graph embedding algorithm is used to locally fine-tune the node vectors, making process recommendations under similar geometric scenarios more accurate. The updated embedding vectors are synchronously loaded into the graph attention network module used in step S140, adjusting its node similarity calculation logic in the parameter binding stage to improve the accuracy of future decisions.

[0066] For low-quality samples with a comprehensive quality score Q below a preset threshold, a reverse tracing analysis process is automatically triggered. This process uses a decision tree model to trace the root cause of the deviation, identifying key factors leading to quality problems, such as typical patterns like "excessive flow rate in high-curvature concave corner areas" and "insufficient atomization at low temperatures." Based on the tracing results, knowledge graph correction suggestions are generated, including adding hard constraint rules (such as "maximum flow rate in concave areas with an aspect ratio > 2.5 ≤ 0.7 L / min") or adjusting the confidence weights of existing recommendation parameters. After all suggestions are manually reviewed and confirmed, they are injected into the graph database by the backend service, completing the closed-loop evolution of the knowledge graph.

[0067] The entire feedback learning process completes model iteration without affecting real-time production operations. This enables the system to autonomously extract experience from a single painting task, optimize decision-making logic, and continuously improve process quality, forming an intelligent painting closed-loop system.

[0068] Another embodiment of the present invention provides an adaptive spraying robot arm based on multimodal perception. The control method of the robot arm is as described in the above method embodiments and will not be repeated here.

[0069] The embodiments described above are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An adaptive spraying robot arm control method based on multimodal perception, characterized in that, Including the following steps: S110. Multi-source perception data of the area to be repaired is obtained by integrating a dual vision system and a laser scanning device. S120. The multi-source sensing data is fused using a deep learning network to generate a unified representation containing information on defect location and surface morphology. S130. Spatial registration is performed between the fused two-dimensional image data and the high-precision three-dimensional point cloud data, and a geometric model of the area to be repaired is constructed based on the registration result. S140. Based on the local geometric features of each path point in the geometric model, retrieve matching process parameter combinations from the preset knowledge graph to generate a digital spraying instruction set containing trajectory, attitude and dynamic process parameters. S150. Before spraying, the digital spraying instruction set is input into the digital twin simulation environment to simulate the entire process, verify the feasibility of the instructions, and output the predicted coverage result. S160. During the real-time execution of the robotic arm, its motion state parameters are continuously collected and updated synchronously with the digital twin; S170. After the spraying is completed, the actual coating quality is inspected, and the inspection results are fed back to the knowledge graph.

2. The method according to claim 1, characterized in that, Step S110 acquires multi-source perception data of the area to be repaired by integrating a dual vision system and a laser scanning device, further including: A visible light camera was used to acquire color boundary information of the area to be repaired and images of the location of the masking tape. A short-wave infrared camera was used to penetrate the surface paint layer to obtain images of the true edge of the masking tape and the contour of the area to be repaired. The position image from the visible light camera and the contour image from the short-wave infrared camera are dynamically fused to output a two-dimensional patched area segmentation map.

3. The method according to claim 2, characterized in that, A dual-stream encoder structure is used to process visible light and short-wave infrared images separately. A convolutional backbone network is used to extract features from the images and generate a multi-level feature map sequence. A cross-modal attention gating unit is introduced to calculate the cross-modal attention weight matrix; The cross-modal attention weight matrix is ​​used for dynamic weighted fusion to output the fused bimodal features.

4. The method according to claim 3, characterized in that, In the gating unit, global average pooling is performed on the multi-level feature map sequence to generate channel context vectors. After nonlinear transformation through a fully connected layer, cross-weighted operation is performed to enhance key features in the channel dimension. Construct a bilinear spatial attention map, calculate the cross-modal correlation between pixel locations, and generate a spatial weight matrix to suppress false activation regions caused by illumination reflection or masking film reflection. The weighted features are fused and output to the UNet decoder path, and spatial details are restored by combining the skip connection features of the corresponding level on the encoder side.

5. The method according to claim 2, characterized in that, After generating the two-dimensional patched area segmentation map, pixel-level modal confidence mapping is performed on the segmentation results. The visible light or short-wave infrared modal sources that each pixel depends on during the inference process are recorded, and the modal confidence map corresponding to the segmentation map is output. The modal confidence map is used in the spatial registration process of the 3D point cloud and segmentation map obtained by laser scanning. The registration error term is weighted according to the confidence value of each pixel position. Higher confidence regions are given higher weights, and neighborhood smoothing constraints are introduced for low confidence regions to optimize the registration accuracy.

6. The method according to claim 2, characterized in that, Step S130 further includes: inputting the registered 3D point cloud data into a semantic segmentation network to label each point in the point cloud with geometric semantic labels for boundaries, protrusions, or concave corners.

7. The method according to claim 1, characterized in that, In step S140, for each path point on each path segment, the geometric feature vector of its corresponding local point cloud neighborhood is extracted, including the principal curvature tensor, normal rate of change, local surface type, and boundary distance field. The geometric feature vector is input as a query key into the embedding space matching module of the knowledge graph. The similarity score between the geometric feature vector and each process mode node in the knowledge graph is calculated using a graph attention network. The recommendation parameters of adjacent knowledge nodes are weighted and aggregated by combining the semantic label confidence and uncertainty weight.

8. The method according to claim 1, characterized in that, In step S150, when the digital spraying instruction set is input into the digital twin simulation environment for full-process simulation, the coating formation process under different process parameters is simulated offline using the finite element method, and the results are constructed into a response surface database and embedded into the simulation engine for online prediction. The virtual spraying process is visualized and simulated to reproduce the reflectivity, diffuse reflection and specular characteristics of the paint surface, and to generate a predicted visually consistent image. Provide forward warning of instantaneous singular configurations of the robotic arm and determine whether dynamic obstacle avoidance or path fine-tuning is triggered.

9. The method according to claim 1, characterized in that, In step S160, the position, speed, and torque data of each joint of the robotic arm are received through the time synchronization middleware. Event timestamp alignment and state interpolation compensation are performed on multi-frequency heterogeneous feedback data streams to drive the digital twin to achieve virtual-real synchronization; The compensation instruction injection adopts a segmented locking and sliding window rebinding strategy, only updating the process parameters of the path segments that have not yet been executed; A source map of internal maintenance deviations in a digital twin records the trigger location, cause classification, and adjustment amount for each compensation.

10. The method according to claim 1, characterized in that, In step S170, the actual coating quality test results are input into the quality scoring model to generate a five-dimensional structured quality index vector that includes uniformity, orange peel index, sagging risk, missed spray area ratio and gloss deviation. A fuzzy inference rule engine is used to perform nonlinear mapping on each indicator; The original geometric feature vector, environmental parameters, actual process parameter sequence, and final quality score Q of this operation are used to construct a four-tuple training sample.

11. The method according to claim 10, characterized in that, The training samples of the quadruples are input into the graph neural network fine-tuning mechanism to locally update the node vector representation of the spraying process knowledge graph; The knowledge graph is stored in a graph database, where nodes represent geometric process quality correlation patterns and edge weights represent empirical confidence levels. For low-quality samples with a quality score Q below a preset threshold, a decision tree model is used to trace the root cause of the deviation and generate correction suggestions for the knowledge graph.

12. An adaptive spraying robotic arm based on multimodal perception, characterized in that, The robotic arm performs operations according to the control method described in any one of claims 1-11.

Images