Curved track painting method and system based on machine vision

Abstract

The present application provides a curve trajectory painting method and system based on machine vision. The method includes: obtaining workpiece data of the workpiece to be sprayed, including multi-view images from an industrial camera and workpiece depth data from a depth camera; constructing a three-dimensional model of the workpiece to be sprayed based on the multi-view images and workpiece depth data; constructing a multi-agent game model based on the three-dimensional model of the workpiece, and determining the spraying trajectory of the workpiece to be sprayed based on the multi-agent game model; constructing a recursive Gaussian process state space model based on the spraying trajectory, and optimizing the spraying trajectory based on the model to obtain a control instruction sequence; finally, tracking and controlling the spraying operation. The method provided in the present application can achieve precise spraying of complex surfaces and improve spraying quality and efficiency.

Description

Technical Field

[0001] The present application relates to the field of machine vision and intelligent manufacturing technology, and in particular to a curve trajectory spray painting method and system based on machine vision. The method and system are mainly used in scenarios such as sheet metal spray repair in the automotive aftermarket that require precise spraying of complex curved surfaces. Background Art

[0002] In modern industrial production, the painting process is an important part of surface treatment, and its quality directly affects the appearance and protective effect of the product. At present, in the fields of sheet metal spray repair in the automotive aftermarket, the painting process mainly relies on manual operation or simple automated equipment. Among them, although manual painting has a certain degree of flexibility, it is limited by the experience and skill level of technicians, and the labor cost is high and the quality is unstable. Simple automatic painting systems usually use preset straight lines or simple curve trajectories, which cannot adapt well to complex curved shapes.

[0003] In recent years, with the development of machine vision technology, some studies have begun to try to introduce visual feedback into automatic painting systems. Such systems obtain workpiece surface information through visual sensors to achieve real-time adjustment of the spraying process. However, the existing automatic painting systems based on single visual feedback still have obvious shortcomings, mainly manifested in the lack of accurate modeling and trajectory optimization capabilities for complex surfaces, making it difficult to achieve accurate trajectory planning for complex surfaces, which in turn affects the spraying uniformity and coverage effect. Summary of the invention

[0004] In view of this, the present application provides a curved trajectory painting method and system based on machine vision, which solves the problem that it is difficult to achieve accurate trajectory planning for complex surfaces in the prior art.

[0005] The embodiment of the present application provides a curve trajectory painting method based on machine vision, which is used for processing equipment, including:

[0006] Acquire workpiece data of the workpiece to be sprayed, wherein the workpiece data includes a multi-view image from an industrial camera and workpiece depth data from a depth camera;

[0007] Constructing a three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data;

[0008] According to the three-dimensional model of the workpiece, a multi-agent game model is constructed, and a spraying trajectory of the workpiece to be sprayed is determined based on the multi-agent game model;

[0009] According to the spraying trajectory, a recursive Gaussian process state space model is constructed, and the spraying trajectory is optimized based on the recursive Gaussian process state space model to obtain a control instruction sequence;

[0010] Track and control the spraying operation.

[0011] Optionally, before constructing the three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data, the method further includes:

[0012] Using a preset adaptive Gaussian filtering algorithm to reduce noise on the multi-view images;

[0013] The preset multi-scale Retinex algorithm is used to perform illumination compensation on the denoised multi-view images;

[0014] The preset Zhang calibration algorithm is used to perform distortion correction on the multi-view image after illumination compensation to obtain the target multi-view image.

[0015] Optionally, constructing a three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data includes:

[0016] Extracting features from the target multi-view images to obtain image feature data, where the image feature data is used to indicate a corresponding relationship between images from different viewpoints;

[0017] Performing triangulation on the image feature data and the workpiece depth data to generate a dense point cloud;

[0018] The dense point cloud is reconstructed into a curved surface to obtain a three-dimensional model of the workpiece to be sprayed.

[0019] Optionally, constructing a multi-agent game model according to the workpiece three-dimensional model includes:

[0020] According to the three-dimensional model of the workpiece, a multi-agent game framework is constructed, wherein the multi-agent game framework is used to instruct multiple agents configured therein to generate initial candidate trajectories according to normal vector information and local curvature features of the three-dimensional model of the workpiece;

[0021] The projection distribution of the spray fan on the surface is simulated by the preset Monte Carlo algorithm to determine the spray coverage objective function;

[0022] Taking curvature continuity and tangential acceleration constraint as evaluation indicators, and determining a trajectory smoothness objective function based on the evaluation indicators;

[0023] Determine a robot motion efficiency objective function, wherein the robot motion efficiency objective function is used to evaluate the joint speed, acceleration and energy consumption of the robot;

[0024] Determine the paint utilization rate objective function based on the pre-established dynamic relationship model between spraying amount and moving speed;

[0025] The spray coverage objective function, the trajectory smoothness objective function, the robot motion efficiency objective function and the paint utilization objective function are weighted and integrated to obtain a game utility function model;

[0026] The multi-agent game framework and the game utility function model are subjected to hybrid game optimization to obtain a multi-agent game model.

[0027] Optionally, the performing hybrid game optimization on the multi-agent game framework and the game utility function model to obtain a multi-agent game model includes:

[0028] Based on the multi-agent game framework, a preset response dynamics algorithm is used to solve the strategy combination, wherein the strategy combination is used to instruct each agent to iteratively update its respective strategy to maximize the game utility function model. The multi-agent game framework includes a top-level structure, a middle-level structure and a bottom-level structure. The top-level structure is used to indicate global trajectory planning, the middle-level structure is used to indicate coordination between processing agents, and the bottom-level structure is used to indicate local trajectory optimization.

[0029] Optionally, constructing a recursive Gaussian process state space model according to the spraying trajectory, optimizing the spraying trajectory based on the recursive Gaussian process state space model, and obtaining a control instruction sequence includes:

[0030] According to the spraying trajectory, a state vector is constructed, wherein the state vector includes kinematic parameters of the robot end effector and process parameters of the spray gun;

[0031] constructing a recursive Gaussian process state space model based on the state vector and the acquired conditional variables, wherein the conditional variables include the workpiece surface curvature and material properties of the workpiece to be sprayed;

[0032] The preset sparse variational Gaussian process algorithm is used to model the dynamic characteristics of the system and obtain a predictive controller;

[0033] Based on the recursive Gaussian process state space model and the predictive controller, the spraying trajectory is adaptively optimized to obtain a control instruction sequence.

[0034] Optionally, the adaptive trajectory optimization of the spraying trajectory based on the recursive Gaussian process state space model and the predictive controller to obtain a control instruction sequence includes:

[0035] Construct a sliding time domain observation window;

[0036] Based on the sliding time-domain observation window, the recursive Gaussian process state-space model and the predictive controller, a preset Bayesian inference algorithm is used to update system state data and to evaluate uncertainty data used to characterize state uncertainty;

[0037] According to the updated system status data, the trajectory parameters of the spraying trajectory are adjusted, wherein the trajectory parameters include speed planning parameters, acceleration smoothing parameters and posture compensation parameters;

[0038] The weight of the trajectory smoothness constraint of the spraying trajectory is dynamically adjusted according to the uncertainty data.

[0039] Optionally, the tracking and controlling of the spraying operation includes:

[0040] Based on the control instruction sequence, an AMS-PiM trajectory tracking controller is constructed, wherein a state feedback channel of the AMS-PiM trajectory tracking controller is used to indicate that a sensor signal generated after executing the control instruction sequence is converted into a current domain, and a switched capacitor network model for indicating dynamic error compensation is introduced into a feedback loop of the AMS-PiM trajectory tracking controller;

[0041] Constructing a cross-coupled current mirror array, and constructing an analog in-memory computing accelerator based on the cross-coupled current mirror array, wherein the cross-coupled current mirror array is used to instruct to perform a matrix multiplication operation in the analog domain;

[0042] Based on the AMS-PiM trajectory tracking controller and the analog in-memory computing accelerator, the spraying operation is tracked and controlled.

[0043] Optionally, the method further comprises:

[0044] Preprocessing the operation images collected during the spraying operation;

[0045] Detect the edges and defects of the spraying area in the pre-processed working image and generate corresponding spraying status data;

[0046] Conduct quality assessment of spray uniformity of spray status data;

[0047] According to the quality assessment results, the preset fuzzy neural network is used to dynamically optimize the spraying process parameters in the spraying operation.

[0048] The embodiment of the present application also provides a curve trajectory painting system based on machine vision, including: a processing device, an industrial camera and a depth camera;

[0049] The industrial camera is used to obtain multi-view images of the workpiece to be sprayed;

[0050] The depth camera is used to obtain workpiece depth data of the workpiece to be sprayed;

[0051] The processing device is connected to the industrial camera and the depth camera, and the processing device includes:

[0052] A data acquisition module, used for acquiring workpiece data of the workpiece to be sprayed, wherein the workpiece data includes the multi-view images and the workpiece depth data;

[0053] A three-dimensional reconstruction module, used for constructing a three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data;

[0054] A trajectory planning module, used to construct a multi-agent game model according to the three-dimensional model of the workpiece, and determine the spraying trajectory of the workpiece to be sprayed based on the multi-agent game model;

[0055] A trajectory optimization module, used for constructing a recursive Gaussian process state space model according to the spraying trajectory, and performing trajectory optimization on the spraying trajectory based on the recursive Gaussian process state space model to obtain a control instruction sequence, wherein the control instruction sequence includes joint data for controlling robot joints, posture data for controlling the end effector of the robot, and operation data for controlling a spray gun;

[0056] The trajectory tracking module is used to track and control the trajectory of the spraying operation, and the spraying operation is used to instruct the spraying of the workpiece to be sprayed according to the control instruction sequence.

[0057] This application has the following technical effects:

[0058] By constructing a multi-agent game model and a recursive Gaussian process state space model, accurate trajectory planning and real-time optimization of complex surfaces are achieved, which significantly improves the spraying uniformity and coverage effect. Then, the trajectory of the spraying operation is tracked and controlled, which greatly reduces the system delay and improves the control accuracy. In this way, by combining multi-sensor fusion technology and deep learning methods, high-precision surface reconstruction and real-time quality monitoring are achieved, which solves the problem of difficult to achieve accurate trajectory planning for complex surfaces in existing technologies and ensures the stability of spraying quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0060] Figure 1 A schematic diagram of a flow chart of a curve trajectory painting method based on machine vision provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of the construction process of a three-dimensional model of a workpiece in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of the construction process of a multi-agent game model in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of the process of trajectory optimization in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of the structure of a curved trajectory painting system based on machine vision provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the structure of a processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] First of all, it should be noted that in the description of this application, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or may also include elements that are inherent to such process, method, article or system.

[0068] Figure 1 A flow chart of a curve trajectory spray painting method based on machine vision is provided in an embodiment of the present application. This embodiment provides a curve trajectory spray painting method based on machine vision, including steps S1 to S5.

[0069] S1, obtaining workpiece data of a workpiece to be sprayed, wherein the workpiece data includes a multi-view image from an industrial camera and workpiece depth data from a depth camera;

[0070] In this application, multiple industrial cameras are arranged to shoot the workpiece at different positions and angles to obtain multi-view images. Each camera is equipped with an LED structured light source to enhance the capture effect of surface details. For example, four industrial cameras can be arranged: Camera 1 is installed at the top front view position, with a resolution of 4096×3072 pixels and a ring LED light source; Camera 2 and Camera 3 are installed at 45 degrees to the left and right, respectively, with a resolution of 3840×2880 pixels and a strip LED light source; Camera 4 is mounted on the robotic arm as a mobile camera, with a resolution of 4096×3072 pixels and a dot matrix LED light source.

[0071] In addition, a binocular depth camera is used to collect 3D information of the workpiece. It should be noted that the depth camera uses binocular stereo vision technology to obtain an accurate depth map through parallax calculation, and the depth resolution can reach 0.1mm. Among them, the baseline distance of the depth camera is 120mm, the working distance is 400-800mm, the depth resolution can reach 0.1mm, and the field of view is 60°×45°. Through this multi-sensor collaborative collection method, complete workpiece surface information can be obtained.

[0072] S2, constructing a three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data;

[0073] In step S2, the system processes and three-dimensionally reconstructs the collected workpiece data. Before model reconstruction, this embodiment pre-processes the collected workpiece data, specifically including:

[0074] Using a preset adaptive Gaussian filtering algorithm to reduce noise on the multi-view images;

[0075] The preset multi-scale Retinex algorithm is used to perform illumination compensation on the denoised multi-view images;

[0076] The preset Zhang calibration algorithm is used to perform distortion correction on the multi-view image after illumination compensation to obtain the target multi-view image.

[0077] In this embodiment, an adaptive Gaussian filter algorithm is used to perform noise reduction on multi-view images. By dynamically adjusting the parameters of the filter kernel, noise can be effectively suppressed while retaining image details. In this process, the standard deviation σ of the Gaussian filter is set to 1.2, which can achieve a good balance between noise reduction effect and edge retention. Next, a multi-scale Retinex algorithm is used for illumination compensation. The algorithm uses three scale parameters {5, 20, 240} to effectively eliminate the influence of uneven illumination and improve the contrast and detail performance of the image. Then, distortion correction is performed based on Zhang's calibration method, and a geometrically accurate image is obtained by optimizing the reprojection error to be less than 0.1 pixel.

[0078] Specifically, an adaptive Gaussian filter algorithm is used to reduce the noise of multi-view images. First, the noise level of different areas in the image is automatically evaluated, and the parameters of the Gaussian filter kernel are dynamically adjusted according to the local image features. For areas with rich details, a smaller filter kernel is used to maintain edge and texture information; while for smooth areas, a larger filter kernel is used to enhance the noise reduction effect. Practice shows that when the standard deviation σ of the Gaussian kernel is set to 1.2, the best balance can be achieved between maintaining image details and suppressing noise. In addition, an adaptive weight mechanism based on image gradient is introduced to make the filter strength adaptively change with the image content, effectively avoiding edge blur.

[0079] Then, the multi-scale Retinex algorithm is used to perform illumination compensation on the denoised image. In the specific implementation, three different scale parameters are selected: 5 is used to enhance local details, 20 is used to adjust the contrast of the medium scale, and 240 is used to correct the uneven illumination of the large scale. The processing results of these three scales are fused through adaptive weights, and the weight coefficients are dynamically adjusted according to the local features of the image. At the same time, a color restoration module is added to the algorithm to ensure that the original color characteristics are maintained while enhancing the image contrast. Through this multi-scale processing method, the influence of uneven illumination is effectively eliminated and the overall visual quality of the image is improved.

[0080] Then, the preset Zhang calibration algorithm is used to correct the distortion of the illumination-compensated image. First, multiple sets of calibration images are collected through the calibration plate, the corner features are extracted, and the mapping relationship between the world coordinate system and the image coordinate system is established. Then, the intrinsic parameter matrix and distortion coefficient of the camera are solved by minimizing the reprojection error. It should be noted that a complete model considering radial distortion and tangential distortion is adopted, and the reprojection error is controlled within 0.1 pixel through iterative optimization, ensuring the geometric accuracy of the correction result. In addition, the system also establishes a lookup table for distortion correction, which significantly improves the real-time processing efficiency.

[0081] Therefore, this embodiment ensures data quality and improves the accuracy of subsequent processing by performing denoising, illumination compensation and distortion correction on workpiece data.

[0082] In some embodiments, after data preprocessing is completed, Figure 2 This is a schematic diagram of the construction process of the three-dimensional model of a workpiece in an embodiment of the present application. This embodiment provides a method for constructing a three-dimensional model of a workpiece, and step S2 includes steps S201 to S203.

[0083] S201, extracting features from the target multi-view image to obtain image feature data, where the image feature data is used to indicate a correspondence between images from different viewpoints;

[0084] S202, performing triangulation on the image feature data and the workpiece depth data to generate a dense point cloud;

[0085] S203 , performing surface reconstruction on the dense point cloud to obtain a three-dimensional model of the workpiece to be sprayed.

[0086] In this embodiment, the SIFT algorithm is used for feature extraction and matching, which can extract about 20,000 stable key points between images of different viewing angles and achieve a matching accuracy of more than 85%. Based on the matched feature points and depth information, a dense point cloud is generated by the triangulation principle, with an average point cloud density of 5 points / mm² and a total of about 3 million points. In order to improve the efficiency of subsequent processing, an octree-based point cloud simplification algorithm is used to optimize the data volume while maintaining geometric features. Finally, the surface is reconstructed using an improved Poisson surface reconstruction algorithm. The reconstructed surface model has a grid resolution of 0.5 mm and G2-level continuity, and the normal vector accuracy can reach ±0.5°.

[0087] Specifically, feature extraction and matching are performed on the corrected multi-view images. First, the SIFT algorithm is used to extract feature points in each image, and usually about 20,000 stable key points can be obtained. These feature points contain not only position information, but also scale and direction information, and have strong perspective invariance. Then, an improved nearest neighbor ratio matching strategy is used for feature matching, and geometric consistency constraints are introduced to improve the accuracy of matching. In this way, image feature data is extracted, which is used to indicate the correspondence between images (or the above-mentioned feature points) at different perspectives. Practice has shown that this method can achieve a feature matching accuracy of more than 85%, laying a solid foundation for subsequent three-dimensional reconstruction.

[0088] Furthermore, triangulation is performed based on the matched feature points and depth information to generate a dense point cloud. Specifically, the multi-view geometric relationship is first calculated through the matched feature points to establish accurate correspondence between images. Then, the three-dimensional coordinates of the feature points are calculated through the principle of triangulation in combination with the depth information obtained by the depth camera. In order to obtain a denser point cloud, a pixel-level matching algorithm based on the local plane assumption is used for encryption, so that the point cloud density reaches an average of 5 points / mm². At the same time, CUDA parallel computing technology is used to accelerate the point cloud generation process, significantly improving processing efficiency.

[0089] Furthermore, the generated dense point cloud is reconstructed. First, the system uses a statistical outlier filtering algorithm to remove noise points, and uses an octree-based point cloud simplification algorithm to optimize the data volume, which improves the efficiency of subsequent processing while maintaining geometric features. Then, the system uses an improved Poisson surface reconstruction algorithm for surface reconstruction, and introduces curvature constraints to improve the smoothness and accuracy of the reconstructed surface. The reconstructed surface model has a grid resolution of 0.5mm and G2-level continuity, and the normal vector accuracy can reach ±0.5°. Therefore, the reconstructed surface model (i.e., the three-dimensional model of the workpiece) contains normal vector information, which provides a geometric basis for subsequent trajectory planning. In addition, the system has established an adaptive mesh refinement mechanism to increase the mesh density in areas with large curvature changes, ensuring the reconstruction quality of complex areas.

[0090] Therefore, this embodiment constructs a three-dimensional model of the workpiece through feature matching, point cloud generation and surface reconstruction, which improves the accuracy of the model and helps to improve the accuracy of subsequent trajectory planning.

[0091] S3. Construct a multi-agent game model according to the three-dimensional model of the workpiece, and determine the spraying trajectory of the workpiece to be sprayed based on the multi-agent game model.

[0092] Specifically, trajectory planning is performed based on the reconstructed three-dimensional model. The workpiece surface (such as the spraying area) is divided into multiple overlapping sub-areas, each of which is 80 mm wide. Appropriate overlap is maintained between adjacent areas to ensure the continuity of spraying, thereby constructing a multi-agent game framework. Exemplarily, the multi-agent game framework can be set up with a three-layer structure. The top-level structure is responsible for global trajectory planning. By analyzing the overall geometric features of the workpiece, the main spraying path and key posture points are determined; the middle-level structure handles the coordination between agents, solves the conflict problem in the overlapping area, and ensures the uniformity of the spraying; the bottom-level structure focuses on local trajectory optimization, and adjusts the spraying parameters and motion trajectory according to the local surface features.

[0093] In some embodiments, Figure 3This is a schematic diagram of the construction process of a multi-agent game model in an embodiment of the present application. This embodiment provides a method for constructing a multi-agent game model, and step S3 includes steps S301 to S307.

[0094] S301, constructing a multi-agent game framework according to the workpiece three-dimensional model, wherein the multi-agent game framework is used to instruct multiple agents configured therein to generate initial candidate trajectories according to normal vector information and local curvature features of the workpiece three-dimensional model;

[0095] S302, simulating the projection distribution of the spraying fan on the curved surface by a preset Monte Carlo algorithm to determine the spraying coverage target function;

[0096] S303, taking curvature continuity and tangential acceleration constraint as evaluation indicators, and determining a trajectory smoothness objective function based on the evaluation indicators;

[0097] S304, determining a robot motion efficiency objective function, wherein the robot motion efficiency objective function is used to evaluate the joint speed, acceleration and energy consumption of the robot;

[0098] S305, determining a paint utilization rate objective function according to a pre-established dynamic relationship model between spraying amount and moving speed;

[0099] S306, performing weighted integration on the spray coverage objective function, the trajectory smoothness objective function, the robot motion efficiency objective function, and the paint utilization objective function to obtain a game utility function model;

[0100] S307, performing hybrid game optimization on the multi-agent game framework and the game utility function model to obtain a multi-agent game model.

[0101] In this embodiment, a multi-agent game framework is constructed based on the reconstructed three-dimensional model (i.e., the three-dimensional model of the workpiece). Specifically, the entire spraying area is divided into multiple overlapping sub-areas, and the overlapping width of 80 mm is maintained between adjacent sub-areas to ensure the continuity of the spraying. Each sub-area is responsible for an agent, and the agent generates an initial candidate trajectory based on the surface features in the area. It should be noted that there is a competitive and cooperative relationship between the agents: competition is reflected in the competition for shared resources (such as robot movement time and spraying space), and cooperation is reflected in the need to ensure the smooth transition and uniform coverage of adjacent trajectory segments.

[0102] In addition, a game framework with a hierarchical structure was constructed. For example, the top-level structure is responsible for global trajectory planning, which determines the main spraying direction and key posture points by analyzing the overall geometric features of the workpiece to ensure coverage integrity; the middle-level structure handles the coordination between agents, mainly solving the resource competition problem in the overlapping area, including motion time allocation and spatial path planning, and resolving conflicts; the bottom-level structure focuses on local trajectory optimization, adjusting the spraying parameters and motion trajectory according to the local features of the surface to ensure motion smoothness. Therefore, the adoption of this hierarchical architecture can effectively balance the needs of global planning and local optimization.

[0103] Specifically, in step S302, in order to consider the spray coverage, the Monte Carlo method is used to simulate the projection distribution of the spray fan on the surface, and the spray coverage objective function is constructed. First, a mathematical model of the spray fan is established, considering the influence of the opening angle, distance, posture and other parameters of the spray gun on the fan shape. Then, the spray coverage is evaluated through a large number of random sampling points, and the effective coverage area ratio is calculated. It should be noted that the influence of the surface inclination angle on the coverage effect is considered in the evaluation process, and a weight adjustment mechanism based on the normal vector is introduced to ensure the accuracy of the coverage evaluation.

[0104] Then, the system constructed a trajectory smoothness objective function. This function uses curvature continuity and tangential acceleration constraints as the main evaluation indicators. In the specific implementation, the local curvature change rate of the trajectory is first calculated to ensure that the trajectory has G2-level geometric continuity. Secondly, the tangential acceleration change of the trajectory during execution is analyzed, and by setting reasonable acceleration constraints, the robot's violent movement is avoided. In addition, the velocity smoothness constraint is introduced to ensure a smooth transition of the velocity at the turning point of the trajectory.

[0105] Furthermore, the system established a robot motion efficiency objective function. This function comprehensively evaluates the robot's joint velocity, acceleration and energy consumption. First, the velocity and acceleration distribution of each joint is calculated through kinematic analysis to ensure that it does not exceed the robot's dynamic constraints. Then, based on the joint torque and velocity information, an energy consumption evaluation model is established to optimize the robot's motion efficiency. In particular, the singularity and operational performance indicators of the robot in different postures are taken into account to avoid unfavorable postures during work.

[0106] Then, the paint utilization objective function is determined based on the pre-established dynamic relationship model between the spraying amount and the moving speed. The model is fitted by a large amount of experimental data and accurately describes the paint deposition law at different movement speeds. Based on the model, the spraying speed and flow parameters are dynamically adjusted to maximize the paint utilization while ensuring the coating quality. In addition, an adaptive adjustment mechanism based on coating thickness feedback is established to optimize the spraying parameters in real time.

[0107] Furthermore, each objective function is weighted and integrated to construct a comprehensive game utility function model. Specifically, a dynamic weight allocation strategy is adopted to adjust the weights of each objective function according to the spraying requirements of different areas. In key areas or complex curved areas, the weights of coverage and smoothness are increased; while in flat areas, the weights of efficiency and utilization are relatively increased. At the same time, the system introduces a weight optimization mechanism based on deep reinforcement learning, and continuously improves the weight allocation strategy through online learning.

[0108] It should be noted that the composite game utility function is used to evaluate and optimize trajectory quality. For example, the weight of the spray coverage objective function is 0.4, and the Monte Carlo method is used to simulate the projection distribution of the spray fan to evaluate the completeness of the coverage; the weight of the trajectory smoothness objective function is 0.3, which ensures the smoothness of the movement based on the curvature continuity and tangential acceleration constraints; the weight of the motion efficiency objective function is 0.2, which comprehensively considers the joint speed, acceleration and energy consumption; the weight of the paint utilization objective function is 0.1, and the material utilization efficiency is optimized by establishing a dynamic relationship model between the spraying amount and the moving speed.

[0109] Furthermore, the multi-agent game framework and the game utility function model are optimized through hybrid game to obtain a multi-agent game model.

[0110] Therefore, this embodiment converts the model information of the workpiece three-dimensional model into the optimal spraying trajectory, takes into account the static geometric constraints, and also realizes dynamic trajectory optimization through the multi-agent game framework. The output painting trajectory contains dynamic information and quality evaluation indicators, which provides a reliable optimization basis for the subsequent construction of the recursive Gaussian process state space model. At the same time, the whole process builds a bridge from geometric features to motion planning, ensuring the high-quality completion of the spraying task and improving the spraying efficiency and accuracy.

[0111] Based on the above embodiments, in some embodiments, step S307 includes:

[0112] Based on the multi-agent game framework, a preset response dynamics algorithm is used to solve the strategy combination, wherein the strategy combination is used to instruct each agent to iteratively update its respective strategy to maximize the game utility function model. The multi-agent game framework includes a top-level structure, a middle-level structure and a bottom-level structure. The top-level structure is used to indicate global trajectory planning, the middle-level structure is used to indicate coordination between processing agents, and the bottom-level structure is used to indicate local trajectory optimization.

[0113] In this embodiment, the best response dynamics method is used to solve the Nash equilibrium, and each agent iteratively updates its own strategy to maximize the utility function. Specifically, each agent iteratively updates its own strategy to achieve local optimality based on the current state and the game utility function. Then, by introducing an artificial potential field to assist in the search, the convergence speed of the algorithm is accelerated. In particular, an adaptive learning rate adjustment mechanism is adopted to dynamically adjust the parameters according to the convergence situation during the optimization process to effectively avoid local optimality. Finally, the optimal trajectory sequence represented by a NURBS curve is output, which not only meets the geometric continuity requirements, but also contains complete speed and acceleration planning information.

[0114] Therefore, this embodiment realizes adaptive control of different curvature areas, improves the accuracy of the spraying trajectory, ensures the uniformity of the spraying effect, and improves the spraying efficiency.

[0115] S4. According to the spraying trajectory, a recursive Gaussian process state space model is constructed, and the spraying trajectory is optimized based on the recursive Gaussian process state space model, wherein the control instruction sequence includes joint data for controlling the robot joints, posture data for controlling the end effector of the robot, and operation data for controlling the spray gun.

[0116] In this application, a recursive Gaussian process state space model is constructed for real-time trajectory optimization. The state vector of the model contains kinematic parameters such as the position, posture, and speed of the robot end effector, as well as process parameters such as spray gun pressure, flow rate, and atomization rate. A sparse variational Gaussian process algorithm is used to establish a predictive controller, and 100 induction points are set for dynamic characteristic modeling. In the actual control process, a 20-step prediction time domain and a 5-step control time domain are used, with a sampling period of 10ms. The system state is updated in real time through a sliding time domain observation window, and the state uncertainty is evaluated based on Bayesian inference.

[0117] In some embodiments, Figure 4 Schematic diagram of the trajectory optimization process in the embodiment of the present application. The present embodiment provides a trajectory optimization method, wherein step S4 includes steps S401 to S404.

[0118] S401, constructing a state vector according to the spraying trajectory, wherein the state vector includes kinematic parameters of the robot end effector and process parameters of the spray gun;

[0119] S402, constructing a recursive Gaussian process state space model based on the state vector and the acquired conditional variables, wherein the conditional variables include the workpiece surface curvature and material properties of the workpiece to be sprayed;

[0120] S403, using a preset sparse variational Gaussian process algorithm to model the system dynamic characteristics and obtain a predictive controller;

[0121] S404, performing adaptive trajectory optimization on the spraying trajectory based on the recursive Gaussian process state space model and the predictive controller to obtain a control instruction sequence.

[0122] It should be noted that in terms of kinematic parameters, the state vector includes information such as position coordinates, Euler angle posture, linear velocity and angular velocity. In terms of process parameters, key process variables such as spray gun pressure, flow rate, atomization rate, etc. are included. Optionally, workpiece surface feature parameters, such as local curvature, normal vector, etc., are also introduced as extended components of the state vector. Optionally, environmental information such as workpiece surface curvature and material properties are also introduced as conditional variables to improve the accuracy of system modeling. Therefore, this embodiment provides a sufficient information basis for subsequent system modeling and control optimization. Subsequently, based on the state vector and conditional variables such as workpiece surface curvature and material properties, a recursive Gaussian process state space model is constructed to improve the accuracy of system modeling.

[0123] Furthermore, the dynamic characteristics of the system are modeled using a preset sparse variational Gaussian process algorithm to obtain a predictive controller. The predictive controller adopts a hierarchical structure. The prediction layer builds the prior distribution of the system dynamic characteristics based on historical data, and selects the most representative set of induction points through the variational inference method, which significantly reduces the computational complexity. The optimization layer adopts the model predictive control framework to solve the optimal control sequence in the prediction time domain.

[0124] For example, the prediction layer uses sparse variational Gaussian process (SVGP) to represent the system dynamic characteristics, and optimizes the induction points and hyperparameters by maximizing the evidence lower bound (ELBO). The optimization layer constructs an objective function that considers multiple constraints and dynamically adjusts it through online learning. In particular, the predictive controller also includes an adaptive learning module that can update the kernel function parameters of the Gaussian process model in real time according to the control effect.

[0125] Furthermore, the spraying trajectory is adaptively optimized based on the recursive Gaussian process state space model and predictive controller to obtain a control instruction sequence. The instructions in the control instruction sequence are represented by piecewise continuous spline functions to ensure the smoothness of the robot's motion. Specifically, they include: the target angle of each joint and its derivative; the pose matrix SE(3) of the end effector and its derivative; the process parameter vector of the spray gun, including dynamic adjustment quantities such as pressure and flow. At the same time, an uncertainty estimate based on the Gaussian process is also output to provide a reliability assessment basis for subsequent execution control. In particular, the generation of control instructions takes into account the robot's dynamic constraints and actuator saturation limits, and the executable nature of the instructions is ensured through quadratic programming optimization.

[0126] Therefore, this embodiment realizes real-time optimization of the trajectory sequence through the recursive Gaussian process state space model, establishes a dynamic model of the system based on multi-sensor feedback, and realizes robust trajectory tracking through predictive control and adaptive optimization. The output control instruction sequence not only contains complete kinematic and process parameters, but also provides uncertainty assessment, providing a reliable execution basis for subsequent trajectory tracking control. The whole process builds a bridge from optimized trajectory to real-time control, ensuring the precise execution and stable quality of the spraying process.

[0127] Based on the above embodiments, in some embodiments, step S404 includes:

[0128] Construct a sliding time domain observation window;

[0129] Based on the sliding time-domain observation window, the recursive Gaussian process state-space model and the predictive controller, a preset Bayesian inference algorithm is used to update system state data and to evaluate uncertainty data used to characterize state uncertainty;

[0130] According to the updated system status data, the trajectory parameters of the spraying trajectory are adjusted, wherein the trajectory parameters include speed planning parameters, acceleration smoothing parameters and posture compensation parameters;

[0131] The weight of the trajectory smoothness constraint of the spraying trajectory is dynamically adjusted according to the uncertainty data.

[0132] In this embodiment, the sliding time domain observation window integrates multi-source feedback information from visual sensors, force sensors, etc. Exemplarily, a 20-step prediction time domain and a 5-step control time domain can be set, and the sampling period is 10ms. Through the sliding window mechanism, the system can capture the dynamic changes of the system state in a timely manner and provide accurate state estimation for real-time control. In addition, an adaptive window mechanism based on data credibility is established to dynamically adjust the size of the observation window according to the quality of the measurement data.

[0133] Then, based on the updated system state data, the trajectory parameters are dynamically adjusted, including velocity planning, acceleration smoothing, attitude compensation, etc. The system state is estimated a priori using a predictive controller, and then updated a posteriori in combination with multi-sensor feedback information. In this process, the system simultaneously evaluates the uncertainty of the state estimation and provides reliability indicators for subsequent trajectory optimization. In particular, the improved unscented Kalman filter algorithm is used to process the nonlinear observation equation, which improves the accuracy of state estimation.

[0134] Furthermore, specifically, according to the uncertainty of state estimation, the constraint weights in trajectory optimization are dynamically adjusted. When the state estimation is more reliable, the system pays more attention to tracking accuracy; when there is greater uncertainty, the weight of smoothness constraints is increased. At the same time, a robustness guarantee mechanism based on Lyapunov stability theory is established to ensure that the system can maintain stable operation in the presence of model errors and external disturbances. Finally, an optimized control instruction sequence is generated, including complete control quantities such as joint angles, end postures, and spray gun parameters, providing accurate instruction basis for actual execution.

[0135] S5. Track and control the spraying operation, wherein the spraying operation is used to instruct the workpiece to be sprayed to be sprayed according to the control instruction sequence.

[0136] In this application, trajectory tracking control based on AMS-PiM technology is implemented. Exemplarily, the controller used for trajectory tracking control adopts an ADC with 8-bit resolution and 1MSPS sampling rate, with an effective number of bits reaching 7.4 bits and a power consumption of only 4.2mW. The analog in-memory computing accelerator adopts a 32×32 cross-coupled current mirror array with a bias current of 10μA, a bandwidth of 2MHz, and a nonlinear error controlled within 0.5%. At the same time, real-time quality monitoring and parameter optimization are achieved. The visual monitoring system has a sampling rate of 60fps, a defect detection accuracy of more than 95%, and a response delay of less than 16.7ms. Based on the detection results, the system uses a fuzzy neural network to dynamically optimize the spraying process parameters, with an adjustment period of 50ms, ensuring the stability of the spraying quality.

[0137] In some embodiments, step S5 includes:

[0138] Based on the control instruction sequence, an AMS-PiM trajectory tracking controller is constructed, wherein a state feedback channel of the AMS-PiM trajectory tracking controller is used to indicate that a sensor signal generated after executing the control instruction sequence is converted into a current domain, and a switched capacitor network model for indicating dynamic error compensation is introduced into a feedback loop of the AMS-PiM trajectory tracking controller;

[0139] Constructing a cross-coupled current mirror array, and constructing an analog in-memory computing accelerator based on the cross-coupled current mirror array, wherein the cross-coupled current mirror array is used to instruct to perform a matrix multiplication operation in the analog domain;

[0140] Based on the AMS-PiM trajectory tracking controller and the analog in-memory computing accelerator, the spraying operation is tracked and controlled.

[0141] In this embodiment, a trajectory tracking controller based on AMS-PiM technology is constructed. The AMS-PiM trajectory tracking controller adopts an innovative analog in-memory computing architecture, the core of which is a low-precision ADC array and a computing unit without floating-point / post-training quantization. In the specific implementation, the system uses an 8-bit resolution SAR ADC, the sampling rate reaches 1MSPS, the effective number of bits is 7.4 bits, and the overall power consumption is only 4.2mW. Through improved capacitor matching technology and dynamic error correction, the quantization error is guaranteed to be within a controllable range while reducing the accuracy. The state feedback channel of the AMS-PiM trajectory tracking controller adopts current mode operation to directly convert the sensor signal into the current domain for processing, which significantly reduces the number of AD / DA conversions. In order to suppress the influence of quantization noise, a dynamic error compensation mechanism based on a switched capacitor network is introduced in the feedback loop, so that the nonlinear error is controlled within 0.5%, which effectively suppresses the influence of quantization noise on control performance.

[0142] Then, an analog in-memory computing accelerator is constructed. In this embodiment, a cross-coupled current mirror array is constructed to realize analog in-memory computing, which can directly complete matrix multiplication operations in the analog domain. It should be noted that the core of the accelerator is a 32×32 cross-coupled current mirror array, which can directly complete matrix multiplication operations in the analog domain. The bias current of the current mirror is set to 10μA, and the multiplication operation is realized through a differential pair structure, and the bandwidth reaches 2MHz. Adaptive bias technology is used to dynamically adjust the bias current according to the amplitude of the input signal, optimizing power consumption while ensuring the calculation accuracy. It is worth noting that a local feedback loop is introduced in the array, and the voltage and current of key nodes are monitored in real time to effectively compensate for the effects of temperature drift and process deviation. At the same time, the symmetrical design layout scheme minimizes parasitic effects and signal crosstalk.

[0143] Furthermore, the optimized trajectory is converted into a form suitable for simulation calculation, and then fast matrix operations are performed through the current mirror array to achieve trajectory tracking control. During the control process, piecewise continuous spline functions are used to represent control instructions, including the target angles of each joint and their derivatives, the pose matrix of the end effector and its derivatives, and the process parameters of the spray gun. At the same time, the control error is evaluated in real time and the control parameters are adjusted dynamically to ensure tracking accuracy. In particular, a stability analysis framework based on Lyapunov theory is established, and the stability of the control system is guaranteed by online monitoring of the changes in the energy function.

[0144] Therefore, this embodiment realizes efficient execution of the control instruction sequence by constructing an AMS-PiM trajectory tracking controller. Low-latency trajectory tracking control is realized at the hardware level, and closed-loop control of spraying quality is realized through visual feedback. The whole process establishes an efficient channel from control instructions to actual execution, ensuring the accuracy and stability of the spraying process. In particular, through the low-latency characteristics of analog calculations and the real-time nature of visual feedback, precise spraying control of complex surfaces is achieved, realizing closed-loop control.

[0145] In some embodiments, the method further comprises:

[0146] Preprocessing the operation images collected during the spraying operation;

[0147] Detect the edges and defects of the spraying area in the pre-processed working image and generate corresponding spraying status data;

[0148] Conduct quality assessment of spray uniformity of spray status data;

[0149] According to the quality assessment results, the preset fuzzy neural network is used to dynamically optimize the spraying process parameters in the spraying operation.

[0150] In this embodiment, the visual monitoring system constructed by multiple industrial cameras has a sampling rate of 60fps and can analyze the spraying quality in real time. The improved YOLOv5 algorithm is used for defect detection, with a detection accuracy of more than 95%, and the response delay is controlled within 16.7ms. Based on the detection results (i.e., spraying state data), the fuzzy neural network is used to dynamically optimize the spraying process parameters, and the parameter adjustment cycle is 50ms. During the optimization process, a mapping relationship model between the spraying parameters and the quality indicators is established, and then the parameter adjustment amount is calculated by the back propagation algorithm based on the quality deviation detected in real time. Taking into account the process stability requirements, multiple constraints are introduced in the parameter adjustment process, including the spray gun pressure change rate limit, flow stability requirements, atomization effect control, etc., to ensure the stability and controllability of the adjustment process.

[0151] Therefore, this embodiment realizes dynamic optimization of the spraying process parameters in the spraying operation by detecting the edges and defects of the spraying area in the spraying operation, ensures the stability and controllability of the adjustment process, improves the spraying efficiency, and ensures the spraying effect.

[0152] For example, the process of painting the curved surface of a car hood is used as an example. First, in the data collection stage, in the painting workstation, four industrial cameras are arranged at different positions of the workstation for the car hood to be painted: Camera 1 (top view): resolution 4096×3072 pixels, equipped with a ring LED light source; Cameras 2 and 3 (45 degrees left and right): resolution 3840×2880 pixels, equipped with a strip LED light source; Camera 4 (mobile camera): mounted on a robotic arm, resolution 4096×3072 pixels, equipped with a dot matrix LED light source. Secondly, a binocular depth camera is used for 3D information collection: baseline distance: 120mm; working distance: 400-800mm; depth resolution: 0.1mm; field of view: 60°×45°. Therefore, the workpiece data actually collected includes: 12 high-definition images (3 for each fixed camera and 3 for the mobile camera at different angles); 8 depth maps (collected by the binocular camera at 4 key positions); workpiece size: 1400mm×1200mm; surface curvature variation range: 0.001-0.05mm^-1.

[0153] Then, in the 3D reconstruction stage, the collected data is processed and reconstructed. First, image preprocessing: Adaptive Gaussian filtering is used for noise reduction, σ=1.2; Multi-scale Retinex algorithm is used for illumination compensation, scale parameters: {5, 20,240}; Distortion correction is performed based on Zhang calibration method, and the reprojection error is <0.1 pixel. Then, feature matching and point cloud generation: Extract SIFT feature points: about 20,000 key points; Feature matching accuracy: >85%; Generate point cloud density: average 5 points / mm²; Total number of point clouds: about 3 million points. Finally, surface reconstruction: Improved Poisson reconstruction algorithm is used; Grid resolution: 0.5mm; Surface smoothness: G2 continuous; Normal vector accuracy: ±0.5°.

[0154] In the trajectory planning stage, trajectory planning is performed based on the reconstructed 3D model. First, the area is divided: the hood surface is divided into 12 overlapping sub-areas, the overlapping area width is 80mm, and an intelligent agent is configured in each sub-area. Then, in the game optimization process, the utility function weight is configured: spray coverage weight: 0.4; trajectory smoothness weight: 0.3; motion efficiency weight: 0.2; paint utilization weight: 0.1. Among them, the optimization parameters include: number of iterations: 200; convergence threshold: 1e-4; learning rate: initial 0.01, dynamic adjustment. The final generation: main trajectory segments: 24; NURBS curve control points: 15-20 per segment; trajectory spacing: theoretical overlap rate 30%; trajectory smoothness: G2 continuity.

[0155] In the real-time optimization stage, the recursive Gaussian process state space model configuration: state vector, including position, velocity, quaternion attitude and process parameters. Control parameter settings: prediction time domain: 20 steps; control time domain: 5 steps; sampling period: 10ms;

[0156] Number of SVGP induction points: 100. Optimization target weights: trajectory tracking error: 0.5; velocity smoothness: 0.3; attitude continuity: 0.2.

[0157] In the execution control stage, the AMS-PiM hardware configuration is as follows: ADC parameters: resolution: 8 bits; sampling rate: 1MSPS; effective number of bits: 7.4 bits; power consumption: 4.2mW. Current mirror array: scale: 32×32; bias current: 10μA; bandwidth: 2MHz; nonlinear error: <0.5%. Real-time monitoring parameters: visual sampling rate: 60fps; defect detection accuracy: >95%; response delay: <16.7ms; parameter adjustment cycle: 50ms.

[0158] In this regard, through the above configuration and parameter settings, the system achieves the following effects when spraying the car hood: coverage uniformity: coating thickness variation coefficient <5%; trajectory accuracy: position error <0.3mm, posture error <0.5°; work efficiency: single hood spraying time <8 minutes; paint utilization: >85%; defect rate: <0.5%. Therefore, the feasibility and superiority of this application in practical applications, especially in the precise spraying control of complex surfaces.

[0159] Figure 5 A schematic diagram of a curved trajectory painting system based on machine vision provided in an embodiment of the present application, and Figure 6 A schematic diagram of the structure of a processing device provided in an embodiment of the present application, the curved trajectory painting system 50 based on machine vision comprises: a processing device 51, an industrial camera 52 and a depth camera 53;

[0160] The industrial camera 52 is used to obtain multi-view images of the workpiece to be sprayed;

[0161] The depth camera 53 is used to obtain the workpiece depth data of the workpiece to be sprayed;

[0162] The processing device 51 is connected to the industrial camera and the depth camera, and the processing device includes:

[0163] A data acquisition module 511 is used to acquire workpiece data of the workpiece to be sprayed, wherein the workpiece data includes the multi-view image and the workpiece depth data;

[0164] A three-dimensional reconstruction module 512 is used to construct a three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data;

[0165] A trajectory planning module 513 is used to construct a multi-agent game model according to the three-dimensional model of the workpiece, and determine the spraying trajectory of the workpiece to be sprayed based on the multi-agent game model;

[0166] A trajectory optimization module 514 is used to construct a recursive Gaussian process state space model according to the spraying trajectory, and optimize the spraying trajectory based on the recursive Gaussian process state space model to obtain a control instruction sequence, wherein the control instruction sequence includes joint data for controlling robot joints, posture data for controlling the end effector of the robot, and operation data for controlling the spray gun;

[0167] The trajectory tracking module 515 is used to track and control the trajectory of the spraying operation, and the spraying operation is used to instruct the spraying of the workpiece to be sprayed according to the control instruction sequence.

[0168] In some embodiments, the processing device 51 further includes:

[0169] A noise reduction module, used for reducing noise on the multi-view images by using a preset adaptive Gaussian filtering algorithm;

[0170] An illumination compensation module is used to perform illumination compensation on the denoised multi-view images using a preset multi-scale Retinex algorithm;

[0171] The distortion correction module is used to use a preset Zhang calibration algorithm to perform distortion correction on the multi-view image after illumination compensation to obtain a target multi-view image.

[0172] In some embodiments, the 3D reconstruction module 512 includes:

[0173] A feature extraction unit, used to extract features from the target multi-view image to obtain image feature data, wherein the image feature data is used to indicate the correspondence between images of different viewpoints;

[0174] A triangulation unit, used for performing triangulation on the image feature data and the workpiece depth data to generate a dense point cloud;

[0175] The surface reconstruction unit is used to perform surface reconstruction on the dense point cloud to obtain a three-dimensional model of the workpiece to be sprayed.

[0176] In some embodiments, the trajectory planning module 513 includes:

[0177] A multi-agent game framework construction unit, used to construct a multi-agent game framework according to the workpiece three-dimensional model, wherein the multi-agent game framework is used to instruct multiple agents configured therein to generate initial candidate trajectories according to normal vector information and local curvature features of the workpiece three-dimensional model;

[0178] A spray coverage target function determination unit, used to simulate the projection distribution of the spray fan on the curved surface through a preset Monte Carlo algorithm to determine the spray coverage target function;

[0179] A trajectory smoothness objective function determination unit, configured to use curvature continuity and tangential acceleration constraints as evaluation indicators and determine a trajectory smoothness objective function based on the evaluation indicators;

[0180] A robot motion efficiency objective function determination unit, used to determine the robot motion efficiency objective function, wherein the robot motion efficiency objective function is used to evaluate the joint speed, acceleration and energy consumption of the robot;

[0181] A paint utilization rate target function determination unit, used to determine the paint utilization rate target function according to a pre-established dynamic relationship model between the spraying amount and the moving speed;

[0182] A game utility function model construction unit is used to perform weighted integration on the spray coverage objective function, the trajectory smoothness objective function, the robot motion efficiency objective function and the paint utilization objective function to obtain a game utility function model;

[0183] The hybrid game optimization unit is used to perform hybrid game optimization on the multi-agent game framework and the game utility function model to obtain a multi-agent game model.

[0184] In some embodiments, the hybrid game optimization unit includes:

[0185] A solving unit is used to solve the strategy combination based on the multi-agent game framework using a preset response dynamics algorithm, wherein the strategy combination is used to instruct each agent to iteratively update its own strategy to maximize the game utility function model, and the multi-agent game framework includes a top-level structure, a middle-level structure and a bottom-level structure, wherein the top-level structure is used to instruct global trajectory planning, the middle-level structure is used to instruct coordination between processing agents, and the bottom-level structure is used to instruct local trajectory optimization.

[0186] In some embodiments, the trajectory optimization module 514 includes:

[0187] A state vector construction unit, used to construct a state vector according to the spraying trajectory, wherein the state vector includes kinematic parameters of the robot end effector and process parameters of the spray gun;

[0188] A recursive Gaussian process state space model building unit, used to build a recursive Gaussian process state space model based on the state vector and the acquired conditional variables, wherein the conditional variables include the workpiece surface curvature and material properties of the workpiece to be sprayed;

[0189] A predictive controller building unit, used to model the dynamic characteristics of the system using a preset sparse variational Gaussian process algorithm to obtain a predictive controller;

[0190] An adaptive trajectory optimization unit is used to perform adaptive trajectory optimization on the spraying trajectory based on the recursive Gaussian process state space model and the predictive controller to obtain a control instruction sequence.

[0191] In some embodiments, the adaptive trajectory optimization unit includes:

[0192] A window construction unit, used to construct a sliding time domain observation window;

[0193] An evaluation unit, configured to update system state data using a preset Bayesian inference algorithm based on the sliding time domain observation window, the recursive Gaussian process state space model and the predictive controller, and to evaluate uncertainty data for characterizing state uncertainty;

[0194] A trajectory adjustment unit, used to adjust the trajectory parameters of the spraying trajectory according to the updated system status data, wherein the trajectory parameters include speed planning parameters, acceleration smoothing parameters and posture compensation parameters;

[0195] A weight adjustment unit is used to dynamically adjust the weight of the trajectory smoothness constraint of the spraying trajectory according to the uncertainty data.

[0196] In some embodiments, the trajectory tracking module 515 includes:

[0197] An AMS-PiM trajectory tracking controller construction unit is used to construct an AMS-PiM trajectory tracking controller based on the control instruction sequence, wherein a state feedback channel of the AMS-PiM trajectory tracking controller is used to indicate that a sensor signal generated after executing the control instruction sequence is converted into a current domain, and a switch capacitor network model for indicating dynamic error compensation is introduced into a feedback loop of the AMS-PiM trajectory tracking controller;

[0198] An accelerator construction unit, used to construct a cross-coupled current mirror array, and to construct an analog in-memory computing accelerator based on the cross-coupled current mirror array, wherein the cross-coupled current mirror array is used to instruct to perform a matrix multiplication operation in the analog domain;

[0199] A control unit is used to track and control the spraying operation based on the AMS-PiM trajectory tracking controller and the analog in-memory computing accelerator.

[0200] In some embodiments, the processing device 51 further includes:

[0201] A preprocessing module, used for preprocessing the operation images collected during the spraying operation;

[0202] A detection module is used to detect the edges and defects of the spraying area in the pre-processed working image and generate corresponding spraying status data;

[0203] A quality assessment module is used to conduct quality assessment on the spraying uniformity of the spraying status data;

[0204] The dynamic optimization module is used to dynamically optimize the spraying process parameters in the spraying operation according to the quality evaluation results using a preset fuzzy neural network.

[0205] The industrial camera uses a high-resolution CMOS sensor, equipped with a professional-grade lens and LED light source to obtain multi-view images of the workpiece; the depth camera uses a binocular stereo vision solution and has high-precision depth measurement capabilities. The processing equipment integrates data acquisition modules, 3D reconstruction modules, trajectory planning modules, trajectory optimization modules and trajectory tracking modules, and realizes the coordinated work of various functional modules through high-performance processors and dedicated accelerators. The modules of the system are connected through a high-speed bus to ensure the real-time and reliability of data transmission.

[0206] Through the above technical solution, this application realizes precise spraying control on complex curved surfaces. In practical applications, the system can ensure that the coefficient of variation of coating thickness is less than 5%, the position error is less than 0.3mm, the posture error is less than 0.5°, the paint utilization rate exceeds 85%, and the defect rate is less than 0.5%, which significantly improves the spraying quality and efficiency.

[0207] The system of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the system of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the system, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0208] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A curve trajectory painting method based on machine vision, characterized in that: For processing equipment, including: Acquire workpiece data of the workpiece to be sprayed, wherein the workpiece data includes a multi-view image from an industrial camera and workpiece depth data from a depth camera; Constructing a three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data; According to the three-dimensional model of the workpiece, a multi-agent game model is constructed, and a spraying trajectory of the workpiece to be sprayed is determined based on the multi-agent game model; According to the spraying trajectory, a recursive Gaussian process state space model is constructed, and the spraying trajectory is optimized based on the recursive Gaussian process state space model to obtain a control instruction sequence, wherein the control instruction sequence includes joint data for controlling a robot joint, posture data for controlling an end effector of the robot, and operation data for controlling a spray gun; Tracking and controlling the spraying operation, wherein the spraying operation is used to instruct the workpiece to be sprayed to be sprayed according to the control instruction sequence; Wherein, constructing a multi-agent game model according to the workpiece three-dimensional model includes: According to the three-dimensional model of the workpiece, a multi-agent game framework is constructed, wherein the multi-agent game framework is used to instruct multiple agents configured therein to generate initial candidate trajectories according to the normal vector information and local curvature features of the three-dimensional model of the workpiece; wherein the entire spraying area is divided into multiple overlapping sub-areas, each of which is in charge of an agent; The projection distribution of the spray fan on the surface is simulated by the preset Monte Carlo algorithm to determine the spray coverage objective function; Taking curvature continuity and tangential acceleration constraint as evaluation indicators, and determining a trajectory smoothness objective function based on the evaluation indicators; Determine a robot motion efficiency objective function, wherein the robot motion efficiency objective function is used to evaluate the joint speed, acceleration and energy consumption of the robot; Determine the paint utilization rate objective function based on the pre-established dynamic relationship model between the spraying amount and the moving speed; The spray coverage objective function, the trajectory smoothness objective function, the robot motion efficiency objective function and the paint utilization objective function are weighted and integrated to obtain a game utility function model; Performing hybrid game optimization on the multi-agent game framework and the game utility function model to obtain a multi-agent game model; Wherein, constructing a recursive Gaussian process state space model according to the spraying trajectory, optimizing the spraying trajectory based on the recursive Gaussian process state space model, and obtaining a control instruction sequence, including: According to the spraying trajectory, a state vector is constructed, wherein the state vector includes kinematic parameters of the robot end effector and process parameters of the spray gun; constructing a recursive Gaussian process state space model based on the state vector and the acquired conditional variables, wherein the conditional variables include the workpiece surface curvature and material properties of the workpiece to be sprayed; The preset sparse variational Gaussian process algorithm is used to model the dynamic characteristics of the system and obtain a predictive controller; Based on the recursive Gaussian process state space model and the predictive controller, the spraying trajectory is adaptively optimized to obtain a control instruction sequence.

2. The method according to claim 1, characterized in that Before constructing the three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data, the method further includes: Using a preset adaptive Gaussian filtering algorithm to reduce noise on the multi-view images; The preset multi-scale Retinex algorithm is used to perform illumination compensation on the denoised multi-view images; The preset Zhang calibration algorithm is used to perform distortion correction on the multi-view image after illumination compensation to obtain the target multi-view image.

3. The method according to claim 2, characterized in that The step of constructing a three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data comprises: Extracting features from the target multi-view images to obtain image feature data, where the image feature data is used to indicate a corresponding relationship between images from different viewpoints; Performing triangulation on the image feature data and the workpiece depth data to generate a dense point cloud; The dense point cloud is reconstructed into a curved surface to obtain a three-dimensional model of the workpiece to be sprayed.

4. The method according to claim 1, characterized in that: The hybrid game optimization is performed on the multi-agent game framework and the game utility function model to obtain a multi-agent game model, including: Based on the multi-agent game framework, a preset response dynamics algorithm is used to solve the strategy combination, wherein the strategy combination is used to instruct each agent to iteratively update its respective strategy to maximize the game utility function model. The multi-agent game framework includes a top-level structure, a middle-level structure and a bottom-level structure. The top-level structure is used to indicate global trajectory planning, the middle-level structure is used to indicate coordination between processing agents, and the bottom-level structure is used to indicate local trajectory optimization.

5. The method according to claim 1, characterized in that The method of performing adaptive trajectory optimization on the spraying trajectory based on the recursive Gaussian process state space model and the predictive controller to obtain a control instruction sequence includes: Construct a sliding time domain observation window; Based on the sliding time-domain observation window, the recursive Gaussian process state-space model and the predictive controller, a preset Bayesian inference algorithm is used to update system state data and to evaluate uncertainty data used to characterize state uncertainty; According to the updated system status data, the trajectory parameters of the spraying trajectory are adjusted, wherein the trajectory parameters include speed planning parameters, acceleration smoothing parameters and posture compensation parameters; The weight of the trajectory smoothness constraint of the spraying trajectory is dynamically adjusted according to the uncertainty data.

6. The method according to claim 1, characterized in that The trajectory tracking and control of the spraying operation includes: Based on the control instruction sequence, an AMS-PiM trajectory tracking controller is constructed, wherein a state feedback channel of the AMS-PiM trajectory tracking controller is used to indicate that a sensor signal generated after executing the control instruction sequence is converted into a current domain, and a switched capacitor network model for indicating dynamic error compensation is introduced into a feedback loop of the AMS-PiM trajectory tracking controller; Constructing a cross-coupled current mirror array, and constructing an analog in-memory computing accelerator based on the cross-coupled current mirror array, wherein the cross-coupled current mirror array is used to instruct to perform a matrix multiplication operation in the analog domain; Based on the AMS-PiM trajectory tracking controller and the analog in-memory computing accelerator, the spraying operation is tracked and controlled.

7. The method according to claim 1, characterized in that The method further comprises: Preprocessing the operation images collected during the spraying operation; Detect the edges and defects of the spraying area in the pre-processed working image and generate corresponding spraying status data; Conduct quality assessment of spray uniformity of spray status data; According to the quality assessment results, the preset fuzzy neural network is used to dynamically optimize the spraying process parameters in the spraying operation.

8. A curve trajectory painting system based on machine vision, characterized in that: include: Processing equipment, industrial cameras, and depth cameras; The industrial camera is used to obtain multi-view images of the workpiece to be sprayed; The depth camera is used to obtain workpiece depth data of the workpiece to be sprayed; The processing device is connected to the industrial camera and the depth camera, and the processing device includes: A data acquisition module, used for acquiring workpiece data of the workpiece to be sprayed, wherein the workpiece data includes the multi-view images and the workpiece depth data; A three-dimensional reconstruction module, used for constructing a three-dimensional model of the workpiece to be sprayed according to the multi-view images and the workpiece depth data; A trajectory planning module, used to construct a multi-agent game model according to the three-dimensional model of the workpiece, and determine the spraying trajectory of the workpiece to be sprayed based on the multi-agent game model; A trajectory optimization module, used for constructing a recursive Gaussian process state space model according to the spraying trajectory, and performing trajectory optimization on the spraying trajectory based on the recursive Gaussian process state space model to obtain a control instruction sequence, wherein the control instruction sequence includes joint data for controlling robot joints, posture data for controlling the end effector of the robot, and operation data for controlling a spray gun; A trajectory tracking module, used for tracking and controlling the trajectory of a spraying operation, wherein the spraying operation is used to instruct the spraying of the workpiece to be sprayed according to the control instruction sequence; The trajectory planning module is used to construct a multi-agent game model according to the three-dimensional model of the workpiece, including: According to the three-dimensional model of the workpiece, a multi-agent game framework is constructed, wherein the multi-agent game framework is used to instruct multiple agents configured therein to generate initial candidate trajectories according to the normal vector information and local curvature features of the three-dimensional model of the workpiece; wherein the entire spraying area is divided into multiple overlapping sub-areas, each of which is in charge of an agent; The projection distribution of the spray fan on the surface is simulated by the preset Monte Carlo algorithm to determine the spray coverage objective function; Taking curvature continuity and tangential acceleration constraint as evaluation indicators, and determining a trajectory smoothness objective function based on the evaluation indicators; Determine a robot motion efficiency objective function, wherein the robot motion efficiency objective function is used to evaluate the joint speed, acceleration and energy consumption of the robot; Determine the paint utilization rate objective function based on the pre-established dynamic relationship model between the spraying amount and the moving speed; The spray coverage objective function, the trajectory smoothness objective function, the robot motion efficiency objective function and the paint utilization objective function are weighted and integrated to obtain a game utility function model; Performing hybrid game optimization on the multi-agent game framework and the game utility function model to obtain a multi-agent game model; The trajectory optimization module is used to construct a recursive Gaussian process state space model according to the spraying trajectory, and optimize the spraying trajectory based on the recursive Gaussian process state space model to obtain a control instruction sequence, including: According to the spraying trajectory, a state vector is constructed, wherein the state vector includes kinematic parameters of the robot end effector and process parameters of the spray gun; constructing a recursive Gaussian process state space model based on the state vector and the acquired conditional variables, wherein the conditional variables include the workpiece surface curvature and material properties of the workpiece to be sprayed; The preset sparse variational Gaussian process algorithm is used to model the dynamic characteristics of the system and obtain a predictive controller; Based on the recursive Gaussian process state space model and the predictive controller, the spraying trajectory is adaptively optimized to obtain a control instruction sequence.

Images