Cooperative reinforcement learning control method and system for automobile spraying task

Abstract

The invention discloses a collaborative reinforcement learning control method and system for an automobile spraying task, and relates to the technical field of automatic spraying, and the method comprises the steps: obtaining image data of a to-be-sprayed vehicle at a plurality of visual angles, and carrying out the preprocessing, and obtaining enhanced image data; performing three-dimensional reconstruction based on the enhanced image data to obtain a three-dimensional grid model; paint film thickness data in the spraying process are collected in real time, and a dynamic paint film thickness map is generated; based on the three-dimensional grid model and the dynamic paint film thickness map, the multi-arm collaborative spraying task is converted into a multi-agent Markov decision model, and a control instruction is output; the strategy network of the intelligent agent executes the control instruction, obtains local observation information and generates an initial action; inputting all the local observation information and the initial action into a centralized intelligent network to obtain a global value; and updating parameters of the strategy network by taking global value maximization as a target, guiding an intelligent agent to generate a current optimal action, and executing a corresponding spraying task. The coating quality is improved; the service life of equipment is prolonged.

Description

Technical Field

[0001] This invention relates to the field of automated spraying technology, and more specifically to a collaborative reinforcement learning control method and system for automotive spraying tasks. Background Technology

[0002] Automotive painting is a crucial process in automobile manufacturing, directly determining the vehicle's appearance quality and corrosion resistance. The technology in this field has evolved from manual painting to automated robotic painting. Currently, mainstream automated painting production lines generally employ single-arm or dual-arm six-axis industrial robots, executing pre-programmed painting trajectories through a "teach-and-playback" method. In this mode, engineers manually guide the robot, recording the spray gun's movement path, posture, and process parameters such as spray flow rate and atomization pressure point by point, forming a fixed program for a specific vehicle model. During production, the robot precisely repeats this program, replacing manual labor, thereby improving production efficiency and freeing workers from harmful environments filled with volatile organic compounds (VOCs).

[0003] However, despite the significant advancements in existing automated spraying technology compared to manual spraying, its "open-loop" control nature exposes the following core shortcomings when addressing the ever-increasing demands for quality and flexible production:

[0004] 1. High dependence on programming and poor production flexibility: Existing technology heavily relies on time-consuming and labor-intensive manual teaching programming. Whenever a new model needs to be introduced or an existing model needs to be modified, experienced engineers must spend weeks or even months reprogramming and debugging. This makes the production line slow to respond when switching models, making it difficult to adapt to the flexible production needs of modern manufacturing for small batches, multiple varieties, and personalized customization.

[0005] 2. Poor Coating Quality on Complex Curved Surfaces: Automotive bodies are covered with complex free-form surfaces and structural details, such as rearview mirrors, door handles, body seams, and waistlines. Single-arm or dual-arm robots, with fixed programs, struggle to maintain the optimal distance and perpendicular orientation of the spray gun to the target surface in these areas, often leading to under-spraying, over-spraying, and uneven paint film thickness, severely impacting appearance quality. This is especially true at surface boundaries, where fixed path planning algorithms often neglect geometric boundaries, resulting in deteriorated coating uniformity.

[0006] 3. Poor adaptability to dynamic disturbances and frequent defects: Spraying is a dynamic process affected by multiple variables. The temperature and humidity of the workshop environment, airflow disturbances, and batch differences in the viscosity of the paint itself all affect the final paint film quality. The preset fixed trajectory cannot sense these real-time changes and make adjustments, resulting in the continued prevalence of defects such as orange peel, sagging, pinholes, and fisheyes.

[0007] 4. Rigid task allocation limits equipment lifespan and efficiency: In dual-arm collaborative systems, tasks are typically statically allocated, easily leading to uneven workloads between the two arms. Over long-term operation, some robots may experience accelerated wear of critical components (such as servo motors and harmonic reducers) due to overuse, while others may remain under low load for extended periods. This uneven wear results in significant differences in the mean time between failures (MTBF) between the arms, increasing the risk of unplanned downtime and maintenance costs, and shortening the overall lifespan of the system.

[0008] Therefore, how to achieve intelligent control of the automotive painting process, optimize collaborative efficiency, and thus improve coating quality and equipment lifespan is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] In view of the above problems, this invention is proposed to provide a collaborative reinforcement learning control method and system for automotive painting tasks that overcomes or at least partially solves the above problems. By combining multi-view three-dimensional perception, collaborative reinforcement learning and real-time closed-loop feedback, intelligent control of the automotive painting process is achieved, collaborative efficiency is optimized, and coating quality and equipment life are improved.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a collaborative reinforcement learning control method for automotive painting tasks, comprising: Image data of the vehicle to be painted from multiple perspectives are acquired and preprocessed to obtain enhanced image data; Based on the enhanced image data, a three-dimensional reconstruction is performed to obtain a three-dimensional mesh model of the vehicle to be painted; Real-time acquisition of paint film thickness data during the spraying process, combined with the three-dimensional mesh model, generates a dynamic paint film thickness map; Based on the three-dimensional mesh model and the dynamic paint film thickness map, the multi-arm collaborative spraying task is transformed into a multi-agent Markov decision model, and control commands are output. The control instructions are executed based on the policy network of all agents, local observation information is acquired, and initial actions are generated. Based on all the local observation information and the initial action input, the global value is obtained by inputting them into a centralized intelligent network; The parameters of the policy network are updated based on the goal of maximizing the global value, guiding the agent to generate the current optimal action and execute the corresponding painting task.

[0011] In one embodiment, the enhanced image data is obtained by: Multiple sets of overlapping high-resolution image data are acquired from multiple preset or dynamically planned perspectives around the vehicle to be painted; Based on the image data, image scaling, normalization, and channel normalization operations are performed to obtain preprocessed image data; Data augmentation is performed on the preprocessed image data to obtain the augmented image data.

[0012] In one embodiment, the method for constructing the three-dimensional mesh model is as follows: Based on the enhanced image data, the SfM algorithm is used to identify and match key feature points between different views; Based on the geometric relationship of the matching key feature points, triangulation is performed to obtain the six-degree-of-freedom pose of the camera when capturing each image, which is used as the camera calibration parameters; Based on the camera calibration parameters, the MVS algorithm is used to perform dense, pixel-level matching between overlapping images to generate dense point clouds. The dense point clouds generated from various perspectives are fused to obtain a continuous overall model, and a Poisson surface reconstruction algorithm is used to fit a continuous and smooth surface to obtain the three-dimensional mesh model.

[0013] In one embodiment, the method for obtaining the multi-agent Markov decision model is as follows: Based on the three-dimensional mesh model, the dynamic paint film thickness map, and the motion state of the robotic arm agent, a joint state space is obtained. The set of joint actions taken by all the robotic arm agents is used as the joint action space; The probability distribution of the system transitioning to the next state after all the robotic arm agents perform a joint action in the current state is used as the state transition function; Set a set of reward functions for each robotic arm agent; Based on the joint state space S and the joint action space The state transition function P and the set of reward functions R The multi-agent Markov decision model consists of: ; in, c Indicates the discount factor. n This represents the total number of robotic arm agents that make independent decisions within the system.

[0014] In one embodiment, the team reward function is obtained by summing the reward function sets of each robotic arm agent. All the robotic arm agents share the same team reward function. R total : ; in, oh 1. oh 2. oh 3. oh 4 and oh 5 represents the corresponding weighting coefficient. This represents the variance of the dynamic paint film thickness map. This indicates the new surface area covered by the paint. n s Indicates the number of main spray arms. n f Indicates the number of fine spraying arms. W i This represents the cumulative workload of each spraying arm agent. This represents the average load of all the spray arm agents. Indicates a safety reward. This indicates the amount of paint that was wasted because it failed to adhere to the vehicle body.

[0015] In one embodiment, the local observation information o i for: ; in, r i Represents the joint angle vector. v i Represents the joint velocity vector. p i Indicates the attitude of the end nozzle. I i Represents a local image segment. n i This represents the normal vector of the sprayed surface. d i This represents the vector from the nozzle to the target point. c i This indicates the coverage value of a localized area of ​​the coating. M thick This represents a dynamic paint film thickness graph. This represents the relative position vector with other robotic arm agents. Indicates the current task phase. t Represents a timestamp; The distributed intelligent agent network is implemented using a hybrid deep network architecture: a multilayer perceptron is used to extract features of vector and scalar data in the local observation information to obtain multi-source features, and a convolutional neural network is used to extract features of image data and graph data in the local observation information to obtain spatial features; The initial action is obtained by concatenating the multi-source features and the spatial features and then inputting them into a multilayer perceptron.

[0016] In one embodiment, the global value acquisition method is: Global state information is generated by stitching together all the aforementioned local observation information. Based on the concatenation of all the initial actions, a combined action is generated; Based on the global state information and the input of the joint action, the centralized intelligent network is used to obtain the global value, which is used to evaluate the merits of the current joint action.

[0017] In one embodiment, the centralized intelligent network includes a multilayer perceptron and an action-value function network; Based on the multilayer perceptron, the global state information and the features of the joint action are extracted, and the extracted features are input into the action-value function network to output the global value. The centralized intelligent network updates parameters by minimizing the time-series difference error.

[0018] In one embodiment, it also includes: When the strategy network determines that the entire surface of the vehicle to be painted has reached the target thickness and uniformity standards, it automatically stops the painting process. Perform a full surface quality scan on the painted vehicle and generate a quality report.

[0019] In a second aspect, embodiments of the present invention provide a collaborative reinforcement learning control system for automotive painting tasks, comprising: an image acquisition and processing module, a vehicle 3D reconstruction module, a paint film data acquisition module, a control command output module, an initial action generation module, an initial action evaluation module, and an optimal action generation module; The image acquisition and processing module is used to acquire image data of the vehicle to be painted from multiple perspectives and preprocess it to obtain enhanced image data; The vehicle 3D reconstruction module is used to perform 3D reconstruction based on the enhanced image data to obtain a 3D mesh model of the vehicle to be painted. The paint film data acquisition module is used to collect paint film thickness data in real time during the spraying process and combine it with the three-dimensional mesh model to generate a dynamic paint film thickness map. The control command output module is used to transform the multi-arm cooperative spraying task into a multi-agent Markov decision model based on the three-dimensional mesh model and the dynamic paint film thickness map, and output control commands. The initial action generation module is used to execute the control instructions based on the policy network of all agents, obtain local observation information, and generate initial actions; The initial action evaluation module is used to obtain the global value by feeding all the local observation information and the initial action input into the centralized intelligent network. The optimal action generation module is used to update the parameters of the policy network based on the goal of maximizing the global value, guide the agent to generate the current optimal action, and execute the corresponding painting task.

[0020] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a collaborative reinforcement learning control method and system for automotive painting tasks, which has the following beneficial effects: 1. Equipped with a multi-agent collaborative control framework, achieving full surface coverage and uniformity control, thus realizing high-quality spraying with "no dead angles": Existing technologies, when dealing with car bodies with complex geometric features, rely on traditional single-arm or dual-arm robots that follow fixed paths generated from CAD model slices. This makes it difficult to adapt to free-form surfaces with continuously changing curvature, especially in detailed areas such as door edges, rearview mirror housings, and grilles. Inconsistent coating thickness, overspray, or underspray can easily occur due to poor spray gun posture and distance. This invention solves this problem from a physical perspective by decoupling the functions of multiple arms (main spraying arm + fine spraying arm). Path planning based on reinforcement learning is no longer a rigid reproduction of geometric trajectories, but a dynamic process optimization. The agent can adjust the six-dimensional pose, operating speed, and paint flow rate of the spray gun in real time to ensure optimal spraying conditions on any curved surface, thereby fundamentally guaranteeing the uniformity of the paint film.

[0021] 2. Reinforcement Learning-Driven Spraying Path Optimization: Existing automated spraying systems suffer from a disconnect between execution and results. The system blindly repeats preset actions, lacking the ability to perceive quality issues caused by dynamic disturbances such as changes in environmental temperature, humidity, and paint viscosity. This leads to frequent defects like sagging, orange peel, and pinholes, ultimately requiring costly and delayed manual inspection and rework to compensate. This invention introduces a non-contact real-time thickness sensor mounted on a sensing arm, constructing a closed-loop control system. The actual paint film thickness, as key state information, is fed back to the reinforcement learning decision model in real time. This means the system has "eyes" and can "see" its work results. Once a thickness deviation is detected, the agent immediately adjusts the spraying strategy to compensate, significantly reducing rework costs and material waste.

[0022] 3. High-efficiency collaboration and dynamic load balancing significantly improve overall equipment efficiency and lifespan: Traditional dual-arm collaborative systems typically employ static task allocation logic. This not only leads to efficiency losses due to waiting between robotic arms but also causes severe load unevenness. Under long-term operation, the key components of the heavier-loaded robotic arm will wear out faster, resulting in a significantly lower mean time between failures (MTBF) than other robotic arms. This increases the risk of unplanned downtime and maintenance costs, ultimately reducing the lifespan of the entire system. The collaborative reinforcement learning framework of this invention solves this problem at the algorithmic level by explicitly introducing a load balancing term (Rbalance) into the reward function. While pursuing spraying efficiency and quality, the agent also actively and consciously distributes the workload evenly among all spraying arms. This intelligent dynamic task allocation not only shortens the production cycle by maximizing parallel operations but also ensures a "work-rest balance" among the robots, effectively preventing premature failure due to overwork of a single arm. This extends the lifespan of the entire system and reduces the high maintenance costs and spare parts inventory caused by uneven equipment wear.

[0023] 4. Improve coating quality and consistency: Through intelligent path planning and real-time closed-loop control, ensure that a uniform and precise paint film thickness is formed on all vehicle body surfaces, including complex curved surfaces, to prevent the occurrence of painting defects.

[0024] 5. Achieve true flexible production: Completely eliminate manual teaching and programming, enabling the system to generate the optimal painting strategy for any new model in real time through autonomous perception, and realize "batch size one" production.

[0025] 6. Establish a quality control system: Introduce a closed-loop control mechanism based on real-time sensory feedback to proactively predict and prevent defects. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This is a flowchart of a collaborative reinforcement learning control method for automotive painting tasks provided in an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the overall architecture of the multi-arm system provided in an embodiment of the present invention.

[0029] Figure 3This is a schematic diagram of a collaborative reinforcement learning control system for automotive painting tasks provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example 1 like Figure 1 As shown, this embodiment of the invention discloses a collaborative reinforcement learning control method for automotive painting tasks, including the following steps. For ease of description, these steps are numbered S1 to S7, and these numbers are not used to limit the sequential relationship between the various steps of this invention: S1 acquires image data of the vehicle to be painted from multiple perspectives and preprocesses it to obtain enhanced image data.

[0032] Furthermore, such as Figure 2 As shown, this invention employs n industrial robots (n=4 in this embodiment) to form a collaborative multi-arm system. To achieve efficient collaboration, the multi-arm system undergoes clear functional decoupling and specialized division of labor, dividing them into three roles: Main painting arm (ns station): Responsible for efficient painting of large, flat curved surfaces of the vehicle body; Fine coating arm (NF station): Responsible for handling complex geometries and hard-to-reach areas; Sensing and Assisting Arm (NP Stage): Responsible for environmental perception (3D modeling, paint film thickness monitoring) and assisting operation; The numerical relationship of the three types of robotic arms satisfies n = ns + nf + np, and their division of labor is shown in Table 1: Table 1: Configuration and Role Division of Multi-Arm Robot System

[0033] This configuration, through functional specialization, ensures comprehensive coverage and efficient execution of complex painting tasks.

[0034] Furthermore, the core of the overall architecture of the multi-arm system of the present invention consists of a spraying workstation and a central control system.

[0035] Inside the workstation, multiple robots work collaboratively around the car body to be painted, and are divided into main painting arms, fine painting arms, and sensing and auxiliary arms according to their functions.

[0036] The central control system is the brain of the system and consists of three core modules: a multi-view 3D modeling module, a collaborative reinforcement learning decision-making module, and a closed-loop control and feedback module.

[0037] The workflow is as follows: multi-view image data collected by the sensing and auxiliary arms is input into the modeling module, and after generating an accurate model, it is sent to the decision module; the decision module generates the optimal collaborative action instructions based on this model and the real-time paint film thickness data fed back by the sensing arm, and controls various robots to perform spraying, collection and auxiliary tasks respectively, forming a complete intelligent closed-loop control system.

[0038] Furthermore, the image data is enhanced, and the acquisition method is as follows: The high-resolution industrial optical camera mounted on the sensing and assisting arm (NP stage) collects multiple sets of overlapping high-resolution image data from multiple preset or dynamically planned perspectives around the vehicle to be painted. Image scaling, normalization, and channel normalization based on the ImageNet dataset are performed on the image data to eliminate differences under different camera and lighting conditions, resulting in preprocessed image data. Data augmentation based on preprocessed image data yields enhanced image data, which improves the robustness of subsequent models, making them less prone to overfitting. The enhanced image data can then be used to reconstruct the 3D model using computer vision techniques, enabling it to better generalize to the ever-changing real physical world.

[0039] Furthermore, data augmentation includes at least: geometric transformations (rotation, translation), color dithering, and simulated occlusion.

[0040] S2 performs 3D reconstruction based on enhanced image data to obtain a 3D mesh model of the vehicle to be painted.

[0041] Furthermore, the method for constructing the 3D mesh model is as follows: The SfM algorithm is used to identify and match key feature points between different views based on enhanced image data. Triangulation is performed based on the geometric relationship of matching key feature points to obtain the six-degree-of-freedom pose of the camera when capturing each image, which is used as camera calibration parameters; Based on camera calibration parameters, the MVS algorithm is used to perform dense, pixel-level matching between overlapping images to generate dense point clouds. A continuous overall model is obtained by fusing dense point clouds generated from various perspectives, and a Poisson surface reconstruction algorithm is used to fit a continuous and smooth surface to obtain a three-dimensional mesh model.

[0042] Furthermore, based on the enhanced image data, an image-based 3D reconstruction technique is used to generate a high-fidelity 3D mesh model of the vehicle to be painted. The 3D mesh model consists of a set of 3D vertices and triangles that define the shape of the vehicle body surface. Based on enhanced image data, the SfM algorithm is used to identify and match key feature points (such as SIFT or ORB features) between different views. By triangulating the geometric relationship of these matched key feature points, the algorithm can simultaneously estimate the sparse 3D point cloud structure of the scene, as well as the precise six-degree-of-freedom pose (position and orientation) of the camera when capturing each image, as camera calibration parameters. This step is the cornerstone of the entire reconstruction process and establishes the geometric constraint relationship between all captured views. Based on camera calibration parameters, the MVS algorithm is used to perform dense, pixel-level matching between overlapping images to generate a high-resolution dense point cloud that can capture rich details of the vehicle surface. Unlike the sparse point cloud generated by SfM, MVS aims to find the corresponding position of each pixel in the field of view in three-dimensional space. Dense point clouds generated from various perspectives are fused into a unified and coherent overall model. A Poisson surface reconstruction algorithm is then applied, and a continuous, smooth surface is fitted based on the normal vector information of the point cloud. Finally, a high-fidelity, watertight triangular mesh model is generated. M geo This model provides an authoritative and accurate environmental geometry representation for all subsequent path planning, collision detection, and simulation training.

[0043] Furthermore, the triangular mesh model M geo The expression is as follows: ; in, Each vertex represents its physical coordinates in three-dimensional space. , , f i Describe a triangular face whose three vertices are formed by... V The midpoint set consists of elements. ,in i 1. i 2. i 3 represents the index of a vertex in V.

[0044] S3 collects paint film thickness data in real time during the spraying process and combines it with a 3D mesh model to generate a dynamic paint film thickness map.

[0045] Furthermore, this invention also collects paint film thickness data in real time for model training. During the spraying process, a non-contact optical thickness sensor (such as a confocal chromatograph or laser triangulation sensor) mounted on the sensing and auxiliary arm scans the sprayed area in real time to collect the thickness data of the wet or dry film, which serves as the paint film thickness data. A three-dimensional mesh model is dynamically constructed and updated based on paint film thickness data. M geo Dynamic film thickness map corresponding to geometric structure M thick ; Dynamic film thickness chart M thick It is a 3D mesh model M geo A dynamic data structure that is completely topologically corresponding, recording the paint film thickness at various points on the vehicle body surface in real time, is the core of achieving closed-loop feedback control. It stores data related to... M geo The current paint film thickness value corresponding to each geometric element (usually each vertex or each face), in micrometers (μm); Before spraying begins ( t = 0), dynamic paint film thickness map M thick All values ​​are zero. During the spraying process, the sensing arm collects thickness data in real time to update the dynamic paint film thickness map. M thick Dynamic paint film thickness diagram M thick It can be implemented as a... M geo An array or hash table associated with vertices or faces, if associated with vertices, can be represented as a mapping from vertices to thickness values: .

[0046] S4, based on a 3D mesh model and dynamic paint film thickness map, transforms the multi-arm collaborative spraying task into a multi-agent Markov decision model and outputs control commands.

[0047] Furthermore, the method for obtaining the multi-agent Markov decision model is as follows: Based on the 3D mesh model, dynamic paint film thickness map, and motion state of the robotic arm agent, a joint state space is obtained. The joint action space is based on the set of joint actions taken by all robotic arm agents. The state transition function is based on the probability distribution of the system transitioning to the next state after all robotic arm agents perform joint actions in the current state. Set the set of reward functions for each robotic arm agent; Based on joint state space S Joint Action Space State transition function P and reward function set R Constructing a multi-agent Markov decision model: ; in, c Indicates the discount factor. n This represents the total number of robotic arm agents that make independent decisions within the system.

[0048] Furthermore, the joint state space S It is the set describing all possible configurations of the system at any time t. A specific state. s t ∈ S It is a high-dimensional vector containing all the information needed to make the optimal decision. Its structure is as follows: ; in, q n Represents the robotic arm intelligent agent n The kinematic state of the robotic arm agent is a vector containing the kinematic state of the robotic arm agent. n All joint angles and angular velocities, i.e. ,in, j Represents the robot's degrees of freedom. Indicates joint angle, This represents the joint angular velocity, which describes the robot's current physical posture and motion trend. 3D mesh model M geo As a static geometric model, serving as a static part of the environment, it provides the geometric basis for path planning and collision detection; Dynamic film thickness chart M thick At t=0, M thick All values ​​are zero; during the spraying process, this diagram is updated in real time by the sensor data of the sensing arm, which is the core of the invention to achieve closed-loop control.

[0049] Furthermore, joint action space It is the set of joint actions that all intelligent agents can take. Represents the robotic arm intelligent agent i The individual motion space; for the spraying arms (main spraying arm and fine spraying arm), their motion... It is a composite instruction vector containing continuous variables: ; Among them, △ q i The motion command is represented by a vector of the same dimension as the degrees of freedom of the robotic arm agent, representing the angle increment of each joint in the next time step; flow i It represents the paint flow rate, a scalar value that controls the amount of paint sprayed from the spray gun per unit time; pressure i This represents atomization pressure, a scalar value that controls the atomization effect of the coating. For the sensing and assisting arm, its movement... Mainly movement commands △ q i It is used to drive sensors to scan and monitor.

[0050] Furthermore, the state transition function P , Defined in state s t Execute joint actions a t Then, the system transitions to the next state. s t+1 The probability distribution. In deterministic environments such as robotic arm agent control, this function is typically deterministic. This function is determined by the underlying physical laws of the environment and is implicitly provided by the simulation engine during model training. Its transition process can be conceptually decomposed as follows: (1) State transition of the robotic arm agent: q i,t+1 Depend on q i,t and action △ q i,t This was calculated based on the dynamic model of the robotic arm's intelligent agent.

[0051] (2) Coating film thickness state transfer: M thick,t+1 Depend on M thick,t The position of the end effector is determined by the movement of the spray arm. When the spray arm performs the spraying action, the position of its end effector is determined by... q i,t+1 Calculated) and process parameters ( flow i , pressure i The dynamic paint film thickness map will be based on a preset paint deposition model (such as a Beta distribution model). M thick The thickness value is increased; simultaneously, the real-time measurement data from the sensing arm also affects the dynamic paint film thickness map. M thick Perform calibration and updates.

[0052] In the model-free reinforcement learning framework of this invention, there is no need to know the state transition function. P Instead of using explicit mathematical expressions, it directly samples state transition sequences by interacting with the simulation environment. .

[0053] Furthermore, It is the set of reward functions for each agent. Reward function R i This is designed to guide agents in learning desired cooperative behavior, and it is a multi-objective reward function. All agents share the same team reward. In order to promote cooperation.

[0054] Furthermore, the team reward function R total : ; in, oh 1. oh 2. oh 3. oh 4 and oh 5 represents the corresponding weighting coefficient. This represents the variance of the dynamic paint film thickness map. This indicates the new surface area covered by the paint. n s Indicates the number of main spray arms. n f Indicates the number of fine spraying arms. W i This represents the cumulative workload of each spraying arm agent. This represents the average load of all the spray arm agents. Indicates a safety reward. This indicates the amount of paint that was wasted because it failed to adhere to the vehicle body.

[0055] Furthermore, the team reward function R total middle: First item The balanced reward is the core quality objective of the entire reward function, which is calculated and penalized based on the dynamic paint film thickness map. M thick variance ( Var This is achieved by using a method that minimizes variance; the smaller the variance, the more uniform the coating, and the smaller the penalty (the higher the reward). This transforms the macroscopic, visual quality indicator of "coating uniformity" into a specific numerical objective that can be optimized by the algorithm, thereby guiding the reinforcement learning strategy to converge towards a high-quality spraying scheme. oh 1 is a weighting factor used to adjust the importance of this item in the total reward.

[0056] Second item A coverage reward is given to encourage the agent (robot) to explore and complete the coating of all target surfaces as quickly as possible; whenever a new surface area is reached... When a section of the paint is covered, the system awards a positive reward. This ensures the integrity of the painting task and effectively avoids missed areas caused by the strategy getting stuck in local optima. oh 2 is its corresponding weighting coefficient.

[0057] The third item is an efficiency reward (constant), primarily used to drive the model to make the correct decision on the shortest path. This is achieved by adding weights in the steps. oh 3. This avoids the model obtaining a correct but suboptimal solution with longer steps. At each decision step, a fixed deduction is applied to the reward function to incentivize the agent to find the most efficient solution.

[0058] Fourth item Load balancing rewards are used to optimize equipment health management and extend the entire lifecycle, by penalizing each spray arm (from the 1st to the 2nd). n s + n f Cumulative workload of (number) W i Average load of all spray arms This is achieved through the deviation between them. If a robotic arm "does too much work," widening the gap with other arms, the penalty value for this item will increase significantly. This guides the algorithm to consciously ensure a "work-rest balance" among the arms when assigning tasks, avoiding overwork of any single arm, thus incorporating equipment health management into the optimization objective. oh 4 is its weighting coefficient.

[0059] Fifth item This is a safety reward, a strong constraint used to ensure operational safety. When the system detects a collision between robotic arms or between a robotic arm and the vehicle body, or when the distance between them is less than a preset safety threshold, a large negative reward is applied. The penalty value is much larger than other rewards to ensure that the agent will avoid collisions at all costs during the learning process, guaranteeing that the ultimately learned strategy is inherently safe and collision-free.

[0060] Item 6 Incentivize material conservation by penalizing overspray (i.e., waste of paint that fails to adhere to the vehicle body). Economic costs and environmental factors were taken into account during the optimization process. In order to obtain higher overall rewards, the robotic arm's intelligent system was incentivized to optimize the spray gun's posture, distance, and process parameters to maximize paint transfer efficiency and reduce waste. oh 5 is its weighting coefficient.

[0061] Furthermore, the discount factor γ is a value between 0 and 1, which determines the importance of future rewards relative to current immediate rewards. In the objective function of reinforcement learning (i.e., maximizing cumulative rewards), the reward obtained at step t in the future is multiplied by γ when calculating the current value. c t .

[0062] Furthermore, n It is a positive integer representing the total number of robotic arm agents (robots) making independent decisions in the system. In a specific embodiment of the present invention, n =4.

[0063] Furthermore, based on the aforementioned data, the model is trained in a simulation environment. Training in the simulation environment aims to improve the model's decision-making capabilities until it can truly make correct decisions. Once training is complete, a multi-arm collaborative decision-making system is formed and ready for application in real-world scenarios.

[0064] The model is mainly divided into two parts: (1) Multi-agent Markov decision model for collaborative spraying: mainly outputs abstract instructions (such as moving to point X, spraying area X, etc.) based on the current state of the robotic arm. (2) Collaborative strategy optimization model based on centralized training and distributed execution: Based on the abstract instructions of the Markov decision model and combined with the agent network, specific motion parameters (how to move to point X, how to spray area X, etc.) are output. This part uses CNN as the backbone network, which is trained in the ROS+MultiROS simulation environment and then applied to the actual environment after manual fine-tuning.

[0065] S5 executes control commands based on the policy network of all agents, acquires local observation information, and generates initial actions.

[0066] Furthermore, multi-agent Markov decision models focus on the training process of the model (algorithm), which is relatively abstract (mainly outputting abstract instructions such as movement and spraying based on the current state). Their purpose is to train cooperation among multiple agents and encourage them to work together to solve problems. However, in practical applications, it is necessary to formulate detailed training strategies for each agent. To address the non-stationarity problem in multi-agent learning (i.e., the learning environment of each agent changes continuously due to policy updates from other agents), this invention adopts a centralized training with decentralized execution (CTDE) framework.

[0067] To ensure that each robotic arm performs a confirmed and effective action, the model assumes two types of agents: a centralized agent and a distributed agent. The distributed agent is each robotic arm, while the centralized agent is an agent used to score the distributed agents.

[0068] Furthermore, local observation information o i for: ; in, r i Represents the joint angle vector, ( j =1,2,3,...6 (a total of 6 degrees of freedom). v i Represents the joint velocity vector. p i Indicates the attitude of the end nozzle. I i Represents a local image segment. n i This represents the normal vector of the sprayed surface. d i This represents the vector from the nozzle to the target point. c i ∈[0,1] represents the local spray coverage value. M thick This represents a dynamic paint film thickness graph. This represents the relative position vector with other robotic arm agents. Indicates the current task phase (0, 1, 2, ...). t Represents a timestamp; The distributed agent network is implemented using a hybrid deep network architecture: a multilayer perceptron is used to extract features of vector and scalar data in local observation information to obtain multi-source features, and a convolutional neural network is used to extract features of image and graph data in local observation information to obtain spatial features; The initial action is obtained by concatenating multi-source features and spatial features and then inputting them into a multilayer perceptron.

[0069] Furthermore, the six dimensions of the end nozzle attitude represent the x-axis, y-axis, z-axis, roll angle (corresponding to the rolling along the vertical axis (x-axis), which determines how the object rolls around its own direction of travel), pitch angle (corresponding to the pitch motion along the horizontal axis (y-axis), which affects the pitch angle at which the robotic arm grasps the object), and yaw angle (corresponding to the left and right turning along the vertical axis (z-axis), which determines the direction of travel of the robot).

[0070] Furthermore, distributed intelligent agents: each robotic arm intelligent agent i It has an independent policy network The network relies solely on its own local observation information. o i To generate deterministic actions a i Among them, local observation information o i Structured tensors composed of multimodal information: .

[0071] Furthermore, policy networks This ensures that during the actual execution phase, each robotic arm agent can make decisions independently and quickly without the need for high-bandwidth global communication.

[0072] Furthermore, the distributed agent network employs a hybrid deep network architecture: for vector and scalar data (such as joint angles, velocities, and poses), a multilayer perceptron (MLP) is used for feature extraction. For image data (such as local image fragments) and graph data (such as paint film thickness maps), spatial features are first extracted using a convolutional neural network (CNN). Finally, all features extracted from the MLP and CNN are concatenated together and then passed through a final MLP network to output deterministic action commands.

[0073] Furthermore, the distributed agent network updates through policy gradient ascent, aiming to select actions that elicit high scores from the centralized agent. For robotic arm agents... i Its policy gradient Calculated according to the deterministic policy gradient theorem: ; in, This indicates the influence of the network parameters of a distributed intelligent agent on its output actions. The core component represents the "guidance" from the centralized agent to the distributed agent; it computes the distributed agent's... i action a i Small changes affect the overall value Qi The impact.

[0074] In order to take the action that scores the highest among centralized agents, the distributed agent network will follow the path that maximizes the global value. Q i The parameters are updated in the direction of the action, and the updated parameters are used to guide the distributed agent to make definite actions.

[0075] By alternately executing these two update steps in a simulation environment, the distributed agent network learns how to make high-quality decisions, while the centralized agent network learns how to accurately evaluate these decisions, ultimately forming an efficient and collaborative intelligent decision-making system.

[0076] S6 obtains global value by inputting all local observation information and initial actions into a centralized intelligent network.

[0077] Furthermore, the method for obtaining global value is as follows: Global state information is generated by stitching together all local observation information. Generate a combined action by concatenating all the initial actions; The global state information and joint actions are input into a centralized intelligent network to obtain a global value, which is used to evaluate the merits of the current joint actions.

[0078] Furthermore, centralized intelligent networks include multilayer perceptrons and action-value function networks; Global state information and joint action features are extracted based on a multilayer perceptron, and the extracted features are input into an action-value function network to output the global value. Centralized intelligent networks update parameters by minimizing temporal differential errors.

[0079] Furthermore, centralized agents: During the offline training phase, each robotic arm agent is equipped with a centralized action-value function network. This network is able to access global state information. (Including observations from all agents) and joint actions from all agents Through this access method, the centralized agent can provide each distributed agent with a stable and globally optimal gradient signal, effectively guiding the learning of cooperative strategies.

[0080] The centralized agent is implemented using a multilayer perceptron (MLP). Its task is to map the state and action information of all agents to a scalar Q-value. Similar to the policy network, its front end may also include modules such as CNNs to process the image and graph data in the input, concatenate the extracted features with the remaining vector data, and then feed them into the core MLP for processing. Its input contains two parts: global state information...x And the joint action of all intelligent agents; Global status information x Local observation information from all agents o i It is pieced together: ; Joint Action a The action chosen by all agents in the current state. a i It is pieced together: ; The output of a centralized agent network is a global value, denoted as . This value represents the global state. x Under these circumstances, all intelligent agents jointly execute coordinated actions. a This refers to the expected cumulative reward that the system can obtain in the future. It is a quantitative indicator for evaluating the "goodness" or "badness" of the current joint actions.

[0081] Furthermore, centralized intelligent networks update by minimizing the temporal difference error. Their loss function... Defined as the mean squared error between the predicted global value and the target global value: ; Where D is the experience replacement pool, and the target value is... y i The calculation method is as follows: ; in, This is the predicted value of the current centralized agent network; y i This is the numerical value used as the "correct answer" or "learning objective" when training centralized agent networks. Its calculation follows the Bellman equation in reinforcement learning, consisting of the immediate reward and the global value of the objective in the next state. composition; and These are a centralized target agent network and a distributed target agent network, respectively. Their parameters are the smoothed average of the main network, used to stabilize the learning process.

[0082] S7 updates the parameters of the policy network with the goal of maximizing global value, guiding the agent to generate the current optimal action and execute the corresponding painting task.

[0083] Furthermore, it also includes: automatically stopping the painting process when the strategy network determines that the entire surface of the vehicle body to be painted has reached the target thickness and uniformity standards; Perform a full surface quality scan on the painted vehicle and generate a quality report.

[0084] Furthermore, it also includes: Model training in a simulation environment: To train a Multi-Agent Markov Decision (MADDPG) model and an agent network.

[0085] By utilizing reinforcement learning techniques, collaboration among multi-armed robots is encouraged and strengthened; CTDE guides each arm of the robot to perform detailed, efficient, and correct movements. First, they need to be trained in a virtual environment to "learn" how to properly cooperate to complete car painting tasks, achieving multi-armed collaborative execution and closed-loop control, before being applied to the real world.

[0086] The entire learning process takes place in a high-fidelity physical simulation environment to safely and efficiently generate a large amount of training data. The simulation is performed using the Gazebo industrial-grade open-source simulation platform, equipped with a ROS+MultiROS multi-robot collaborative control environment. After the model is trained in the simulation environment, it undergoes secondary training and manual calibration in real-world application scenarios to ensure it meets actual production requirements. In the simulation environment, the Gazebo particle plugin simulates the spraying effect, and factors such as spraying speed, color, material, diffusion, and attenuation are calibrated manually to conform to real-world physical laws. Data from different vehicle models is used to simulate the curved surfaces and uneven structures of different vehicle types. Simultaneously, ambient lighting and material reflection are used to verify the spraying effect.

[0087] For model learning, the AdamW optimizer was used for model training. The learning rate of the distributed agent network (policy network) was set to 1×10⁻⁶. -4 The learning rate for centralized agent networks (value networks) is set to 1×10. -3 These values ​​are typical initial settings and will be fine-tuned through hyperparameter search during actual training. Simultaneously, a learning rate scheduling strategy is employed, using gradual warm-up in the early stages of training to stabilize the learning process, allowing the learning rate to increase linearly from an initial value of 0, reaching the preset base learning rate (1×10⁻⁶ for distributed agent networks) at the 10th epoch. -4 Centralized agent network: 1×10 -3 ).

[0088] By collecting data in advance, the real environment is accurately simulated. Then, each intelligent agent is initialized and its collaborative working ability is trained to achieve multi-arm collaborative execution and closed-loop control, thus forming a highly efficient and collaborative intelligent decision-making system.

[0089] Furthermore, after training is complete, the learned strategies are deployed into real robotic systems.

[0090] The core advantage of this invention lies in achieving true multi-arm collaboration, rather than simple task parallelism. This collaboration is manifested in three aspects: Unified Decision Making: At each decision-making moment, the reinforcement learning model outputs a joint action vector. This vector contains all n Synchronization commands for the robots; Implicit coordination: Since all robots' strategies are trained in a focused manner to maximize a common team reward function, their learned behaviors naturally include predictions and adaptations to teammates' actions. For example, when a main painting arm is painting a car door, its path may subconsciously leave collision-free workspace for the main painting arm on the other side, or utilize the overspray paint on the opposite side to improve overall efficiency, without any explicit communication or rules. Dynamic role allocation: During the spraying process, the model may dynamically adjust the task based on the real-time paint film thickness map. For example, if the main spray arm leaves a slight thickness deficiency in a certain area, the model will instruct the fine spray arm to perform a quick touch-up spray after completing its main task. This dynamic task allocation and collaboration is something that pre-programmed systems cannot achieve.

[0091] Example 2 After training, the learned strategies are deployed to a real robotic system. The overall workflow of this invention follows a clear and automated sequence, starting with the vehicle entering the spray booth and being precisely positioned.

[0092] The first stage is 3D modeling, in which the sensing arm acquires multi-view images of the vehicle body, and the central controller processes the image data to generate an accurate 3D model. The second stage is the initial planning, where the trained collaborative reinforcement learning model generates an initial collaborative spraying scheme based on the 3D model. The third stage is collaborative execution and real-time monitoring. The spraying arm starts working, while the sensing arm monitors the paint film thickness of the sprayed area in real time. The fourth stage is dynamic closed-loop adjustment. The monitored thickness data is fed back to the decision model in real time. The model continuously and dynamically adjusts the robot's actions and spraying parameters based on this feedback. This process is repeated until the entire car body reaches the preset quality standard. Once the system determines that the spraying is complete, the process enters the completion and verification phase. All robots stop working, and the sensing arm can perform a final comprehensive quality scan and generate a report. Finally, the system resets, ready to receive the next vehicle.

[0093] The specific process is as follows: (1) Initialization: The vehicle to be painted enters the painting booth and stops at the designated work station.

[0094] (2) 3D Modeling: Under the guidance of the central controller, the high-resolution industrial optical camera mounted on the sensing arm collects multiple sets of overlapping high-resolution image data from multiple different perspectives around the vehicle. The central controller adopts image-based 3D reconstruction technology. First, it uses Structure from Motion (SfM) technology to estimate the camera pose. Then, it uses the Multi-View Stereo (MVS) algorithm to generate a dense point cloud. Finally, it performs Poisson surface reconstruction to generate a 3D mesh model of the vehicle.

[0095] (3) Initial planning: Based on the initial state (when the paint film thickness map is all zero), the trained cooperative reinforcement learning policy network generates an initial cooperative motion trajectory and process parameter sequence for all spray arms. The policy network performs a complete "rollout" to generate a joint action sequence covering the entire task duration and including all spray arms. Among them, at any time step t joint action a t It includes all the independent movements of the spray arm: The movement of each spray arm is determined by its motion and process parameters: .

[0096] (4) Collaborative execution and real-time monitoring: The spraying arm begins to perform the spraying operation. At the same time, the non-contact optical thickness sensor (such as a confocal chromatography sensor) mounted at the end of the sensing arm starts the monitoring path and feeds back real-time paint film thickness data to the central controller.

[0097] (5) Dynamic closed-loop adjustment: The central controller updates the dynamic paint film thickness map in the status with new thickness data at each time step. M thick The strategy network then adjusts its actions based on the updated state output, thereby optimizing the spraying process in real time and continuously.

[0098] (6) Completion and Verification: When the strategy network determines that the entire vehicle body surface has reached the target thickness and uniformity standards, the painting process automatically stops. The sensing arm can perform a final full-surface quality scan to generate a quality report.

[0099] Example 3 like Figure 3 As shown, based on the same inventive concept, this embodiment of the invention also provides a collaborative reinforcement learning control system for automotive painting tasks, including: an image acquisition and processing module, a vehicle 3D reconstruction module, a paint film data acquisition module, a control command output module, an initial action generation module, an initial action evaluation module, and an optimal action generation module; The image acquisition and processing module is used to acquire and preprocess image data of the vehicle to be painted from multiple perspectives to obtain enhanced image data. The vehicle 3D reconstruction module is used to perform 3D reconstruction based on enhanced image data to obtain a 3D mesh model of the vehicle to be painted. The paint film data acquisition module is used to collect paint film thickness data in real time during the spraying process and combine it with a three-dimensional mesh model to generate a dynamic paint film thickness map. The control command output module is used to transform the multi-arm collaborative spraying task into a multi-agent Markov decision model based on a 3D mesh model and a dynamic paint film thickness map, and output control commands. The initial action generation module is used to execute control commands based on the policy network of all agents, acquire local observation information, and generate initial actions; The initial action evaluation module is used to obtain the global value based on all local observation information and initial action inputs fed into the centralized intelligent network; The optimal action generation module is used to update the parameters of the policy network with the goal of maximizing global value, guiding the agent to generate the current optimal action and execute the corresponding painting task.

[0100] Furthermore, the functional implementation of each functional module in this embodiment corresponds one-to-one with the above-described method, and will not be described in detail here.

[0101] Example 4 Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When the processor executes a program stored in memory, it can implement a cooperative reinforcement learning control method for an automotive painting task, as shown in Example 1.

[0102] The electronic device may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions in the memory to execute a cooperative reinforcement learning control method for automotive painting tasks, as described in Embodiment 1.

[0103] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0105] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A collaborative reinforcement learning control method for automotive painting tasks, characterized in that, include: Image data of the vehicle to be painted from multiple perspectives are acquired and preprocessed to obtain enhanced image data; Based on the enhanced image data, a three-dimensional reconstruction is performed to obtain a three-dimensional mesh model of the vehicle to be painted; Real-time acquisition of paint film thickness data during the spraying process, combined with the three-dimensional mesh model, generates a dynamic paint film thickness map; Based on the three-dimensional mesh model and the dynamic paint film thickness map, the multi-arm collaborative spraying task is transformed into a multi-agent Markov decision model, and control commands are output. The control instructions are executed based on the policy network of all agents, local observation information is acquired, and initial actions are generated. Based on all the local observation information and the initial action input, the global value is obtained by inputting them into a centralized intelligent network; The parameters of the policy network are updated based on the goal of maximizing the global value, guiding the agent to generate the current optimal action and execute the corresponding painting task.

2. The collaborative reinforcement learning control method for automotive painting tasks according to claim 1, characterized in that, The enhanced image data is obtained by the following method: Multiple sets of overlapping high-resolution image data are acquired from multiple preset or dynamically planned perspectives around the vehicle to be painted; Based on the image data, image scaling, normalization, and channel normalization operations are performed to obtain preprocessed image data; Data augmentation is performed on the preprocessed image data to obtain the augmented image data.

3. The collaborative reinforcement learning control method for automotive painting tasks according to claim 1, characterized in that, The method for constructing the three-dimensional mesh model is as follows: Based on the enhanced image data, the SfM algorithm is used to identify and match key feature points between different views; Based on the geometric relationship of the matching key feature points, triangulation is performed to obtain the six-degree-of-freedom pose of the camera when capturing each image, which is used as the camera calibration parameters; Based on the camera calibration parameters, the MVS algorithm is used to perform dense, pixel-level matching between overlapping images to generate dense point clouds. The dense point clouds generated from various perspectives are fused to obtain a continuous overall model, and a Poisson surface reconstruction algorithm is used to fit a continuous and smooth surface to obtain the three-dimensional mesh model.

4. The collaborative reinforcement learning control method for automotive painting tasks according to claim 1, characterized in that, The method for obtaining the multi-agent Markov decision model is as follows: Based on the three-dimensional mesh model, the dynamic paint film thickness map, and the motion state of the robotic arm agent, a joint state space is obtained. The joint action space is defined as the set of joint actions taken by all the robotic arm agents. The probability distribution of the system transitioning to the next state after all the robotic arm agents perform a joint action in the current state is used as the state transition function; Set a set of reward functions for each robotic arm agent; Based on the joint state space S and the joint action space The state transition function P and the set of reward functions R The multi-agent Markov decision model consists of: ； in, γ Indicates the discount factor. n This represents the total number of robotic arm agents that make independent decisions within the system.

5. The collaborative reinforcement learning control method for automotive painting tasks according to claim 4, characterized in that, The team reward function is obtained by summing the reward function sets of each robotic arm agent. All the robotic arm agents share the same team reward function. R total : ； in, ω 1. ω 2. ω 3. ω 4 and ω 5 represents the corresponding weighting coefficient. This represents the variance of the dynamic paint film thickness map. This indicates the new surface area covered by the paint. n s Indicates the number of main spray arms. n f Indicates the number of fine spraying arms. W i This represents the cumulative workload of each spraying arm agent. This represents the average load of all the spray arm agents. Indicates a safety reward. This indicates the amount of paint that was wasted because it failed to adhere to the vehicle body.

6. The collaborative reinforcement learning control method for automotive painting tasks according to claim 1, characterized in that, The local observation information o i for: ； in, r i Represents the joint angle vector. v i Represents the joint velocity vector. p i Indicates the attitude of the end nozzle. I i Represents a local image segment. n i This represents the normal vector of the sprayed surface. d i This represents the vector from the nozzle to the target point. c i This indicates the coverage value of a localized area of ​​the coating. M thick This represents a dynamic paint film thickness graph. This represents the relative position vector with other robotic arm agents. Indicates the current task phase. t Represents a timestamp; The distributed intelligent agent network is implemented using a hybrid deep network architecture: a multilayer perceptron is used to extract features of vector and scalar data in the local observation information to obtain multi-source features, and a convolutional neural network is used to extract features of image data and graph data in the local observation information to obtain spatial features; The initial action is obtained by concatenating the multi-source features and the spatial features and then inputting them into a multilayer perceptron.

7. The collaborative reinforcement learning control method for automotive painting tasks according to claim 1, characterized in that, The method for obtaining global value is as follows: Global state information is generated by stitching together all the aforementioned local observation information. Based on the concatenation of all the initial actions, a combined action is generated; Based on the global state information and the input of the joint action, the centralized intelligent network is used to obtain the global value, which is used to evaluate the merits of the current joint action.

8. A collaborative reinforcement learning control method for automotive painting tasks according to claim 7, characterized in that, The centralized intelligent network includes a multilayer sensor and an action-value function network; Based on the multilayer perceptron, the global state information and the features of the joint action are extracted, and the extracted features are input into the action-value function network to output the global value. The centralized intelligent network updates parameters by minimizing the time-series difference error.

9. A collaborative reinforcement learning control method for automotive painting tasks according to claim 1, characterized in that, Also includes: When the strategy network determines that the entire surface of the vehicle to be painted has reached the target thickness and uniformity standards, it automatically stops the painting process. Perform a full surface quality scan on the painted vehicle and generate a quality report.

10. A cooperative reinforcement learning control system for automotive painting tasks, used to execute a cooperative reinforcement learning control method for automotive painting tasks as described in any one of claims 1-9, characterized in that, include: Image acquisition and processing module, vehicle 3D reconstruction module, paint film data acquisition module, control command output module, initial action generation module, initial action evaluation module, and optimal action generation module; The image acquisition and processing module is used to acquire image data of the vehicle to be painted from multiple perspectives and preprocess it to obtain enhanced image data; The vehicle 3D reconstruction module is used to perform 3D reconstruction based on the enhanced image data to obtain a 3D mesh model of the vehicle to be painted. The paint film data acquisition module is used to collect paint film thickness data in real time during the spraying process and combine it with the three-dimensional mesh model to generate a dynamic paint film thickness map. The control command output module is used to transform the multi-arm cooperative spraying task into a multi-agent Markov decision model based on the three-dimensional mesh model and the dynamic paint film thickness map, and output control commands. The initial action generation module is used to execute the control instructions based on the policy network of all agents, obtain local observation information, and generate initial actions; The initial action evaluation module is used to obtain the global value by feeding all the local observation information and the initial action input into the centralized intelligent network. The optimal action generation module is used to update the parameters of the policy network based on the goal of maximizing the global value, guide the agent to generate the current optimal action, and execute the corresponding painting task.

Images