A method and device for aircraft component assembly coordination based on multi-agent reinforcement learning

Abstract

The present invention discloses a method for aircraft component assembly coordination based on multi-agent reinforcement learning, comprising: constructing an assembly accuracy mathematical model for characterizing manufacturing accuracy and coordination accuracy; constructing an aircraft structure multi-object assembly deformation proxy model; constructing a corresponding assembly deformation control model for each assembly object; constructing a multi-agent reinforcement learning assembly coordination model based on the assembly accuracy mathematical model and the aircraft structure multi-object assembly deformation proxy model, as well as the assembly deformation control model of each assembly object; obtaining the aircraft component assembly coordination model through distributed execution of centralized training; inputting the current assembly information of each assembly object in the aircraft assembly process into the optimal aircraft component assembly coordination model to output the assembly object process control parameters of each assembly object at the next moment. The present invention also provides an aircraft component assembly coordination device. The method provided by the present invention can optimize the assembly coordination process of large aircraft components.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft automated assembly, and in particular relates to an aircraft component assembly coordination method and device based on multi-agent reinforcement learning. Background Art

[0002] Aircraft are extremely complex mechanical products, constructed through multi-stage assembly. This paper focuses on the assembly coordination challenges between multiple large aircraft components. Aircraft assembly quality is primarily evaluated by key features distributed across the structure. Currently, most assembly coordination models focus solely on individual components and simply consider the three-dimensional deviations of discrete key reference points as optimization targets. However, in the actual assembly of multiple components, it is necessary to consider not only the deviations between individual assembly objects and the theoretical model, but also the coordination requirements between these objects. For example, when the left and right wing boxes are mounted on the fuselage, the symmetry of the wing boxes is a key indicator for evaluating assembly quality and has a significant impact on the aerodynamic performance of the entire product. Therefore, it is necessary to establish a new assembly coordination model that simultaneously considers the coordination requirements between different assembly objects and the geometric deviations of each assembly object. Furthermore, modern aircraft manufacturing widely utilizes carbon fiber-reinforced composite materials rather than traditional metal structures. Due to the layered manufacturing nature of composite structures, the bonding strength between layers is relatively weak. Therefore, during the manufacturing process, in addition to ensuring geometric accuracy, full attention must also be paid to the stress distribution of the structure.

[0003] At the same time, when establishing a coordination model between multiple assembly objects, existing methods generally treat the assembly objects as rigid bodies and do not consider the possible deformation of the aircraft structure during the assembly process. This not only severely limits the improvement of assembly accuracy, but the rigid body assumption actually limits the solution space of the assembly coordination optimization problem, making it difficult for the solution of the target optimization problem to converge to the global optimal point.

[0004] Patent document CN118607854A discloses an aircraft assembly line scheduling method based on multi-agent learning technology, which includes: collecting assembly process data of workstations in the aircraft assembly line and establishing a priority relationship matrix of the workstations; then initializing the assembly task sorting chromosome population of the genetic algorithm and initializing the relevant parameters of the genetic algorithm, wherein each chromosome in the initialized initial population corresponds to a feasible solution of the workstation assembly scheduling problem that satisfies the priority relationship constraint and is serially executed; initializing the assembly task allocation chromosome population based on the theoretical allocation quantity, the actual allocation number and the ideal number of allocation processes per time, and calculating the global fitness of the collaborative cooperation of all workstations when workers of the same type can share; retaining the optimal chromosome in the workstation population, and performing crossover, mutation and selection operations on the chromosome population to achieve aircraft assembly line scheduling.

[0005] Patent document CN118502364A discloses a large aircraft assembly production robot scheduling method based on deep reinforcement learning, which includes: S1: problem modeling uses an attention-based neural network to abstract heterogeneous AGVs into intelligent agents in the neural network, and concretizes specific task goals as colored coordinate points to achieve the description and modeling of the MRS task scheduling problem; S2: network construction builds the attention-based neural network in step S1 and determines the loss function during neural network training; S3: network training trains the neural network built in step S2 to obtain the trained MRS task scheduling model; S4: task scheduling uses the MRS task scheduling model in step S3 to complete the scheduling of MRS tasks. Summary of the Invention

[0006] The purpose of the present invention is to provide an aircraft component assembly coordination method and device based on multi-agent reinforcement learning. The method can generate control parameters in real time according to the actual status of each assembly object at the assembly site to optimize the assembly manufacturing accuracy and assembly coordination accuracy, and improve the assembly coordination process of large aircraft components.

[0007] To achieve the first objective of the present invention, the following technical solution is provided: a method for coordinating aircraft component assembly based on multi-agent reinforcement learning, comprising the following steps:

[0008] According to the assembly process requirements of multiple assembly objects of aircraft structures, an assembly accuracy mathematical model is constructed, which includes functions for characterizing manufacturing accuracy and coordination accuracy.

[0009] Based on the assembly accuracy control parameters and variation range of each digital tooling in the assembly scenario, a multi-object assembly deformation response dataset is obtained by combining sampling design and numerical simulation. The corresponding aircraft structure multi-object assembly deformation proxy model is trained based on the multi-object assembly deformation response dataset.

[0010] Using a single-agent reinforcement learning method, a corresponding assembly deformation control model is constructed for each assembly object. The assembly deformation control model includes a strategy network and weight parameters for controlling the displacement and stress field distribution of the assembly object.

[0011] Based on the assembly accuracy mathematical model, the aircraft structure multi-object assembly deformation agent model, and the assembly deformation control model of each assembly object, a multi-agent reinforcement learning assembly coordination model is constructed. The multi-agent reinforcement learning assembly coordination model uses the process control parameters of each assembly object as the action of the corresponding agent, the manufacturing accuracy corresponding to the assembly object and the coordination accuracy between the assembly objects as the state of the corresponding agent, and the optimization degree of the assembly accuracy as the reward function to measure whether the current strategy helps to improve the assembly accuracy of the multiple objects;

[0012] Combine the actions, states, and reward functions of all agents at the same moment into corresponding joint actions, joint states, and joint rewards;

[0013] Initializing the multi-agent reinforcement learning assembly coordination model with the strategy network weights of the assembly deformation control model corresponding to each assembly object;

[0014] Based on a centralized training distributed execution architecture, the multi-agent reinforcement learning assembly coordination model is trained by sharing the joint actions, joint states, and joint rewards of all agents to obtain an optimal aircraft component assembly coordination model;

[0015] The current assembly information of each assembly object in the aircraft assembly process is input into the optimal aircraft component assembly coordination model to output the process control parameters of each assembly object at the next moment.

[0016] The present invention combines sampling design with data simulation to construct a multi-object assembly deformation response dataset, and based on this dataset, constructs an aircraft structure multi-object assembly deformation proxy model; multiple intelligent agents continuously interact with the simulation environment created by the aircraft structure multi-object assembly deformation proxy model to collect training data to train the strategies of each intelligent agent and obtain the optimal aircraft component assembly coordination model.

[0017] Specifically, the manufacturing accuracy is the degree to which the geometric shape of the actual assembly object conforms to the nominal dimensions specified in its digital model, and its expression is as follows:

[0018]

[0019] Where L is the number of manufacturing features possessed by the i-th agent, w j is the relative weight of each manufacturing feature, T Ma ij is the actual measurement value of the jth manufacturing feature, N Ma ij is the nominal value of the jth manufacturing feature.

[0020] Specifically, the coordination accuracy is the actual geometric shapes and shape compatibility of the mating features between different assembly objects, and its expression is as follows:

[0021]

[0022] Where M is the number of features that the i-th agent has coordination requirements, V is the number of agents that have coordination requirements with the j-th feature of the i-th agent, T Ma ij is the actual measurement value of the jth manufacturing feature, T Ma kjis the actual measured value of the feature on the k-th agent that matches the feature j on the ith agent.

[0023] Specifically, during the training process, the multi-agent reinforcement learning assembly coordination model is initialized with the policy network weights of the assembly deformation control model corresponding to each assembly object; the initialization method alleviates the high variance and low efficiency of the multi-agent system caused by random exploration in the early stage of training by providing pre-trained weight parameters. In particular, the assembly deformation control based on a single agent has certain similarities with the assembly coordination based on a multi-agent. This pre-training can accelerate the agent to learn effective strategies and quickly improve performance.

[0024] Specifically, the proximal strategy is used to optimize the training of the strategy network and value network of the multi-agent.

[0025] When interacting with a multi-object assembly deformation proxy model, each agent makes independent decisions based on its own observed local state. During network update training, all agents share a centralized value function. By sharing the actions, states, and reward information of all agents, the value network is updated, and the global generalized advantage estimate of all agents is calculated based on the centralized value function to update the policy network.

[0026] Specifically, the loss function of the value network is calculated based on the target value function estimate and the current value function estimate. The formula for the target value function estimate is as follows:

[0027]

[0028] Here, λ is a parameter between 0 and 1. is the joint cumulative reward of each agent starting from time t and considering the next n steps, N s is the upper limit of the number of future steps to be considered.

[0029] Specifically, the joint state of N agents Input the value network to get the current value function estimate

[0030] The loss function of the value network is as follows:

[0031]

[0032] Where Q is the total number of samples.

[0033] Specifically, during the training process, the loss function of the policy network is expressed as follows:

[0034]

[0035] Among them, θi represents the current strategy parameters of the i-th agent, r t (θ i ) is the strategy probability ratio of the i-th agent; is the global advantage function estimate of all agents, used to evaluate the joint state s of all agents t Take joint action under t Advantage relative to the average joint action; clip(r t (θ i ),1-,1+) is a clipping function used to reduce the policy probability ratio r t (θ i ) is limited to [1-,1+]; ε is a hyperparameter used to control the degree of clipping.

[0036] In order to achieve the second purpose of the present invention, the following technical solution is provided: an aircraft component assembly coordination device, used to implement the steps of the above-mentioned aircraft component assembly coordination method based on multi-agent reinforcement learning.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] Based on the geometric shape evaluation standard, the assembly accuracy requirements are subdivided into manufacturing accuracy and coordination accuracy. A mathematical model is established to characterize manufacturing accuracy and coordination accuracy to comprehensively measure the assembly quality of multiple objects.

[0039] An assembly coordination model is constructed based on a multi-object deformable proxy model and multi-agent deep reinforcement learning. A multi-agent deep reinforcement learning simulation training environment is built based on the multi-object proxy model, so that each agent can quickly learn the multi-object assembly coordination optimization strategy.

[0040] Initialize the multi-agent strategy with a single-agent deformation control strategy to alleviate the high variance and low efficiency caused by random exploration in the early stages of multi-agent system training, and further accelerate the learning of effective strategies by the agents;

[0041] The reward function is constructed based on the mathematical model of assembly accuracy. The reward function of each agent consists of a local reward function and a global reward function. The local reward function reflects the manufacturing accuracy and stress distribution of a single object, and the global reward function reflects the coordination accuracy requirements between objects. By sharing rewards, actions, and status information during training, the multi-agent training process is accelerated and optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the wall panel splicing station provided for this embodiment;

[0043] Figure 2A flowchart of the aircraft component assembly coordination method based on multi-agent reinforcement learning provided in this embodiment;

[0044] Figure 3 The training process of the multi-agent reinforcement learning assembly coordination model provided in this embodiment;

[0045] Figure 4 This embodiment provides a flowchart of randomly taking a group of experiences in the experience buffer as samples to train the assembly coordination model. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, this embodiment is illustrated by an application example of a fuselage panel splicing station, that is, the assembly objects are four single-curved panels, and each assembly pair is discretely positioned by a CNC positioning device, and the control parameters are the spatial coordinates of each positioning point.

[0048] like Figure 2 As shown, this embodiment provides an aircraft component assembly coordination method based on multi-agent reinforcement learning. First, machine learning technology is used to construct a deformable proxy model of each assembly object. Based on the multi-object proxy model, a multi-agent deep reinforcement learning simulation training environment is constructed. The state of each agent is composed of its manufacturing accuracy, stress distribution, and coordination accuracy. In this embodiment, the manufacturing accuracy is the manufacturing accuracy of the shape of each panel, and the coordination accuracy is the fitting clearance between each panel. The reward function of each agent is composed of a local reward function and a global reward function. The local reward function reflects the optimization of the manufacturing accuracy and stress distribution of a single object, and the global reward function reflects the optimization of the coordination accuracy between objects. By sharing rewards, actions, and state information during the training process, the multi-agent training process is accelerated and optimized.

[0049] The specific process is as follows:

[0050] S1. According to the multi-object assembly process requirements of aircraft structures, the assembly accuracy requirements are subdivided into manufacturing accuracy and coordination accuracy based on the geometric shape evaluation standard, and an assembly accuracy mathematical model that characterizes manufacturing accuracy and coordination accuracy is established.

[0051] Among them, manufacturing accuracy refers to the degree to which the geometric shape of the actual assembly object conforms to the nominal dimensions specified in its digital model; coordination accuracy refers to the actual geometric shapes and shape compatibility of the mating features between different assembly objects. By analyzing the manufacturing accuracy requirements of the key features of each assembly object and the mating relationship between multiple assembly objects, a mathematical model of multi-object assembly accuracy is established.

[0052] S2. Based on the assembly accuracy control parameters and variation ranges of each digital tooling in the assembly scenario, a multi-object assembly deformation response dataset is obtained by combining sampling design and numerical simulation. A corresponding aircraft structure multi-object assembly deformation proxy model is obtained by training based on the multi-object assembly deformation response dataset.

[0053] Among them, the deformation control parameters and their ranges of each assembly object are determined according to the digital tooling used by each assembly object, and the control parameters are set as the boundary conditions of the simulation model of each assembly object. By sampling in the control parameter space and combining parametric simulation modeling technology, a deformation simulation dataset for each assembly object is constructed; based on the deformation simulation dataset, a multi-object assembly deformation proxy model of the aircraft structure is established, which represents the relationship between the deformation control parameters of each object and the manufacturing accuracy.

[0054] S3. Using a single-agent reinforcement learning method, a corresponding assembly deformation control model is constructed for each assembly object. The assembly deformation control model includes a strategy network and weight parameters for controlling the displacement field and stress field distribution of the assembly object.

[0055] Based on the mathematical model of assembly accuracy, the aircraft structure multi-object assembly deformation agent model, and the assembly deformation control model of each assembly object, a multi-agent reinforcement learning assembly coordination model is constructed. The multi-agent reinforcement learning assembly coordination model uses the process control parameters of each assembly object as the action of the corresponding agent, the manufacturing accuracy of the assembly object and the coordination accuracy of the remaining assembly objects as the state of the corresponding agent, and the optimization degree of assembly accuracy as the reward function to measure whether the current strategy helps improve the assembly accuracy of multiple objects.

[0056] Combine the actions, states, and reward functions of all agents at the same moment into corresponding joint actions, joint states, and joint rewards.

[0057] Initializing the multi-agent reinforcement learning assembly coordination model with the strategy network weights of the assembly deformation control model corresponding to each assembly object;

[0058] Based on a centralized training distributed execution architecture, the multi-agent reinforcement learning assembly coordination model is trained by sharing the joint actions, joint states, and joint rewards of all agents to obtain the optimal aircraft component assembly coordination model.

[0059] like Figure 3 As shown, Figure 3 This is only an execution order provided for this embodiment and does not represent the only execution order of the aircraft component assembly coordination method based on multi-agent reinforcement learning. If the final result can be achieved, the steps can be executed in parallel or in reverse.

[0060] Step 301: Construct an assembly deformation control model based on single-agent reinforcement learning for each assembly object, and establish a reinforcement learning simulation training environment based on the deformation response agent model of each assembly object. The strategy network and its weight for controlling the deformation of each assembly object are obtained by training in the simulation environment.

[0061] Step 302: Based on the manufacturing accuracy and coordination accuracy and the aircraft structure multi-object agent model, an aircraft component assembly coordination simulation environment based on a multi-agent reinforcement learning algorithm is constructed, and an experience buffer area is set for each agent.

[0062] Step 303: Initialize the policy network of the assembly coordination model based on multi-agent reinforcement learning with the policy network weights of the single-agent deformation control model of each assembly object.

[0063] The above-mentioned initialization method alleviates the high variance and low efficiency caused by random exploration in the early stages of training of the multi-agent system by providing pre-trained weight parameters. In particular, the assembly deformation control based on a single agent has certain similarities with the assembly coordination based on a multi-agent. This pre-training can accelerate the learning of effective strategies by the agent and quickly improve performance.

[0064] Step 304: Setting the current state and execution action for the N agents, and setting the reward function for the N agents;

[0065] The action of the intelligent agent is the process control parameter of each assembly object. The specific formula is as follows:

[0066] a t =[a 0t ,a 1t ,…a it …a Nt ],0≤i≤N

[0067] Among them, a t is the joint action of N agents at time t, a itis the action taken by the i-th agent at time t, N is the number of agents, that is, the number of assembly objects; similarly, the joint state and joint reward of each agent can be expressed as:

[0068] s t =[s 0t ,s 1t ,…s it …s Nt ],0≤i≤N

[0069] r t =[r 0t ,r 1t ,…r it …r Nt ],0≤i≤N

[0070] Among them, s t is the joint state of all agents at time t, r t is the joint state of all agents at time t.

[0071] The reward function is designed based on the degree of optimization of assembly accuracy, measuring whether the current strategy helps improve the assembly accuracy of multiple assembly objects. The local reward function of each agent in the multi-agent assembly coordination model is determined by the improvement of its manufacturing accuracy, and the formula is as follows:

[0072] L r it =f M (ε i(t+1) )-f M (ε it )

[0073] Among them, f M The evaluation function for measuring assembly accuracy is as follows:

[0074]

[0075] The global reward function of each agent in the multi-agent assembly coordination model is determined by the improvement of its coordination accuracy, and the formula is as follows:

[0076] G r it =f M (γ i(t+1) )-f M (γ it )

[0077] The reward value of the i-th agent at time t is determined based on the local reward function and the global reward function. The formula is as follows:

[0078] r it =L r it + G r it

[0079] Step 305: obtaining current observation values ​​of the N agents observing the assembly coordination simulation environment based on the multi-object proxy model, and inputting the observation values ​​into the current strategy network of each agent to obtain the next action to be taken by each agent;

[0080] The state of the agent in the multi-agent deformation control model is composed of the manufacturing accuracy of each assembly object and the coordination accuracy with other objects, wherein the formula for the manufacturing accuracy of the i-th agent is as follows:

[0081]

[0082] Where L is the number of manufacturing features possessed by the i-th agent, w j is the relative weight of each manufacturing feature, T Ma ij is the actual measurement value of the jth manufacturing feature, N Ma ij is the nominal value of the jth manufacturing feature;

[0083] The formula for the coordination accuracy of the i-th agent is as follows:

[0084]

[0085] Where M is the number of features that the i-th agent has coordination requirements, V is the number of agents that have coordination requirements with the j-th feature of the i-th agent, T Ma ij is the actual measurement value of the jth manufacturing feature, T Ma kj is the actual measurement value of the feature on the k-th agent that matches the feature j on the ith agent;

[0086] According to the manufacturing accuracy and the coordination accuracy, the state of the i-th agent at time t is determined as follows:

[0087] s it =concat(ε it ,γ it )

[0088] Step 306: Executing the joint action in the multi-object assembly coordination simulation environment to obtain rewards for the N agents and observation values ​​at the next moment, and storing the interaction experiences of the N agents and the environment in the N experience buffers; wherein the interaction experiences include the current observation value, the action, the reward value, and the observation value at the next moment;

[0089] Step 307: Determine whether the size of each experience buffer reaches a preset value;

[0090] If the size of each experience buffer area does not reach the preset value, continue to execute step 305 to step 306, otherwise, execute step 308;

[0091] Step 308: Randomly sample a group of the interaction experiences in the experience buffer as samples to train the multi-agent reinforcement learning assembly coordination model.

[0092] like Figure 4 As shown in FIG, a process of randomly taking a set of experiences from the experience buffer as samples to train the assembly coordination model, that is, using proximal policy optimization to update the policy network and value network in the reinforcement learning model, the specific steps are:

[0093] Step 401: Randomly sample a set of interaction experiences from the experience buffer, and determine a target value function estimate based on the sampled interaction experiences using the assembly coordination model; wherein the formula for the interaction experience is as follows:

[0094]

[0095] Where T is the maximum number of time steps that each agent can take in one round;

[0096] The formula for estimating the target value function is as follows:

[0097]

[0098] Here, λ is a parameter between 0 and 1. is the joint cumulative reward of each agent starting from time t and considering the next n steps, N s To consider the upper limit of the number of future steps, the formula for the joint cumulative reward is as follows:

[0099]

[0100] Among them, T n is the time horizon starting from time t and looking forward n steps, γ is a discount factor used to reduce dependence on long-term rewards, is the joint reward received by all agents at time t′.

[0101] Step 402: The joint state of the N agents Input the value network to get the current value function estimate

[0102] Step 403: Determine the loss value of the value network based on the target value function estimate and the current value function estimate; wherein the formula of the loss function of the value network is as follows:

[0103]

[0104] Among them, Q is the total number of samples, which is used to normalize the loss function;

[0105] Step 404: Update the parameters of the value network according to the loss value.

[0106] Step 405: Randomly sample a set of interaction experiences from the experience buffer of each agent, and determine the loss value of each agent's policy network using the deformation control model based on the sampled interaction experiences; wherein the formula of the loss function of the i-th agent's policy network is as follows:

[0107]

[0108] Among them, θ i Represents the current strategy parameters of the i-th agent. t (θ i ) is the strategy probability ratio of the i-th agent That is, the new strategy π θ (a it |s it ) Compared to the old strategy The probability ratio. is the global advantage function estimate of all agents, used to evaluate the joint state s of all agents t Take joint action under t Advantage relative to the average joint action. clip(r t (θ i ),1-,1+) is a clipping function that reduces the policy probability ratio r t (θ i ) is limited to [1-,1+] to avoid excessive changes in the policy update step, thereby maintaining the stability of the learning process. ε is a hyperparameter used to control the degree of clipping and is usually taken as a small value. The formula for estimating the global advantage function of the agent using generalized advantage is as follows:

[0109]

[0110] Among them, δt =r t +γV(s t+1 )-V(s t ), r t is the joint reward of all agents at time t, V(s i ) is the value function estimate, γ is the discount factor, and λ is the trade-off factor used to weigh the bias and variance of the estimate.

[0111] Step 406: Update the policy network according to the loss value of each agent strategy.

[0112] Step 407: Determine whether the number of iterations has reached the preset training number. Specifically, determine whether the number of iterations of the multi-agent assembly coordination model has reached the preset training number. If the number of iterations of the multi-agent assembly coordination model has not reached the preset training number, continue iteratively executing steps 401 to 407, continuously iteratively updating the centralized value network and each agent's policy network. Otherwise, execute step 406.

[0113] Step 408: Iteratively execute the updating step until the number of iterations of the multi-agent assembly coordination model reaches a preset number of training times and the experience buffer is cleared.

[0114] S5. Input the current assembly information of each assembly object in the aircraft assembly process into the optimal aircraft component assembly coordination model to output the assembly object process control parameters of each assembly object at the next moment, thereby realizing real-time assembly coordination and optimization of aircraft components.

[0115] This embodiment also provides an aircraft component assembly coordination device, which is used to implement the steps of the aircraft component assembly coordination method based on multi-agent reinforcement learning provided in the above embodiment.

[0116] In addition, the terms "upper", "lower", "inner", "outer", "front", and "back" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0117] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. Any equivalent changes or modifications made based on the structure, features and principles described in the scope of the patent application of the present invention should be included in the scope of the patent application of the present invention.

[0118] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for aircraft component assembly coordination based on multi-agent reinforcement learning, characterized in that: The following steps are involved: According to the assembly process requirements of multiple assembly objects of aircraft structures, an assembly accuracy mathematical model is constructed, which includes functions for characterizing manufacturing accuracy and coordination accuracy. Based on the assembly accuracy control parameters and variation range of each digital tooling in the assembly scenario, a multi-object assembly deformation response dataset is obtained by combining sampling design and numerical simulation. The corresponding aircraft structure multi-object assembly deformation proxy model is trained based on the multi-object assembly deformation response dataset. Using a single-agent reinforcement learning method, a corresponding assembly deformation control model is constructed for each assembly object. The assembly deformation control model includes a strategy network and weight parameters for controlling the displacement and stress field distribution of the assembly object. Based on the mathematical model of assembly accuracy, the aircraft structure multi-object assembly deformation agent model, and the assembly deformation control model of each assembly object, a multi-agent reinforcement learning assembly coordination model is constructed. The multi-agent reinforcement learning assembly coordination model uses the process control parameters of each assembly object as the action of the corresponding agent, the manufacturing accuracy of the assembly object and the coordination accuracy between the assembly objects as the state of the corresponding agent, and the optimization degree of the assembly accuracy as the reward function to measure whether the current strategy helps improve the assembly accuracy of multiple objects. Combine the actions, states, and reward functions of all agents at the same moment into corresponding joint actions, joint states, and joint rewards; Initializing the multi-agent reinforcement learning assembly coordination model with the strategy network weights of the assembly deformation control model corresponding to each assembly object; Based on a centralized training distributed execution architecture, the multi-agent reinforcement learning assembly coordination model is trained by sharing the joint actions, joint states, and joint rewards of all agents to obtain an optimal aircraft component assembly coordination model; The current assembly information of each assembly object in the aircraft assembly process is input into the optimal aircraft component assembly coordination model to output the process control parameters of each assembly object at the next moment.

2. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The manufacturing accuracy is the degree to which the geometric shape of the actual assembly object conforms to the nominal dimensions specified in its digital model, and its expression is as follows: Where L is the number of manufacturing features possessed by the i-th agent, w j is the relative weight of each manufacturing feature, T Ma ij is the actual measurement value of the jth manufacturing feature, N Ma ij is the nominal value of the jth manufacturing feature.

3. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The coordination accuracy is the actual geometric shape and shape compatibility of the mating features between different assembly objects, and its expression is as follows: Where M is the number of features that the i-th agent has coordination requirements, V is the number of agents that have coordination requirements with the j-th feature of the i-th agent, T Ma ij is the actual measurement value of the jth manufacturing feature, T Ma kj is the actual measured value of the feature on the k-th agent that matches the feature j on the ith agent.

4. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 1 is characterized in that: During the training process, the multi-agent reinforcement learning assembly coordination model is initialized with the policy network weights of the assembly deformation control model corresponding to each assembly object; The initialization method alleviates the high variance and low efficiency of the multi-agent system caused by random exploration in the early stage of training by providing pre-trained weight parameters.

5. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 1, characterized in that: The policy network and value network of the multi-agent are trained using proximal policy optimization. When interacting with a multi-object assembly deformation proxy model, each agent makes independent decisions based on its own observed local state. During network update training, all agents share a centralized value function. The value network is updated by sharing the action, state, and reward information of all agents. The global generalized advantage estimate of all agents is calculated based on the centralized value function to update the policy network.

6. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 5 is characterized in that: The loss function of the value network is calculated based on the target value function estimate and the current value function estimate. The formula for the target value function estimate is as follows: Here, λ is a parameter between 0 and 1. is the joint cumulative reward of each agent starting from time t and considering the next n steps, N s To consider the upper limit of the number of future steps; The joint state of N agents Input the value network to get the current value function estimate The loss function of the value network is as follows: Where Q is the total number of samples.

7. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 5, characterized in that: During the training process, the loss function of the policy network is expressed as follows: Among them, θ i represents the current strategy parameters of the i-th agent, r t (θ i ) is the strategy probability ratio of the i-th agent; is the global advantage function estimate of all agents, used to evaluate the joint state s of all agents t Take joint action under t Advantage relative to the average joint action; clip(r t (θ i ),1-∈,1+∈) is a clipping function used to reduce the policy probability ratio r t (θ i ) is restricted to [1-∈, 1+∈]; ε is a hyperparameter used to control the degree of clipping.

8. An aircraft component assembly coordination device, characterized in that: Steps for implementing the aircraft component assembly coordination method based on multi-agent reinforcement learning as described in any one of claims 1 to 7.

Images