Aircraft component assembly coordination method and device based on multi-agent reinforcement learning

Abstract

The invention discloses an aircraft component assembly coordination method based on multi-agent reinforcement learning. The method comprises the following steps: constructing an assembly accuracy mathematical model for representing manufacturing accuracy and coordination accuracy; constructing an aircraft structure multi-object assembly deformation agent model; constructing a corresponding assembly deformation control model for each assembly object; based on the assembly accuracy mathematical model, the aircraft structure multi-object assembly deformation agent model and the assembly deformation control model of each assembly object, constructing a multi-agent reinforcement learning assembly coordination model; performing centralized training in a distributed manner to obtain an aircraft component assembly coordination model; and inputting the current assembly information of each assembly object in the aircraft assembly process into the optimal aircraft component assembly coordination model so as to output the assembly object process control parameters of each assembly object at the next moment. The invention further provides an aircraft component assembly coordination device. The method provided by the invention can optimize the assembly coordination process of the large aircraft component.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft automated assembly, and in particular relates to an aircraft component assembly coordination method and device based on multi-agent reinforcement learning. Background Art

[0002] Aircraft is an extremely complex mechanical product, which is formed after multi-stage assembly. The present invention focuses on the assembly coordination problem between multiple large aircraft components. The assembly quality of an aircraft is mainly evaluated by key features distributed on the structure. At present, most assembly coordination models only involve a single component and simply regard the three-dimensional deviation of discrete key reference points as the optimization target. However, in the actual assembly process of multiple components, it is necessary to consider not only the deviation of each assembly object from the theoretical model, but also the coordination requirements between different assembly objects. For example, when the left and right wing boxes are installed on the fuselage, the symmetry of the wing boxes is a key indicator for evaluating the assembly quality and has an important impact on the aerodynamic characteristics of the entire product. Therefore, it is necessary to establish a new assembly coordination model that takes into account the coordination requirements between different assembly objects and the geometric shape deviations of each assembly object. In addition, modern aircraft widely use carbon fiber reinforced composite materials instead of traditional metal structures in the manufacturing process. Due to the laminated manufacturing characteristics of composite structures, the bonding strength between layers is weak. Therefore, in the manufacturing process, in addition to ensuring geometric accuracy, full attention must be paid to the stress distribution of the structure.

[0003] At the same time, when establishing a coordination model between multiple assembly objects, existing methods generally regard the assembly objects as rigid bodies and do not consider the possible deformation of the aircraft structure during the assembly process. This not only seriously limits the improvement of assembly accuracy, but the rigid body assumption actually limits the solution space of the assembly coordination optimization problem, making it difficult for the solution of the target optimization problem to converge to the global optimal point.

[0004] Patent document CN118607854A discloses an aircraft assembly line scheduling method based on multi-agent learning technology, which includes: collecting assembly process data of workstations in the aircraft assembly line and establishing a priority relationship matrix of the workstations; then initializing the assembly task sorting chromosome population of the genetic algorithm and initializing the relevant parameters of the genetic algorithm, each chromosome in the initial population obtained by initialization corresponds to a feasible solution of serial execution of the workstation assembly scheduling problem that satisfies the priority relationship constraint; initializing the assembly task allocation chromosome population according to the theoretical allocation quantity, the actual allocation number and the ideal number of each allocation process, and calculating the global fitness of the collaborative cooperation of all workstations when workers of the same type can share; retaining the optimal chromosome in the workstation population, and performing crossover, mutation and selection operations on the chromosome population to realize aircraft assembly line scheduling.

[0005] Patent document CN118502364A discloses a large aircraft assembly production robot scheduling method based on deep reinforcement learning, which includes: S1: problem modeling uses an attention-based neural network to abstract heterogeneous AGVs into intelligent agents in the neural network, and concretizes specific task goals as colored coordinate points to achieve the description and modeling of the MRS task scheduling problem; S2: network construction builds the attention-based neural network in step S1, and determines the loss function during neural network training; S3: network training trains the neural network built in step S2 to obtain a trained MRS task scheduling model; S4: task scheduling uses the MRS task scheduling model in step S3 to complete the scheduling of MRS tasks. Summary of the invention

[0006] The purpose of the present invention is to provide an aircraft component assembly coordination method and device based on multi-agent reinforcement learning. The method can generate control parameters in real time according to the actual status of each assembly object at the assembly site to optimize the assembly manufacturing accuracy and assembly coordination accuracy, and improve the assembly coordination process of large aircraft components.

[0007] In order to achieve the first object of the present invention, the following technical solution is provided: an aircraft component assembly coordination method based on multi-agent reinforcement learning, comprising the following steps:

[0008] According to the assembly process requirements of multiple assembly objects of the aircraft structure, an assembly accuracy mathematical model including the one used to characterize the manufacturing accuracy and coordination accuracy is constructed;

[0009] According to the assembly accuracy control parameters and variation range of each digital tooling in the assembly scene, a multi-object assembly deformation response data set is obtained by combining sampling design and numerical simulation, and a corresponding aircraft structure multi-object assembly deformation proxy model is obtained by training based on the multi-object assembly deformation response data set;

[0010] Using a single-agent reinforcement learning method, a corresponding assembly deformation control model is constructed for each assembly object, wherein the assembly deformation control model includes a strategy network and weight parameters for controlling the displacement field and stress field distribution of the assembly object;

[0011] Based on the assembly accuracy mathematical model and the aircraft structure multi-object assembly deformation agent model, as well as the assembly deformation control model of each assembly object, a multi-agent reinforcement learning assembly coordination model is constructed. The multi-agent reinforcement learning assembly coordination model uses the process control parameters of each assembly object as the action of the corresponding agent, the manufacturing accuracy corresponding to the assembly object and the coordination accuracy between the assembly objects as the state of the corresponding agent, and the optimization degree of the assembly accuracy as the reward function to measure whether the current strategy is helpful to improve the assembly accuracy of multiple objects;

[0012] Combine the actions, states, and reward functions of all agents at the same time into corresponding joint actions, joint states, and joint rewards;

[0013] Initializing the multi-agent reinforcement learning assembly coordination model with the strategy network weights of the assembly deformation control model corresponding to each assembly object;

[0014] Based on a centralized training distributed execution architecture, the multi-agent reinforcement learning assembly coordination model is trained by sharing joint actions, joint states, and joint rewards of all agents to obtain an optimal aircraft component assembly coordination model;

[0015] The current assembly information of each assembly object in the aircraft assembly process is input into the optimal aircraft component assembly coordination model to output the process control parameters of each assembly object at the next moment.

[0016] The present invention combines sampling design with data simulation to construct a multi-object assembly deformation response data set, and constructs an aircraft structure multi-object assembly deformation proxy model based on the data set; multiple agents continuously interact with the simulation environment created by the aircraft structure multi-object assembly deformation proxy model to collect training data to train each agent strategy and obtain the optimal aircraft component assembly coordination model.

[0017] Specifically, the manufacturing accuracy is the degree to which the geometric shape of the actual assembly object matches the nominal size specified in its digital model, and its expression is as follows:

[0018]

[0019] Where L is the number of manufacturing features possessed by the ith agent, w j is the relative weight of each manufacturing feature, T Ma ij is the actual measurement value of the jth manufacturing feature, N Ma ij is the nominal value of the jth manufacturing feature.

[0020] Specifically, the coordination accuracy is the actual geometric shapes and shape compatibility of the matching features between different assembly objects, and its expression is as follows:

[0021]

[0022] Where M is the number of features that the ith agent has coordination requirements, V is the number of agents that have coordination requirements with the jth feature on the ith agent, T Ma ij is the actual measurement value of the jth manufacturing feature, T Ma kjis the actual measured value of the feature on the k-th agent that matches feature j on the ith agent.

[0023] Specifically, during the training process, the multi-agent reinforcement learning assembly coordination model is initialized with the policy network weights of the assembly deformation control model corresponding to each assembly object; the initialization method alleviates the high variance and low efficiency of the multi-agent system caused by random exploration in the early stage of training by providing pre-trained weight parameters, especially since the assembly deformation control based on a single agent and the assembly coordination based on multi-agents have certain similarities, this pre-training can accelerate the agent to learn effective strategies and quickly improve performance.

[0024] Specifically, the proximal strategy is used to optimize the training of the strategy network and value network of the multi-agent.

[0025] When interacting with the multi-object assembly deformation proxy model, each agent makes independent decisions based on its own observed local state. When the network is updated and trained, all agents share a centralized value function. By sharing the action, state and reward information of all agents, the value network is updated, and the global generalized advantage estimate of all agents is calculated based on the centralized value function to update the policy network.

[0026] Specifically, the loss function of the value network is calculated based on the target value function estimate and the current value function estimate. The formula for the target value function estimate is as follows:

[0027]

[0028] Here, λ is a parameter between 0 and 1. is the joint cumulative reward of each agent starting from time t and considering the next n steps, N s is the upper limit of the number of future steps to consider.

[0029] Specifically, the joint state of N agents Input the value network to get the current value function estimate

[0030] The loss function of the value network is as follows:

[0031]

[0032] Among them, Q is the total number of samples.

[0033] Specifically, during the training process, the loss function expression of the strategy network is as follows:

[0034]

[0035] Among them, θi represents the current strategy parameter of the i-th agent, r t (θ i ) is the strategy probability ratio of the ith agent; is the global advantage function estimate of all agents, used to evaluate all agents in the joint state s t Take joint action t Advantage relative to the average joint action; clip(r t (θ i ),1-,1+) is a clipping function used to reduce the policy probability ratio r t (θ i ) is limited to [1-,1+]; ε is a hyperparameter used to control the degree of clipping.

[0036] In order to achieve the second objective of the present invention, the following technical solution is provided: an aircraft component assembly coordination device, used to implement the steps of the above-mentioned aircraft component assembly coordination method based on multi-agent reinforcement learning.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] Based on the geometric shape evaluation standard, the assembly accuracy requirements are subdivided into manufacturing accuracy and coordination accuracy, and a mathematical model for characterizing manufacturing accuracy and coordination accuracy is established to comprehensively measure the assembly quality of multiple objects.

[0039] An assembly coordination model is constructed based on a multi-object deformable proxy model and multi-agent deep reinforcement learning. A multi-agent deep reinforcement learning simulation training environment is constructed based on the multi-object proxy model. Each agent can quickly learn the multi-object assembly coordination optimization strategy.

[0040] Initialize the multi-agent strategy with the single-agent deformation control strategy to alleviate the high variance and low efficiency caused by random exploration in the early stage of multi-agent system training, and further accelerate the learning of effective strategies by the agents;

[0041] The reward function is constructed based on the mathematical model of assembly accuracy. The reward function of each agent is composed of a local reward function and a global reward function. The local reward function reflects the manufacturing accuracy and stress distribution of a single object, and the global reward function reflects the coordination accuracy requirements between objects. By sharing rewards, actions, and status information during training, the multi-agent training process can be accelerated and optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the wall panel splicing station provided for this embodiment;

[0043] Figure 2A flowchart of an aircraft component assembly coordination method based on multi-agent reinforcement learning provided in this embodiment;

[0044] Figure 3 The training process of the multi-agent reinforcement learning assembly coordination model provided in this embodiment;

[0045] Figure 4 A flowchart of randomly taking a group of experiences in the experience buffer as samples for training the assembly coordination model is provided in this embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, this embodiment is explained by an application example of a fuselage panel splicing station, that is, the assembly objects are four single-curved panels, and each assembly object is discretely positioned by a CNC positioning device, and the control parameters are the spatial coordinates of each positioning point.

[0048] like Figure 2 As shown, a method for aircraft component assembly coordination based on multi-agent reinforcement learning provided by this embodiment first uses machine learning technology to construct a deformation proxy model of each assembly object, and then builds a multi-agent deep reinforcement learning simulation training environment based on the multi-object proxy model. The state of each agent is composed of its manufacturing accuracy, stress distribution and coordination accuracy. In this embodiment, the manufacturing accuracy is the manufacturing accuracy of the shape of each wall panel, and the coordination accuracy is the matching gap between each wall panel. The reward function of each agent is composed of a local reward function and a global reward function. The local reward function reflects the optimization of the manufacturing accuracy and stress distribution of a single object, and the global reward function reflects the optimization of the coordination accuracy between objects. By sharing rewards, actions, and state information during the training process, the multi-agent training process is accelerated and optimized.

[0049] The specific process is as follows:

[0050] S1. According to the requirements of multi-object assembly process of aircraft structure, the assembly accuracy requirements are subdivided into manufacturing accuracy and coordination accuracy based on geometric shape evaluation standards, and an assembly accuracy mathematical model that characterizes manufacturing accuracy and coordination accuracy is established.

[0051] Among them, manufacturing accuracy refers to the degree to which the geometric shape of the actual assembly object conforms to the nominal size specified in its digital model; coordination accuracy refers to the actual geometric shape of the matching features between different assembly objects and the degree of shape compatibility. By analyzing the manufacturing accuracy requirements of the key features of each assembly object and the matching relationship between multiple assembly objects, a mathematical model of multi-object assembly accuracy is established.

[0052] S2. According to the assembly accuracy control parameters and variation range of each digital tooling in the assembly scene, a multi-object assembly deformation response data set is obtained by combining sampling design and numerical simulation, and a corresponding aircraft structure multi-object assembly deformation proxy model is obtained by training based on the multi-object assembly deformation response data set.

[0053] Among them, the deformation control parameters and their ranges of each assembly object are determined according to the digital tooling used by each assembly object, and the control parameters are set as the boundary conditions of the simulation model of each assembly object. By sampling in the control parameter space and combining the parametric simulation modeling technology, a deformation simulation data set for each assembly object is constructed; based on the deformation simulation data set, a multi-object assembly deformation proxy model of the aircraft structure is established between the deformation control parameters of each object and the manufacturing accuracy.

[0054] S3. Using a single-agent reinforcement learning method, a corresponding assembly deformation control model is constructed for each assembly object. The assembly deformation control model includes a strategy network and weight parameters for controlling the displacement field and stress field distribution of the assembly object.

[0055] Based on the assembly accuracy mathematical model, the aircraft structure multi-object assembly deformation agent model, and the assembly deformation control model of each assembly object, a multi-agent reinforcement learning assembly coordination model is constructed. The multi-agent reinforcement learning assembly coordination model uses the process control parameters of each assembly object as the action of the corresponding agent, the manufacturing accuracy corresponding to the assembly object and the coordination accuracy of the remaining assembly objects as the state of the corresponding agent, and the optimization degree of the assembly accuracy as the reward function to measure whether the current strategy is helpful to improve the assembly accuracy of multiple objects;

[0056] Combine the actions, states, and reward functions of all agents at the same moment into corresponding joint actions, joint states, and joint rewards.

[0057] Initializing the multi-agent reinforcement learning assembly coordination model with the strategy network weights of the assembly deformation control model corresponding to each assembly object;

[0058] Based on a centralized training distributed execution architecture, the multi-agent reinforcement learning assembly coordination model is trained by sharing the joint actions, joint states and joint rewards of all agents to obtain the optimal aircraft component assembly coordination model.

[0059] like Figure 3 As shown, Figure 3 This is only an execution order provided for this embodiment and does not represent the only execution order of the aircraft component assembly coordination method based on multi-agent reinforcement learning. Its steps can be executed in parallel or in reverse if the final result can be achieved.

[0060] Step 301: construct an assembly deformation control model based on single-agent reinforcement learning for each assembly object, and establish a reinforcement learning simulation training environment based on the deformation response agent model of each assembly object, and obtain the strategy network and its weight for controlling the deformation of each assembly object through training in the simulation environment.

[0061] Step 302: Based on the manufacturing accuracy and coordination accuracy and the aircraft structure multi-object proxy model, an aircraft component assembly coordination simulation environment based on a multi-agent reinforcement learning algorithm is constructed, and an experience buffer area is set for each agent.

[0062] Step 303: Initialize the policy network of the assembly coordination model based on multi-agent reinforcement learning with the policy network weights of the single-agent deformation control model of each assembly object.

[0063] The above-mentioned initialization method alleviates the high variance and low efficiency caused by random exploration in the early stage of training of the multi-agent system by providing pre-trained weight parameters. In particular, the assembly deformation control based on a single agent has certain similarities with the assembly coordination based on multi-agents. This pre-training can accelerate the learning of effective strategies by the agent and quickly improve performance.

[0064] Step 304: setting the current states and execution actions for the N agents, and setting the reward functions for the N agents;

[0065] The action of the intelligent agent is the process control parameter of each assembly object. The specific formula is as follows:

[0066] a t =[a 0t ,a 1t ,…a it …a Nt ],0≤i≤N

[0067] Among them, a t is the joint action of N agents at time t, a itis the action taken by the ith agent at time t, N is the number of agents, that is, the number of assembly objects; similarly, the joint state and joint reward of each agent can be expressed as:

[0068] s t =[s 0t ,s 1t ,…s it …s Nt ],0≤i≤N

[0069] r t =[r 0t ,r 1t ,…r it …r Nt ],0≤i≤N

[0070] Among them, s t is the joint state of all agents at time t, r t is the joint state of all agents at time t.

[0071] The reward function is designed according to the optimization degree of assembly accuracy, and measures whether the current strategy helps to improve the assembly accuracy of multiple assembly objects. The local reward function of each agent in the multi-agent assembly coordination model is determined by the improvement of its manufacturing accuracy, and the formula is as follows:

[0072] L r it =f M (ε i(t+1) )-f M (ε it )

[0073] Among them, f M The evaluation function for measuring assembly accuracy is as follows:

[0074]

[0075] The global reward function of each agent in the multi-agent assembly coordination model is determined by the improvement of its coordination accuracy, and the formula is as follows:

[0076] G r it =f M (γ i(t+1) )-f M (γ it )

[0077] The reward value of the i-th agent at time t is determined according to the local reward function and the global reward function. The formula is as follows:

[0078] r it =L r it + G r it

[0079] Step 305: obtaining current observation values ​​of the N agents observing the assembly coordination simulation environment based on the multi-object proxy model, and inputting the observation values ​​into the current strategy network of each agent to obtain the next action to be taken by each agent;

[0080] The state of the agent in the multi-agent deformation control model is composed of the manufacturing accuracy of each assembly object and the coordination accuracy with other objects, wherein the formula for the manufacturing accuracy of the i-th agent is as follows:

[0081]

[0082] Where L is the number of manufacturing features possessed by the ith agent, w j is the relative weight of each manufacturing feature, T Ma ij is the actual measurement value of the jth manufacturing feature, N Ma ij is the nominal value of the jth manufacturing feature;

[0083] The formula for the coordination accuracy of the i-th agent is as follows:

[0084]

[0085] Where M is the number of features that the ith agent has coordination requirements, V is the number of agents that have coordination requirements with the jth feature on the ith agent, T Ma ij is the actual measurement value of the jth manufacturing feature, T Ma kj is the actual measured value of the feature on the k-th agent that matches feature j on the ith agent;

[0086] According to the manufacturing accuracy and the coordination accuracy, the state of the i-th agent at time t is determined, and the formula is as follows:

[0087] s it =concat(ε it ,γ it )

[0088] Step 306: Execute the joint action in the multi-object assembly coordination simulation environment to obtain the rewards of the N agents and the observation value at the next moment, and store the interaction experience of the N agents and the environment in the N experience buffers respectively; wherein the interaction experience includes the current observation value, the action, the reward value and the observation value at the next moment;

[0089] Step 307: Determine whether the size of each experience buffer area reaches a preset value;

[0090] If the size of each experience buffer area does not reach the preset value, continue to execute step 305 to step 306, otherwise, execute step 308;

[0091] Step 308: Randomly sample a group of the interaction experiences in the experience buffer as samples to train the multi-agent reinforcement learning assembly coordination model.

[0092] like Figure 4 As shown in FIG. 1 , a set of experiences is randomly selected from the experience buffer as a sample to train the assembly coordination model, that is, the policy network and the value network in the reinforcement learning model are updated by using proximal policy optimization. The specific steps are as follows:

[0093] Step 401: randomly sampling a set of the interaction experiences from the experience buffer, and determining a target value function estimate using the assembly coordination model based on the sampled interaction experiences; wherein the formula of the interaction experience is as follows:

[0094]

[0095] Where T is the maximum number of time steps that each agent can take in one round;

[0096] The formula for estimating the target value function is as follows:

[0097]

[0098] Here, λ is a parameter between 0 and 1. is the joint cumulative reward of each agent starting from time t and considering the next n steps, N s To consider the upper limit of the number of future steps; the formula for the joint cumulative reward is as follows:

[0099]

[0100] Among them, T n is the time horizon starting from time t and looking forward n steps, γ is the discount factor used to reduce the dependence on the long-term reward, is the joint reward received by all agents at time t′.

[0101] Step 402: The joint state of the N agents Input the value network to get the current value function estimate

[0102] Step 403: Determine the loss value of the value network according to the target value function estimate and the current value function estimate; wherein the formula of the loss function of the value network is as follows:

[0103]

[0104] Among them, Q is the total number of samples, which is used to normalize the loss function;

[0105] Step 404: Update the parameters of the value network according to the loss value.

[0106] Step 405: randomly sampling a set of the interaction experiences from the experience buffer of each agent, and determining the loss value of each agent strategy network using the deformation control model according to the sampled interaction experiences; wherein the formula of the loss function of the i-th agent strategy network is as follows:

[0107]

[0108] Among them, θ i Represents the current strategy parameters of the i-th agent. t (θ i ) is the strategy probability ratio of the ith agent That is, the new strategy π θ (a it |s it ) Compared to the old strategy probability ratio. is the global advantage function estimate of all agents, used to evaluate all agents in the joint state s t Take joint action t Advantage relative to the average joint action. clip(r t (θ i ),1-,1+) is a clipping function that reduces the strategy probability ratio r t (θ i ) is limited to [1-,1+] to avoid excessive changes in the policy update step, thereby maintaining the stability of the learning process. ε is a hyperparameter used to control the degree of pruning and is usually very small. The formula for estimating the global advantage function of the agent using generalized advantage is as follows:

[0109]

[0110] Among them, δt =r t +γV(s t+1 )-V(s t ), r t is the joint reward of all agents at time t, V(s i ) is the value function estimate, γ is the discount factor, and λ is the trade-off factor used to weigh the bias and variance of the estimate.

[0111] Step 406: Update the policy network according to the loss value of each agent strategy.

[0112] Step 407: Determine whether the number of iterations reaches the preset number of training times. Specifically, determine whether the number of iterations of the multi-agent assembly coordination model at this time reaches the preset number of training times. If the number of iterations of the multi-agent assembly coordination model at this time does not reach the preset number of training times, continue to iterate and execute steps 401 to 407, and update the centralized value network and the strategy network of each agent through continuous iteration. Otherwise, execute step 406.

[0113] Step 408: Iteratively execute the update step until the number of iterations of the multi-agent assembly coordination model reaches a preset number of training times and the experience buffer area is cleared.

[0114] S5. Input the current assembly information of each assembly object in the aircraft assembly process into the optimal aircraft component assembly coordination model to output the assembly object process control parameters of each assembly object at the next moment, thereby realizing real-time assembly coordination and optimization of aircraft components.

[0115] This embodiment also provides an aircraft component assembly coordination device, which is used to implement the steps of the aircraft component assembly coordination method based on multi-agent reinforcement learning provided in the above embodiment.

[0116] In addition, the terms "upper", "lower", "inner", "outer", "front", "rear" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0117] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. All equivalent changes or modifications made according to the structure, characteristics and principles described in the patent application scope of the present invention should be included in the patent application scope of the present invention.

[0118] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for aircraft component assembly coordination based on multi-agent reinforcement learning, characterized in that: The following steps are involved: According to the assembly process requirements of multiple assembly objects of the aircraft structure, an assembly accuracy mathematical model including the one used to characterize the manufacturing accuracy and coordination accuracy is constructed; According to the assembly accuracy control parameters and variation range of each digital tooling in the assembly scene, a multi-object assembly deformation response data set is obtained by combining sampling design and numerical simulation, and a corresponding aircraft structure multi-object assembly deformation proxy model is obtained by training based on the multi-object assembly deformation response data set; Using a single-agent reinforcement learning method, a corresponding assembly deformation control model is constructed for each assembly object, wherein the assembly deformation control model includes a strategy network and weight parameters for controlling the displacement field and stress field distribution of the assembly object; Based on the assembly accuracy mathematical model, the aircraft structure multi-object assembly deformation agent model, and the assembly deformation control model of each assembly object, a multi-agent reinforcement learning assembly coordination model is constructed. The multi-agent reinforcement learning assembly coordination model uses the process control parameters of each assembly object as the action of the corresponding agent, the manufacturing accuracy corresponding to the assembly object and the coordination accuracy between the assembly objects as the state of the corresponding agent, and the optimization degree of the assembly accuracy as the reward function to measure whether the current strategy is helpful to improve the assembly accuracy of multiple objects; Combine the actions, states, and reward functions of all agents at the same time into corresponding joint actions, joint states, and joint rewards; Initializing the multi-agent reinforcement learning assembly coordination model with the strategy network weights of the assembly deformation control model corresponding to each assembly object; Based on a centralized training distributed execution architecture, the multi-agent reinforcement learning assembly coordination model is trained by sharing joint actions, joint states, and joint rewards of all agents to obtain an optimal aircraft component assembly coordination model; The current assembly information of each assembly object in the aircraft assembly process is input into the optimal aircraft component assembly coordination model to output the process control parameters of each assembly object at the next moment.

2. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The manufacturing accuracy is the degree to which the geometric shape of the actual assembly object conforms to the nominal size specified in its digital model, and its expression is as follows: Where L is the number of manufacturing features possessed by the ith agent, w j is the relative weight of each manufacturing feature, T Ma ij is the actual measurement value of the jth manufacturing feature, N Ma ij is the nominal value of the jth manufacturing feature.

3. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The coordination accuracy is the actual geometric shape and shape compatibility of the matching features between different assembly objects, and its expression is as follows: Where M is the number of features that the ith agent has coordination requirements, V is the number of agents that have coordination requirements with the jth feature on the ith agent, T Ma ij is the actual measurement value of the jth manufacturing feature, T Ma kj is the actual measured value of the feature on the k-th agent that matches feature j on the ith agent.

4. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 1 is characterized in that: During the training process, the multi-agent reinforcement learning assembly coordination model is initialized with the policy network weights of the assembly deformation control model corresponding to each assembly object; The initialization method alleviates the high variance and low efficiency caused by random exploration in the early stage of training of the multi-agent system by providing pre-trained weight parameters.

5. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 1, characterized in that: The proximal policy optimization is adopted to train the policy network and value network of the multi-agent. When interacting with the multi-object assembly deformation agent model, each agent makes independent decisions according to its own observed local state. When the network is updated and trained, all agents share a centralized value function. The value network is updated by sharing the action, state and reward information of all agents, and the global generalized advantage estimate of all agents is calculated based on the centralized value function to update the policy network.

6. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 5 is characterized in that: The loss function of the value network is calculated based on the target value function estimate and the current value function estimate. The formula for the target value function estimate is as follows: Here, λ is a parameter between 0 and 1. is the joint cumulative reward of each agent starting from time t and considering the next n steps, N s To consider the upper limit of future steps; The joint state of N agents Input the value network to get the current value function estimate The loss function of the value network is as follows: Among them, Q is the total number of samples.

7. The aircraft component assembly coordination method based on multi-agent reinforcement learning according to claim 5 is characterized in that: During the training process, the loss function expression of the policy network is as follows: Among them, θ i represents the current strategy parameters of the i-th agent, r t (θ i ) is the strategy probability ratio of the ith agent; is the global advantage function estimate of all agents, used to evaluate all agents in the joint state s t Take joint action t Advantage relative to the average joint action; clip(r t (θ i ),1-∈,1+∈) is a clipping function used to reduce the policy probability ratio r t (θ i ) is restricted to [1-∈, 1+∈]; ε is a hyperparameter used to control the degree of clipping.

8. An aircraft component assembly coordination device, characterized in that: Steps for implementing the aircraft component assembly coordination method based on multi-agent reinforcement learning as described in any one of claims 1 to 7.

Images