Aircraft structure assembly deformation real-time control method based on deep reinforcement learning

Abstract

The invention discloses an aircraft structure assembly deformation real-time control method based on deep reinforcement learning, and the method comprises the steps: building a corresponding mapping relation model based on process control parameters and corresponding aircraft structure deformation field variable data; the mapping relation model comprises a first stage and a second stage, the first stage comprises a convolutional neural network, and the convolutional neural network is used for converting aircraft structure deformation field variable data between high-dimensional representation and low-dimensional representation; based on the mapping relation, a deformation control reinforcement learning model is constructed, and a deformation control strategy model is obtained through training by taking the control parameters as actions and taking the low-dimensional representation of the field variables as states; and inputting the aircraft structure deformation field variable data at the current moment into the deformation control strategy model so as to output process control parameters at the next moment. According to the method, the control parameters can be generated in real time according to the actual state of the assembly object on the assembly site so as to optimize deformation of the assembly object.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft automated assembly, and in particular relates to a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning. Background Art

[0002] The aerodynamic shape of an aircraft significantly affects its performance, and the deformation of each aircraft structure needs to be optimized and controlled during the assembly stage. Traditional assembly deformation models rely on finite element models to calculate complex structural deformations through numerical discretization of mechanical and dynamic equations. This process usually requires dense grids and multiple iterations to reach convergence, which is time-consuming. In addition, assembly deformation optimization also requires iterative calculations. Therefore, deformation optimization methods based on finite element models are difficult to be directly used to optimize and control aircraft structural deformation at the assembly site. In addition, modern aircraft widely use carbon fiber reinforced composites instead of traditional metal structures during the manufacturing process. Due to the laminated manufacturing characteristics of composite structures, the bonding strength between layers is weak, which makes them prone to damage such as delamination under stress concentration. Therefore, in the manufacturing process, in addition to ensuring geometric accuracy, full attention must also be paid to stress distribution.

[0003] Patent document CN 118502364A discloses a large aircraft assembly production robot scheduling method based on deep reinforcement learning, including the following steps: S1: Problem modeling uses an attention-based neural network to abstract heterogeneous AGVs into intelligent agents in the neural network, and concretizes specific task goals as colored coordinate points to achieve the description and modeling of the MRS task scheduling problem; S2: Network construction builds the attention-based neural network in step S1, and determines the loss function during neural network training; S3: Network training trains the neural network built in step S2 to obtain a trained MRS task scheduling model; S4: Task scheduling uses the MRS task scheduling model in step S3 to complete the scheduling of MRS tasks.

[0004] Patent document CN 116757398A discloses a method, device and storage medium for generating aircraft assembly scheduling data, including: establishing a MILP model for assembly scheduling of a target aircraft, the MILP model including an objective function and a constraint function, the objective function including the relationship between the maximum completion time of aircraft assembly of the target aircraft and the resources required in the aircraft assembly process; the constraint function including the constraint conditions of the resources required in the aircraft assembly process; minimizing the objective function, and solving the MILP model using a trained target reinforcement learning model to obtain target configuration parameters of the resources, the target reinforcement learning model is built based on a fusion Gomory cutting plane method and is obtained through training based on historical assembly scheduling data; and determining the aircraft assembly schedule of the target aircraft according to the target configuration parameters. Summary of the invention

[0005] The purpose of the present invention is to provide a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning, which can generate control parameters in real time according to the actual state of the assembly object at the assembly site to optimize the deformation of the assembly object.

[0006] In order to achieve the purpose of the present invention, the following technical solution is provided: a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning, comprising the following steps:

[0007] Conduct simulation experiments based on aircraft structure geometry and material parameters to obtain aircraft structure deformation field variable data corresponding to different process control parameters, and build a corresponding mapping relationship model based on the obtained process control parameters and the corresponding aircraft structure deformation field variable data;

[0008] The mapping relationship model includes a first stage and a second stage, wherein the first stage includes a convolutional neural network, and the convolutional neural network is used to convert aircraft structure deformation field variable data between high-dimensional representation and low-dimensional representation;

[0009] The second stage includes a multi-layer perceptron for constructing a mapping between process control parameters and a low-dimensional representation of aircraft structure deformation field variable data;

[0010] The low-dimensional representation generated by the convolutional neural network is used as the agent state, and the process control parameters are used as actions. A corresponding agent strategy network and value network are constructed based on a deep reinforcement learning framework, and the interaction results between the agent and the mapping relationship model are used as training data. The agent strategy network and value network are trained to update the parameter weights, so as to obtain a deformation control strategy model for predicting the process control parameters at the next moment;

[0011] When aircraft structure deformation control is performed at the assembly site, the aircraft structure deformation field variable data at the current moment is input into the deformation control strategy model to output the process control parameters at the next moment.

[0012] The present invention constructs an aircraft structure assembly deformation dataset by combining sampling design, structural mechanics simulation and parametric modeling technology, and constructs an assembly deformation mapping relationship model based on the dataset using a deep neural network, providing a simulation environment for the training of a deep reinforcement learning model. Based on the constructed reinforcement learning simulation environment, the intelligent agent continuously interacts with the environment to complete the learning of the deformation control optimization strategy, which can generate control parameters in real time according to the actual state of the assembly object at the assembly site to optimize the deformation of the assembly object.

[0013] Specifically, the specific process of the simulation experiment is as follows:

[0014] According to the digital tooling used in the aircraft structure assembly process, the process control parameters and the actuation range of the digital tooling are determined, and the process control parameters are sampled as the boundary conditions of the model to obtain the aircraft structure deformation field variable data corresponding to different sampling points.

[0015] Specifically, the convolutional neural network uses reconstruction error as a loss function and is trained by minimizing the loss function value.

[0016] Specifically, in the process of training the agent strategy network, proximal strategy optimization is used to train the agent strategy network and value network, and an orthogonal experiment is designed to select the optimal hyperparameter combination of reinforcement learning, and the agent strategy network and value network are trained based on the optimal hyperparameter combination.

[0017] Specifically, the training process of the proximal strategy optimization includes a strategy part and a value part:

[0018] The loss function of the strategy part is expressed as follows:

[0019]

[0020] Among them, θ represents the current policy parameters, r t (θ) is the strategy probability ratio is the advantage function estimate, clip(r t (θ), 1-ε, 1+ε) is a clipping function used to convert the policy probability ratio r t (θ) is limited to [1-ε, 1+ε];

[0021] The loss function of the value part is expressed as follows:

[0022]

[0023] in, Is the status under the current strategy and parameters The current value function estimate of N is the total number of samples, Estimate the target value function.

[0024] Specifically, the expression of the advantage function estimation is as follows:

[0025]

[0026] Among them, δ t =r t +γV(s t+1 )-V(s t ), r t is the reward at time t, V(s) is the value function estimate, γ is the discount factor, and λ is the trade-off factor.

[0027] Specifically, the reward function of the agent is designed according to the degree of optimization of aircraft structure deformation field variable data in the assembly scenario to measure whether the current strategy is helpful to improve the aircraft structure field variable distribution.

[0028] Specifically, the expression of the reward function is as follows:

[0029] r t =f M [S0+ψ(φ(a t ))]-f M [S0+ψ(φ(a t+1 ))]

[0030] Where S0 is the initial field variable distribution, φ is the multi-layer perceptron in the second stage of the mapping relationship model, ψ is the decoder in the convolutional neural network in the first stage of the mapping relationship model that maps the low-dimensional representation back to the high-dimensional field variable, and f M It is an indicator function to measure the distribution of field variables.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] A framework for assembly deformation optimization combining deformation mapping relationship model and deep reinforcement learning is proposed. A deep reinforcement learning simulation training environment is constructed based on the assembly deformation mapping relationship model, and the intelligent agent can quickly learn assembly deformation optimization strategies.

[0033] The latent variables in the low-dimensional space obtained by the deformation mapping relationship model are used as the state of the intelligent agent in deep reinforcement learning, which solves the problems of low training efficiency, great generalization difficulty, high risk of overfitting and poor stability caused by high-dimensional state space. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A framework flow chart of the real-time control method for aircraft structure assembly deformation provided in this embodiment;

[0035] Figure 2 A schematic diagram of the structure of the mapping relationship model provided in this embodiment;

[0036] Figure 3 The deformation control agent training process provided in this embodiment;

[0037] Figure 4 The inference test results provided for this embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning is provided in this embodiment. Since the aircraft structure deformation field variable data of aircraft components is usually as high as one million dimensions, this embodiment adopts a two-stage mapping relationship model, wherein the first stage is a convolutional neural network for compressing data, and the second stage is a multi-layer perceptron for recovering the aircraft structure deformation field variable data from the compressed data.

[0040] The model can quickly respond to environmental changes in the interaction between the agent and the environment, and take actions based on the current strategy to convert the current state of the agent into t Transform to the next state s t+1 By defining the state of the agent as a low-dimensional representation of the field variables in the latent space after being compressed by the deformation mapping relationship model, the training process of the deep reinforcement learning algorithm is simplified, and the problems of low training efficiency, high generalization difficulty, high risk of overfitting and poor stability caused by the high-dimensional state space are solved. tThe input is sent to the multi-layer perceptron in the second stage of the mapping relationship model to complete the update of the agent state, and the reward obtained by taking this action is converted to the state s of the latent space through the decoder in the first stage of the mapping relationship model. t Converted into deformation index for calculation.

[0041] The specific process is as follows:

[0042] By analyzing the digital assembly of aircraft structures in specific assembly scenarios, the aircraft structure assembly deformation control parameters and their actuation ranges are determined, and the control parameters are set as the boundary conditions of the simulation model. By sampling in the control parameter space and combining parametric simulation modeling technology, an aircraft structure assembly deformation simulation data set is constructed.

[0043] Based on the aircraft structure assembly deformation dataset, a mapping relationship model is constructed to characterize the mapping relationship between control parameters and aircraft structure deformation field variable data, providing a simulation environment for the training of deep reinforcement learning models.

[0044] like Figure 2 As shown, the mapping relationship model includes two stages. In the first stage, a convolutional neural network is used to compress high-dimensional field variable data into a low-dimensional representation. By reducing the dimension of the original high-dimensional field variable and using the low-dimensional representation of the field variable as the agent state of the reinforcement learning model, the training difficulty and instability caused by the high-dimensional state space are alleviated; the formula of the first stage is as follows:

[0045]

[0046] ψ:z→x

[0047] in, represents the encoder in the convolutional neural network that compresses high-dimensional field variable data into low-dimensional data, x is the original high-dimensional field variable data, z is the low-dimensional representation of the field variable obtained after compression, ψ represents the decoder in the convolutional neural network that restores the low-dimensional representation of the field variable data to the original high-dimensional data, and the convolutional neural network uses the reconstruction error as the loss function, and completes the training of the convolutional neural network by minimizing the loss function value; the loss function formula is as follows:

[0048]

[0049] In the second stage, a multi-layer perceptron is used to complete the mapping between the input variables and the potential representation of the field variables. The formula is as follows:

[0050] z=φ(a)

[0051] Among them, φ is the multi-layer perceptron used in the second stage, and a is the input control parameter.

[0052] The low-dimensional representation generated by the convolutional neural network is used as the agent state, and the process control parameters are used as actions. A corresponding agent policy network is constructed based on a deep reinforcement learning framework. The interaction results between the agent and the mapping relationship model are used as training data. The agent policy network and value network are trained to update the parameter weights, so as to obtain a deformation control strategy model for predicting the process control parameters at the next moment.

[0053] More specifically, a deep reinforcement learning model for aircraft structure deformation control is constructed using the assembly deformation mapping relationship model as a simulation environment, wherein the actions of the intelligent agent in the reinforcement learning model are the aircraft structure deformation process control parameters, and an experience buffer is set;

[0054] Set the current state and execution action for the agent, and set the reward function of the agent; the reward function is designed according to the degree of optimization of the field variables, and measures whether the current strategy is helpful to improve the field variable distribution of the aircraft structure. The reward of the agent is defined as the degree of improvement of the deformation index, which is obtained by inputting the state of the agent at time t into the decoder of the first stage of the mapping relationship model, as follows:

[0055] The current observation value of the intelligent agent observing the aircraft structure deformation simulation environment based on the mapping relationship model is obtained; the state of the intelligent agent is represented by the latent variables in the mapping relationship model, which can be obtained by inputting the action taken by the intelligent agent into the multi-layer perceptron of the second stage of the mapping relationship model. The formula of the intelligent agent state is as follows:

[0056] s t =v(a t )

[0057] Among them, s t and a t They represent the state and action of the agent at time t respectively, and φ is the multi-layer perceptron in the second stage of the deformation mapping relationship model.

[0058] Assuming that the initial field variable distribution is S0, the relationship between the action of the agent, i.e., the assembly deformation control parameter, and the initial field variable can be expressed by the following formula:

[0059] S0=ψ[φ(a0)]

[0060] Among them, ψ is the decoder in the first stage convolutional neural network that maps the low-dimensional representation back to the high-dimensional field variable.

[0061] Since the goal of agent learning is to optimize the distribution of field variables by changing actions, the objective function can be expressed as:

[0062]

[0063] Among them, f M To measure the evaluation function of field variable distribution, the specific formula is as follows:

[0064]

[0065] Since the variation range of deformation control parameters in actual assembly is small, the optimization objective can be approximated as:

[0066]

[0067] The reward function can be set as the field variable deformation index f at time t and t+1 M The difference of (S) is as follows:

[0068] r t =f M [S0+S t ]-f M [S0+S t+1 ]

[0069] =f M [S0+ψ(s t )]-f M [S0+ψ(s t+1 )]

[0070] =f M [S0+ψ(φ(a t ))]-f M [S0+ψ(φ(a t+1 ))]

[0071] Among them, s t and t+1 are the states of the agent at time t and t+1 respectively, S t and S t+1 are the distribution of field variables at time t and t+1 respectively, and the formula is as follows:

[0072] s t =φ(a t ), s t+1 =φ(a t+1 )

[0073] S t =ψ(s t ), S t+1 =ψ(s t+1 )

[0074] The action is performed in the aircraft structure deformation simulation environment based on the deformation mapping relationship model, thereby obtaining the reward of the agent and the observation value at the next moment, and storing the interaction experience between the agent and the environment in the experience buffer; wherein the interaction experience includes the current observation value, the action, the reward value and the observation value at the next moment;

[0075] A group of the interaction experiences are randomly sampled in the experience buffer as samples to train the deep reinforcement learning model, and the performance of the deformation control model after training is evaluated; wherein, the random sampling of a group of the interaction experiences in the experience buffer as samples to train the deformation control reinforcement learning model is specifically as follows:

[0076] Iteratively executing the updating step until the number of iterations of the deformation control model reaches a preset number of training times and the experience buffer area is cleared;

[0077] The updating step uses proximal strategy optimization to train the policy network and the value network in the reinforcement learning model, including:

[0078] A set of the interaction experiences is randomly sampled from the experience buffer, and a target value function estimate is determined using the deformation control model according to the sampled interaction experiences; wherein the target value function estimate is formulated as follows:

[0079]

[0080] Here, λ is a parameter between 0 and 1 that controls the depth of the discount, that is, the importance of future rewards when evaluating the value of the current state. is the cumulative return starting from time t and considering the next n steps, N s represents the upper limit of the number of future steps considered; the formula for the cumulative reward is as follows:

[0081]

[0082] Among them, T n is the time horizon starting from time t and looking forward n steps, γ is the discount factor used to reduce the dependence on the long-term reward, is the reward received at time t′.

[0083] The agent's own state Input the value network to obtain a current value function estimate;

[0084] The loss value of the value network is determined according to the target value function estimate and the current value function estimate, and the parameters of the value network are updated according to the loss value; wherein the loss function formula of the value network is as follows:

[0085]

[0086] in, is the state under the current strategy and parameters The current value function estimate of , N is the total number of samples, used to normalize the loss function;

[0087] A set of the interaction experiences is randomly sampled from the experience buffer, and the loss value of the strategy network is determined using the deformation control model according to the sampled interaction experiences; wherein the formula of the loss function of the strategy part is as follows:

[0088]

[0089] Among them, θ represents the current policy parameters. t (θ) is the strategy probability ratio That is, the new strategy π θ (a t |s t ) Compared to the old strategy probability ratio. is the advantage function estimate, used to evaluate t Take action t Advantage over average action. clip(r t (θ), 1-ε, 1+ε) is a clipping function that reduces the policy probability ratio r t (θ) is limited to [1-ε, 1+ε] to avoid excessive changes in the policy update step, thereby maintaining the stability of the learning process. ε is a hyperparameter used to control the strictness of clipping and is usually taken as a small value.

[0090] The formula for estimating the advantage function using generalized advantage is as follows:

[0091]

[0092] Among them, δ t =r t +γV(s t+1 )-V(s t ), r t is the reward at time t, V(s) is the value function estimate, γ is the discount factor, and λ is the trade-off factor used to weigh the bias and variance of the estimate.

[0093] The simulation environment based on the mapping relationship model and the hyperparameters of the proximal policy optimization algorithm are analyzed, and the optimal hyperparameter combination is selected by designing orthogonal experiments to optimize the training process of the deformation control strategy network. Figure 3As shown, after 600,000 steps of interaction with the environment constructed by the deformation mapping relationship model, the average reward return of the agent stabilized at a high level, indicating that the training of the optimization strategy has converged.

[0094] like Figure 4 As shown, the deformation optimization strategy of the intelligent agent is verified on the test set of the fuselage structure deformation data set. The stress field data of the aircraft structure in the initial state is input into the network of the first stage of the deformation mapping relationship model to obtain a low-dimensional representation of the stress field, which is then input as the state of the intelligent agent into the policy network in the deformation control reinforcement learning model and control parameters are generated through policy network reasoning to achieve aircraft structure deformation control and optimization. The experimental results show that the assembly deformation index of the test set is reduced by 90% on average, and the deformation index of 95% of the test samples has decreased by more than 76%.

[0095] In addition, the terms "upper", "lower", "inner", "outer", "front", "rear" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0096] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. All equivalent changes or modifications made according to the structure, characteristics and principles described in the patent application scope of the present invention should be included in the patent application scope of the present invention.

[0097] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A real-time control method for aircraft structure assembly deformation based on deep reinforcement learning, characterized in that: The following steps are involved: Conduct simulation experiments based on aircraft structure geometry and material parameters to obtain aircraft structure deformation field variable data corresponding to different process control parameters, and build a corresponding mapping relationship model based on the obtained process control parameters and the corresponding aircraft structure deformation field variable data; The mapping relationship model includes a first stage and a second stage, wherein the first stage includes a convolutional neural network, and the convolutional neural network is used to convert aircraft structure deformation field variable data between high-dimensional representation and low-dimensional representation; The second stage includes a multi-layer perceptron for constructing a mapping between process control parameters and a low-dimensional representation of aircraft structure deformation field variable data; The low-dimensional representation generated by the convolutional neural network is used as the agent state, and the process control parameters are used as actions. A corresponding agent strategy network and value network are constructed based on a deep reinforcement learning framework, and the interaction results between the agent and the mapping relationship model are used as training data. The agent strategy network and value network are trained to update the parameter weights, so as to obtain a deformation control strategy model for predicting the process control parameters at the next moment; When aircraft structure deformation control is performed at the assembly site, the aircraft structure deformation field variable data at the current moment is input into the deformation control strategy model to output the process control parameters at the next moment.

2. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 1 is characterized in that: The specific process of the simulation experiment is as follows: According to the digital tooling used in the aircraft structure assembly process, the process control parameters and the actuation range of the digital tooling are determined, and the process control parameters are sampled as the boundary conditions of the model to obtain the aircraft structure deformation field variable data corresponding to different sampling points.

3. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 1 is characterized in that: The convolutional neural network uses reconstruction error as a loss function and is trained by minimizing the value of the loss function.

4. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 1 is characterized in that: During the training process, proximal strategy optimization is used to train the agent strategy network and value network, and an orthogonal experiment is designed to select the optimal hyperparameter combination of reinforcement learning. The agent strategy network and value network are trained based on the optimal hyperparameter combination.

5. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 4 is characterized in that: The training process of the proximal strategy optimization includes a strategy part and a value part: The loss function of the strategy part is expressed as follows: Among them, θ represents the current policy parameters, r t (θ) is the strategy probability ratio is the advantage function estimate, clip(r t (θ),1-ε,1+ε) is a clipping function used to convert the policy probability ratio r t (θ) is limited to [1-ε, 1+ε]; The loss function of the value part is expressed as follows: in, Is the status under the current strategy and parameters The current value function estimate of , N is the total number of samples, Estimate the target value function.

6. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 5 is characterized in that: The expression of the advantage function estimation is as follows: Among them, δ t =r t +γV(s t+1 )-V(s t ), r t is the reward at time t, V(s) is the value function estimate, γ is the discount factor, and λ is the trade-off factor.

7. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 1 is characterized in that: In the deep reinforcement learning framework, the agent's reward function is designed according to the degree of optimization of the aircraft structure deformation field variable data in the assembly scenario to measure whether the current strategy helps to improve the aircraft structure field variable distribution.

8. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 7 is characterized in that: The expression of the reward function is as follows: r t =f M [S0+ψ(φ(a t ))]-f M [S0+ψ(φ(a t+1 ))] Among them, S0 is the initial field variable distribution, φ is the multi-layer perceptron in the second stage, ψ is the decoder in the first stage convolutional neural network that maps the low-dimensional representation back to the high-dimensional field variable, and f M It is an indicator function to measure the distribution of field variables.

Images