A real-time control method for aircraft structure assembly deformation based on deep reinforcement learning

Abstract

The present invention discloses a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning. The method comprises: constructing a corresponding mapping relationship model based on process control parameters and corresponding aircraft structure deformation field variable data; the mapping relationship model comprises a first stage and a second stage, wherein the first stage comprises a convolutional neural network, which is used to convert the aircraft structure deformation field variable data between high-dimensional and low-dimensional representations; constructing a deformation control reinforcement learning model based on the mapping relationship, using the control parameters as actions and the low-dimensional representation of the field variables as states, to train a deformation control strategy model; and inputting the aircraft structure deformation field variable data at the current moment into the deformation control strategy model to output the process control parameters at the next moment. The method provided by the present invention can generate control parameters in real time based on the actual state of the assembly object at the assembly site to optimize the deformation of the assembly object.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft automated assembly, and in particular relates to a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning. Background Art

[0002] An aircraft's aerodynamic shape significantly impacts its performance, necessitating optimization and control of the deformation of each aircraft structure during assembly. Traditional assembly deformation models rely on finite element models, numerically discretizing mechanical and dynamic equations to calculate complex structural deformations. This process typically requires dense meshes and multiple iterations to reach convergence, resulting in a time-consuming process. Furthermore, assembly deformation optimization also requires iterative calculations. Therefore, deformation optimization methods based on finite element models are difficult to apply directly to optimizing and controlling aircraft structural deformation at the assembly site. Furthermore, modern aircraft manufacturing widely utilizes carbon fiber-reinforced composites (CFRPs) rather than traditional metal structures. Due to the layered nature of composite structures, the bonding between layers is weak, making them susceptible to damage such as delamination under stress concentration conditions. Therefore, in addition to ensuring geometric accuracy, stress distribution must also be carefully considered during the manufacturing process.

[0003] Patent document CN 118502364A discloses a large aircraft assembly production robot scheduling method based on deep reinforcement learning, including the following steps: S1: Problem modeling uses an attention-based neural network to abstract heterogeneous AGVs into intelligent agents in the neural network, and concretizes specific task goals as colored coordinate points to achieve the description and modeling of the MRS task scheduling problem; S2: Network construction builds the attention-based neural network in step S1 and determines the loss function during neural network training; S3: Network training trains the neural network built in step S2 to obtain a trained MRS task scheduling model; S4: Task scheduling uses the MRS task scheduling model in step S3 to complete the scheduling of MRS tasks.

[0004] Patent document CN 116757398A discloses a method, device, and storage medium for generating aircraft assembly scheduling data, comprising: establishing a MILP model for the assembly scheduling of a target aircraft, the MILP model including an objective function and a constraint function, the objective function including the relationship between the maximum completion time of aircraft assembly of the target aircraft and the resources required during the aircraft assembly process; the constraint function including the constraint conditions of the resources required during the aircraft assembly process; minimizing the objective function and solving the MILP model using a trained target reinforcement learning model to obtain target configuration parameters of the resources, the target reinforcement learning model being constructed based on a fusion Gomory cutting plane method and trained based on historical assembly scheduling data; and determining the aircraft assembly schedule of the target aircraft based on the target configuration parameters. Summary of the Invention

[0005] The purpose of the present invention is to provide a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning, which can generate control parameters in real time according to the actual state of the assembly object at the assembly site to optimize the deformation of the assembly object.

[0006] To achieve the purpose of the present invention, the following technical solution is provided: a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning, comprising the following steps:

[0007] Conduct simulation experiments based on aircraft geometry and material parameters to obtain aircraft structural deformation field variable data corresponding to different process control parameters, and construct a corresponding mapping relationship model based on the obtained process control parameters and the corresponding aircraft structural deformation field variable data;

[0008] The mapping relationship model includes a first stage and a second stage, wherein the first stage includes a convolutional neural network, and the convolutional neural network is used to convert aircraft structure deformation field variable data between high-dimensional representation and low-dimensional representation;

[0009] The second stage includes a multi-layer perceptron for constructing a mapping between process control parameters and low-dimensional representations of aircraft structure deformation field variable data;

[0010] Using the low-dimensional representation generated by the convolutional neural network as the agent state and the process control parameters as actions, a corresponding agent policy network and value network are constructed based on a deep reinforcement learning framework. The interaction results between the agent and the mapping relationship model are used as training data. The agent policy network and value network are trained to update parameter weights to obtain a deformation control strategy model for predicting the process control parameters at the next moment;

[0011] When aircraft structure deformation control is performed at the assembly site, the aircraft structure deformation field variable data at the current moment is input into the deformation control strategy model to output the process control parameters at the next moment.

[0012] The present invention constructs an aircraft structure assembly deformation dataset by combining sampling design, structural mechanics simulation and parametric modeling technology, and constructs an assembly deformation mapping relationship model based on the dataset using a deep neural network, providing a simulation environment for the training of the deep reinforcement learning model. Based on the constructed reinforcement learning simulation environment, the intelligent agent continuously interacts with the environment to complete the learning of the deformation control optimization strategy. This strategy can generate control parameters in real time according to the actual state of the assembly object at the assembly site to optimize the deformation of the assembly object.

[0013] Specifically, the specific process of the simulation experiment is as follows:

[0014] According to the digital tooling used in the aircraft structure assembly process, the process control parameters and actuation range of the digital tooling are determined, and the process control parameters are sampled as the boundary conditions of the model to obtain the aircraft structure deformation field variable data corresponding to different sampling points.

[0015] Specifically, the convolutional neural network uses reconstruction error as a loss function and is trained by minimizing the loss function value.

[0016] Specifically, during the training process of the intelligent agent policy network, proximal strategy optimization is used to train the intelligent agent policy network and value network, and an orthogonal experiment is designed to select the optimal hyperparameter combination of reinforcement learning, and the intelligent agent policy network and value network are trained based on the optimal hyperparameter combination.

[0017] Specifically, the training process of the proximal strategy optimization includes a strategy part and a value part:

[0018] The loss function of the strategy part is expressed as follows: ;in, Indicates the current policy parameters, is the strategy probability ratio , is the advantage function estimate, is a clipping function that converts the policy probability ratio Restricted to The loss function of the value part is expressed as follows: ;in, Is the state under the current strategy and parameters The current value function estimate of is the total number of samples, Estimate the target value function.

[0019] Specifically, the expression of the advantage function estimation is as follows: ;in, , is the reward at time t, is the value function estimate, is the discount factor, is a trade-off factor.

[0020] Specifically, the agent's reward function is designed according to the degree of optimization of aircraft structure deformation field variable data in the assembly scenario to measure whether the current strategy helps to improve the aircraft structure field variable distribution.

[0021] Specifically, the reward function is expressed as follows: ;in, is the initial field variable distribution, is the multi-layer perceptron of the second stage of the mapping relationship model, A decoder that maps low-dimensional representations back to high-dimensional field variables in the first stage convolutional neural network of the mapping relationship model, It is an indicator function to measure the distribution of field variables.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] We propose an assembly deformation optimization framework that combines a deformation mapping relationship model with deep reinforcement learning. Based on the assembly deformation mapping relationship model, we construct a deep reinforcement learning simulation training environment, allowing intelligent agents to quickly learn assembly deformation optimization strategies.

[0024] The latent variables in the low-dimensional space obtained by the deformation mapping relationship model are used as the state of the intelligent agent in deep reinforcement learning, which solves the problems of low training efficiency, great generalization difficulty, high overfitting risk and poor stability caused by high-dimensional state space. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A framework flow chart of the real-time control method for aircraft structure assembly deformation provided in this embodiment;

[0026] Figure 2 A schematic diagram of the structure of the mapping relationship model provided in this embodiment;

[0027] Figure 3 The deformation control agent training process provided in this embodiment;

[0028] Figure 4 This is the inference test result provided by this embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, a real-time control method for aircraft structure assembly deformation based on deep reinforcement learning is provided in this embodiment. Since the aircraft structure deformation field variable data of aircraft components usually has up to one million dimensions, this embodiment adopts a two-stage mapping relationship model, wherein the first stage is a convolutional neural network for compressing data, and the second stage is a multi-layer perceptron for recovering the aircraft structure deformation field variable data from the compressed data.

[0031] The model can quickly respond to environmental changes in the interaction between the agent and the environment, and take actions based on the current strategy to change the current state of the agent. Transition to the next state .

[0032] By defining the state of the intelligent body as a low-dimensional representation in the latent space after the field variables are compressed by the deformation mapping relationship model, the training process of the deep reinforcement learning algorithm is simplified, and the problems of low training efficiency, difficulty in generalization, high risk of overfitting and poor stability caused by the high-dimensional state space are solved. The input is sent to the multi-layer perceptron in the second stage of the mapping relationship model to complete the update of the agent state, and the reward obtained by taking this action is converted to the state of the latent space through the decoder in the first stage of the mapping relationship model. Converted into deformation index for calculation.

[0033] The specific process is as follows:

[0034] By analyzing the digital assembly used in aircraft structure assembly scenarios, the aircraft structure assembly deformation control parameters and their actuation ranges are determined, and the control parameters are set as the boundary conditions of the simulation model. By sampling in the control parameter space and combining parametric simulation modeling technology, an aircraft structure assembly deformation simulation dataset is constructed.

[0035] Based on the aircraft structure assembly deformation dataset, a mapping relationship model is constructed to characterize the mapping relationship between control parameters and aircraft structure deformation field variable data, providing a simulation environment for the training of deep reinforcement learning models.

[0036] like Figure 2 As shown, the mapping relationship model includes two stages. In the first stage, a convolutional neural network is used to compress high-dimensional field variable data into a low-dimensional representation. By reducing the dimension of the original high-dimensional field variable and using the low-dimensional representation of the field variable as the agent state of the reinforcement learning model, the training difficulty and instability caused by the high-dimensional state space are alleviated. The formula of the first stage is as follows: ; ;in, represents the encoder in the convolutional neural network that compresses high-dimensional field variable data into low-dimensional data, is the original high-dimensional field variable data, is the low-dimensional representation of the field variable obtained by compression, The decoder in the convolutional neural network that restores the low-dimensional representation of the field variable data to the original high-dimensional data is represented. The convolutional neural network uses the reconstruction error as the loss function and completes the training of the convolutional neural network by minimizing the loss function value. The loss function formula is as follows: In the second stage, a multi-layer perceptron is used to complete the mapping between the input variables and the potential representation of the field variables. The formula is as follows: ;in, The multi-layer perceptron used in the second stage, is the input control parameter.

[0037] The low-dimensional representation generated by the convolutional neural network is used as the agent state, and the process control parameters are used as actions. A corresponding agent policy network is constructed based on the deep reinforcement learning framework, and the interaction results between the agent and the mapping relationship model are used as training data. The agent policy network and value network are trained to update the parameter weights to obtain a deformation control strategy model for predicting the process control parameters at the next moment.

[0038] More specifically, a deep reinforcement learning model for aircraft structure deformation control is constructed using the assembly deformation mapping relationship model as the simulation environment. The actions of the intelligent agent in the reinforcement learning model are the aircraft structure deformation process control parameters, and an experience buffer area is set; the current state and execution action of the intelligent agent are set, and a reward function for the intelligent agent is set; the reward function is designed according to the degree of optimization of the field variables, and measures whether the current strategy helps to improve the distribution of the aircraft structure field variables.

[0039] The reward of the agent is defined as the degree of improvement of the deformation index, which is obtained by inputting the state of the agent at time t into the decoder of the first stage of the mapping relationship model, specifically as follows: the current observation value of the agent observing the aircraft structure deformation simulation environment based on the mapping relationship model is obtained; the agent state is represented by the latent variables in the mapping relationship model, which can be obtained by inputting the action taken by the agent into the multi-layer perceptron of the second stage of the mapping relationship model. The formula of the agent state is as follows ;in, and Represent the state and action of the agent at time t, The multi-layer perceptron in the second stage of the deformation mapping relationship model.

[0040] Assume that the initial field variable distribution is , then the relationship between the action of the intelligent agent, i.e., the assembly deformation control parameters and the initial field variables can be expressed by the following formula: ;in, It is the decoder in the first stage convolutional neural network that maps low-dimensional representations back to high-dimensional field variables.

[0041] Since the goal of agent learning is to optimize the distribution of field variables by changing actions, the objective function can be expressed as: ;in, It is an evaluation function to measure the distribution of field variables. The specific formula is as follows: ; Since the variation range of deformation control parameters in actual assembly is small, the optimization objective can be approximated as: ; The reward function can be set as the field variable deformation index at time t and t+1 The tolerance is as follows: ;in, and are the states of the agent at time t and t+1 respectively, and are the distribution of field variables at time t and t+1 respectively, and the formula is as follows: ; ; Execute the action in the aircraft structure deformation simulation environment based on the deformation mapping relationship model to obtain the reward of the intelligent agent and the observation value at the next moment, and store the interaction experience between the intelligent agent and the environment in the experience buffer; wherein the interaction experience includes the current observation value, the action, the reward value and the observation value at the next moment; randomly sample a group of the interaction experiences in the experience buffer as samples to train the deep reinforcement learning model, and evaluate the performance of the deformation control model after training; wherein, randomly sample a group of the interaction experiences in the experience buffer as samples to train the deformation control reinforcement learning model, specifically as follows:

[0042] Iteratively executing the updating step until the number of iterations of the deformation control model reaches a preset number of training times and the experience buffer area is cleared;

[0043] The updating step uses proximal policy optimization to train the policy network and value network in the reinforcement learning model, including: randomly sampling a set of the interaction experiences from the experience buffer, and determining a target value function estimate based on the sampled interaction experiences using the deformation control model; wherein the target value function estimate is formulated as follows: ;in, is a parameter between 0 and 1 that controls the depth of the discount, that is, how much weight is placed on future rewards when evaluating the value of the current state. is the cumulative return starting from time t and considering the next n steps, represents an upper bound on the number of future steps considered; the formula for the cumulative reward is as follows: ;in, It's from time Start looking forward The time range of the step, is a discount factor used to reduce reliance on distant rewards, It's in time Rewards received.

[0044] The agent's own state The value network is input to obtain a current value function estimate; a loss value of the value network is determined based on the target value function estimate and the current value function estimate, and the parameters of the value network are updated based on the loss value; wherein the loss function formula of the value network is as follows: ;in, Is the state under the current strategy and parameters The current value function estimate of is the total number of samples, used to normalize the loss function; a group of the interaction experiences is randomly sampled from the experience buffer, and the loss value of the strategy network is determined using the deformation control model based on the sampled interaction experiences; wherein the formula of the loss function of the strategy part is as follows: ;in, Indicates the current policy parameters, is the strategy probability ratio , the new strategy Compared with the old strategy The probability ratio, is the advantage function estimate, used to evaluate the Take action Advantages over average action, Is a clipping function that reduces the policy probability ratio Restricted to Avoid large changes in the policy update step to maintain the stability of the learning process. It is a hyperparameter that controls the strictness of clipping and is usually taken as a small value.

[0045] The formula for estimating the advantage function using generalized advantage is as follows: ;in, , is the reward at time t, is the value function estimate, is the discount factor, is a trade-off factor that weighs the bias and variance of the estimate.

[0046] The simulation environment based on the mapping relationship model and the hyperparameters of the proximal policy optimization algorithm are analyzed. The optimal hyperparameter combination is selected by designing orthogonal experiments to optimize the training process of the deformation control strategy network. The training process of the optimized intelligent agent is as follows: Figure 3 As shown in the figure, after 600,000 steps of interaction with the environment constructed by the deformation mapping relationship model, the average reward return of the agent stabilized at a high level, indicating that the training of the optimization strategy has converged.

[0047] like Figure 4 As shown, the agent's deformation optimization strategy was validated on a test set of the fuselage structure deformation dataset. Initial stress field data for the aircraft structure in its current state was input into the first-stage network of the deformation mapping model to obtain a low-dimensional representation of the stress field. This representation was then used as the agent's state and fed into the policy network within the deformation control reinforcement learning model. Control parameters were then generated through policy network inference, achieving aircraft structure deformation control and optimization. Experimental results showed that the assembly deformation index of the test set was reduced by an average of 90%, with 95% of the test samples experiencing a reduction of more than 76%.

[0048] In addition, the terms "upper", "lower", "inner", "outer", "front", and "back" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0049] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. Any equivalent changes or modifications made based on the structure, features and principles described in the scope of the patent application of the present invention should be included in the scope of the patent application of the present invention.

[0050] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A real-time control method for aircraft structure assembly deformation based on deep reinforcement learning, characterized in that: The following steps are involved: Conduct simulation experiments based on aircraft geometry and material parameters to obtain aircraft structural deformation field variable data corresponding to different process control parameters, and construct a corresponding mapping relationship model based on the obtained process control parameters and the corresponding aircraft structural deformation field variable data; The mapping relationship model includes a first stage and a second stage. The first stage includes a convolutional neural network, which is used to convert aircraft structure deformation field variable data between high-dimensional representation and low-dimensional representation. The formula of the first stage is as follows: ; ;in, represents the encoder in the convolutional neural network that compresses high-dimensional field variable data into low-dimensional data, is the original high-dimensional field variable data, is the low-dimensional representation of the field variable obtained by compression, The decoder in the convolutional neural network that restores the low-dimensional representation of the field variable data to the original high-dimensional data is represented. The convolutional neural network uses the reconstruction error as the loss function and completes the training of the convolutional neural network by minimizing the loss function value. The loss function formula is as follows: The second stage includes a multi-layer perceptron, which is used to construct a mapping between process control parameters and low-dimensional representations of aircraft structure deformation field variable data; its formula is as follows: ;in, The multi-layer perceptron used in the second stage, The low-dimensional representation generated by the convolutional neural network is used as the agent state, and the process control parameters are used as actions. A corresponding agent strategy network and value network are constructed based on a deep reinforcement learning framework. The interaction results between the agent and the mapping relationship model are used as training data. The agent strategy network and value network are trained to update the parameter weights to obtain a deformation control strategy model for predicting the process control parameters at the next moment. When aircraft structure deformation control is performed at the assembly site, the aircraft structure deformation field variable data at the current moment is input into the deformation control strategy model to output the process control parameters at the next moment.

2. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 1 is characterized in that: The specific process of the simulation experiment is as follows: According to the digital tooling used in the aircraft structure assembly process, the process control parameters and actuation range of the digital tooling are determined, and the process control parameters are sampled as the boundary conditions of the model to obtain the aircraft structure deformation field variable data corresponding to different sampling points.

3. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 1 is characterized in that: The convolutional neural network uses reconstruction error as a loss function and is trained by minimizing the loss function value.

4. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 1 is characterized in that: During the training process, proximal strategy optimization is used to train the agent policy network and value network, and orthogonal experiments are designed to select the optimal hyperparameter combination of reinforcement learning. The agent policy network and value network are trained based on the optimal hyperparameter combination.

5. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 4 is characterized in that: The training process of the proximal strategy optimization includes a strategy part and a value part. The loss function of the strategy part is expressed as follows: ;in, Indicates the current policy parameters, is the strategy probability ratio , is the advantage function estimate, is a clipping function that converts the policy probability ratio Restricted to The loss function of the value part is expressed as follows: ;in, Is the state under the current strategy and parameters The current value function estimate of is the total number of samples, Estimate the target value function.

6. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 5 is characterized in that: The expression of the advantage function estimation is as follows: ;in, ; is the reward at time t, is the value function estimate, is the discount factor, is a trade-off factor.

7. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 1 is characterized in that: In the deep reinforcement learning framework, the agent's reward function is designed based on the degree of optimization of the aircraft structure deformation field variable data in the assembly scenario to measure whether the current strategy helps to improve the aircraft structure field variable distribution.

8. The real-time control method for aircraft structure assembly deformation based on deep reinforcement learning according to claim 7 is characterized in that: The reward function is expressed as follows: ;in, is the initial field variable distribution, is the second stage multi-layer perceptron, is the decoder in the first stage convolutional neural network that maps the low-dimensional representation back to the high-dimensional field variable, It is an indicator function to measure the distribution of field variables.

Images