Twin modeling method, device and equipment for modular object production line

Abstract

The invention relates to a twin modeling method for a modular object production line, and the method comprises the steps: obtaining a characteristic parameter of each component of a modular object as a first input, and obtaining an environment parameter of a target working environment of the modular object as a second input; creating an initial virtual contour of the modular object based on at least a portion of the first input; importing the first input, the second input and the initial virtual contour into a simulation system; in the simulation system, training the first input, the second input and the initial virtual contour by using a model-free reinforcement learning model, and automatically adjusting the first input, the second input and the initial virtual contour at the same time; at least one of a first output associated with the first input, a second output associated with the second input, and a finished virtual contour associated with the initial virtual contour is calculated. The method supports self-creation of a modular object model, also supports modification of existing entity modular objects, and realizes simulation similar to the real world.

Description

Technical Field

[0001] The present invention relates to the technical field of computer simulation, and in particular to a twin modeling method, device and equipment for a modular object production line. Background Art

[0002] At present, a large number of physical manufacturing industries tend to adopt the design and manufacturing process of modular products based on considerations such as product updates, cost control, and corresponding production line modifications. On this basis, in order to further facilitate design, modification, and even testing, the corresponding digital twin system, that is, digitizing the physical modular products and reproducing them in the virtual world / system at a 1:1 ratio, has gradually become popular. At present, the digital twin system of modular objects (for example, robots, vehicle chassis, etc.) usually collects data from a ready-made object or a fixed production line and simulates and reproduces it. It can be seen that the application of digital twins is generally limited to fixed objects or fixed fields, and the scope of application is relatively narrow, which is not conducive to structural innovation. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, according to a first aspect of the present invention, a twin modeling method for a modular object production line is provided, which comprises the following steps:

[0004] - obtaining characteristic parameters of each component of the modular object as a first input, and obtaining environmental parameters of a target working environment of the modular object as a second input;

[0005] - creating an initial virtual outline of the modular object based on at least a portion of the first input;

[0006] - importing the first input, the second input and the initial virtual contour into the simulation system;

[0007] -In the simulation system, the first input, the second input, and the initial virtual contour are trained and automatically adjusted simultaneously using a model-free reinforcement learning model;

[0008] - calculating at least one of a first output associated with the first input, a second output associated with the second input, and a finished virtual profile associated with the initial virtual profile.

[0009] In one embodiment, the model-free reinforcement learning model is in the form of a deep Q-network, where the Q function in the deep Q-network is:

[0010]

[0011] Among them, α represents the learning rate, γ represents the discount factor of the Markov reward process, s and S ′ Represent the current and next environment states, a and a respectively.′ Represent the current and next executable operations respectively, and r represents the reward obtained in the current state.

[0012] In another embodiment, the Q function is updated by:

[0013]

[0014] Where ω represents the loss function and N represents the total number of states / executions.

[0015] In yet another embodiment, the twin modeling method of the modular object production line further comprises:

[0016] -Select any executable operation with probability ∈ as the exploration operation;

[0017] - Select the executable action with the maximum expected reward estimate from past experience as the exploitation action with probability 1-∈;

[0018] - Balance exploration and exploitation to avoid local optima t :

[0019]

[0020] in, Represents the set of all operations.

[0021] In yet another embodiment, the probability ∈ is set to decay over time:

[0022]

[0023] Wherein, t represents time, and its value range is t>0.

[0024] In another embodiment, the first input includes at least one of geometric characteristic parameters, material characteristic parameters, mass characteristic parameters, connection methods, constraints, actuator types, output characteristic parameters and dynamic characteristic parameters; the second input includes at least one or more of gravity environment parameters, friction environment parameters, collision environment parameters and vibration environment parameters.

[0025] In yet another embodiment, the simulation system has dynamics calculation and kinematics calculation, and the dynamics calculation and kinematics calculation are based on an ODE physics engine.

[0026] According to a second aspect of the present invention, there is provided a twin modeling device for a modular object production line, comprising:

[0027] An input module configured to obtain characteristic parameters of each component of the modular object as a first input, and to obtain environmental parameters of a target working environment of the modular object as a second input;

[0028] a creation module configured to create an initial virtual outline of the modular object based on at least a portion of the first input;

[0029] an import module configured to import the first input, the second input, and the initial virtual profile into the simulation system;

[0030] A training module is configured to train the first input, the second input, and the initial virtual profile using a model-free reinforcement learning model in a simulation system and automatically adjust the first input, the second input, and the initial virtual profile simultaneously;

[0031] A calculation module is configured to calculate at least one of a first output associated with the first input, a second output associated with the second input, and a finished virtual profile associated with the initial virtual profile.

[0032] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0033] a memory storing non-transitory computer-readable instructions;

[0034] At least one processor is configured to, when executing non-transitory computer-readable instructions, perform at least one step of the twin modeling method for a modular object production line according to the first aspect of the present invention.

[0035] The present invention supports the creation of modular object models and the modification of existing physical modular objects, thus achieving simulation processes and results that are comparable to the real world. With the help of the present invention, manufacturers in various industrial fields can use virtual components to complete the structural construction and high-precision testing of modular objects in advance, and then use hardware components to complete the mass production of physical modular objects, which greatly improves the testing efficiency of each version of the solution and reduces the cost of testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings, wherein:

[0037] Figure 1 A flowchart of a twin modeling method for a modular object production line according to an embodiment of the present invention is shown.

[0038] Figure 2 A flowchart of solving the problem using a deep Q network according to an embodiment of the present invention is shown.

[0039] Figure 3A flowchart of constructing a twin model of a modular vehicle chassis according to another embodiment of the present invention is shown.

[0040] Figure 4 A schematic diagram of a twin modeling device for a modular object production line according to yet another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0041] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention.

[0042] Embodiment 1

[0043] In this embodiment, the modular object is a modular robot. The modular robot hardware components and virtual components are respectively composed of various robot modules, wherein the various robot modules include structural parts, drive devices, control devices, sensors, etc. These modules use universal matching interfaces and can be quickly assembled into various common types of robots, such as mechanical arms, mobile robots, multi-legged robots, etc., thereby constructing a robot working environment.

[0044] Hardware components include various types of physical hardware, which are used in real scenarios; virtual components are composed of various digital models, which are reproduced in virtual simulation systems. Hardware components and virtual components have the same physical and electrical properties, and together constitute a digital twin. Users can use the reinforcement learning system to expand the hardware / software component system. The reinforcement learning system will collect the characteristic parameters of the newly added hardware components during operation, analyze the parameters, and correct the parameter calibration of the digital model corresponding to the newly added components, thereby establishing a new digital twin.

[0045] Hardware components include various common robot modules and other auxiliary modules, which can be freely combined to the greatest extent. At the module type level, robot modules and other auxiliary modules can include structural modules, actuator modules, control modules, and sensor modules. Virtual components are digital models of hardware components. The module type of virtual components is the same as that of hardware components, and consists of two parts: a three-dimensional model and simulation parameters. The three-dimensional model is modeled by the modules in the hardware components in a 1:1 manner, defining the appearance and collision boundaries of the modules. The simulation parameters determine the physical and electrical properties of the modules in the simulation system, including mass, moment of inertia, friction coefficient, collision coefficient, damping, stiffness, and so on.

[0046] The virtual simulation system is mainly used to simulate the virtual robot built by the user through virtual components and perform motion simulation. The dynamics and kinematics calculations of the simulation system are based on the open source physics engine (Open Dynamics Engine, ODE). According to the parameters of each module in the virtual component, the physical processes such as force, collision, and constraint are calculated with high precision and in real time. At the same time, ODE can also simulate the process of motor output power and various sensor data collection. The simulation system can adopt the BS (Browser / Server Architecture) architecture, in which the server side is used for simulation calculation, and the browser side is used to present the simulation graphics and real-time presentation of simulation data. Users can write control programs in the virtual simulation system to control the robot in the simulation system, and the written programs can also be copied to the hardware module for direct operation.

[0047] The reinforcement learning system is a model-free reinforcement learning model, which collects the operating characteristics of the hardware model and inputs them into the model-free reinforcement learning model. Based on the intelligent parameter adjustment function of the deep reinforcement learning model, the hardware components and virtual components are constructed as digital twins, so that the virtual components have the same physical and electrical properties as the hardware components. That is, when the system input is the same, the system output is always consistent with it.

[0048] The process of building a virtual component using a reinforcement learning system is:

[0049] -The user completes the 1:1 3D model construction of the virtual component by measuring and disassembling the hardware components. At this point, the physical and electrical parameters of the virtual component are to be determined;

[0050] -Use sensor components to collect various data during the operation of the robot and send them to the control system or storage system, including acquisition frequency, measurement accuracy, noise, data structure, communication form, etc.; at the same time, read the characteristic parameters of the actuator, including the motor and the associated transmission device, especially its power, speed, force / torque, time constant, etc.;

[0051] The first input includes: geometric characteristic parameters such as the length, width, height and shape of the structural module; material characteristic parameters such as the purpose, elastic modulus and Poisson's ratio of the structural module; mass characteristic parameters such as the total mass, center position and inertia tensor of the structural module; connection mode and constraint conditions refer to the connection mode between the structural modules and the corresponding constraint conditions; actuator types include rotary motors or linear motors; output characteristic parameters such as the output force, torque, speed and frequency of the actuator; and dynamic characteristic parameters such as the response time or bandwidth of the actuator;

[0052] The second input includes: gravity environment parameters, i.e., the magnitude and direction of gravity acceleration; friction environment parameters, i.e., the friction coefficient between the contact surfaces of each module; collision environment parameters, i.e., the elastic and inelastic characteristics of the collision; vibration environment parameters, i.e., the natural vibration frequency and damping characteristics of the system; and natural environment parameters, i.e., the airflow, humidity, temperature and other natural environment parameters of the environment;

[0053] -Use the Deep Q-Network (DQN) model-free reinforcement learning model to train the above parameters and complete the construction of the digital twin model;

[0054] Among them, state s is used to describe the current situation of the environment; operation a represents the operation that the robot can perform; reward r represents the value of the environment feedback after performing the action, which is a reward function used to guide learning; and strategy p represents the rule that determines whether the robot takes a specific operation a in a certain state s; Q represents the expected cumulative reward after taking operation a in state s. On this basis, in order to learn the optimal strategy by continuously updating the Q value, the Q function of reinforcement learning based on the temporal difference algorithm is:

[0055]

[0056] Among them, α represents the learning rate, γ represents the discount factor of the Markov reward process, s and S ′ Represent the current and next environment states, a and a respectively. ′ Represent the current and next executable operations respectively, and r represents the reward obtained in the current state. Preferably, the value range of γ is [0,1), and particularly preferably, its value is 0.95; the value of the learning rate α is 0.001;

[0057] Then, a deep neural network is used to fit the value of the above Q function. Commonly used neural network structures include multi-layer perceptron (MLP) and convolutional neural network (CNN), and the parameters of these neural networks are usually optimized through training. Moreover, the output size of these neural networks is comparable to the dimension of the operation space.

[0058] DQN can perform model updates by minimizing the mean square error loss of the Q function:

[0059]

[0060] Where ω represents the loss function and N represents the total number of states / executions.

[0061] In addition, DQN introduces an experience replay mechanism, which stores past experiences in a buffer, for example, the state transitions (s, a, r, S) experienced by the robot. ′) is stored in a specific memory bank, and samples are randomly extracted during the training process, thereby improving the utilization of data, the stability of training, and reducing the correlation between samples. Specifically, the introduction of the experience replay mechanism can make the samples meet the independence assumption. The data obtained by interactive sampling in the Markov decision process (MDP) does not originally meet the independence assumption, because the state s at the current moment and the state S at the next moment are different. ′ The non-IID data has a great impact on the training of neural networks, which will make the neural network fit the most recent training data. Experience replay breaks / reduces the correlation between samples so that they meet the independence assumption. On the other hand, each sample can be reused, which is suitable for gradient learning of deep neural networks.

[0062] To further stabilize the training process, DQN uses two neural networks with the same structure but different parameters: one for predicting Q values, the so-called main network, and the other for calculating the target Q values, the so-called target network. The parameters of the target network are updated regularly, which helps reduce instability during training.

[0063] In the training process, in order to save training time as much as possible while expecting to obtain the optimal solution for the training data, a greedy algorithm is introduced. Therefore, the twin modeling method of the modular robot production line of this embodiment further includes:

[0064] -Select any executable operation with probability ∈ as the exploration operation;

[0065] - Select the executable action with the maximum expected reward estimate from past experience as the exploitation action with probability 1-∈;

[0066] - Balance the exploration operation and the exploitation operation to avoid local optimal operation a t :

[0067]

[0068] in, Represents the set of all operations.

[0069] In some embodiments, the probability ∈ is set to decay over time t:

[0070] ∈_t=1 / t

[0071] Wherein, t represents time, and its value range is t>0.

[0072] Reference Figure 1-2 The twin modeling method of a modular object (especially a robot) production line provided in this embodiment includes:

[0073] - obtaining characteristic parameters of each component of the modular robot as the first input, and obtaining environmental parameters of the target working environment of the modular robot as the second input;

[0074] - creating an initial virtual outline of the modular robot based on at least part of the first input, in particular geometric characteristic parameters, material characteristic parameters, mass characteristic parameters, connection methods and constraints, etc.;

[0075] - importing the first input, the second input and the initial virtual profile into a simulation system, such as a BS framework based on ODE;

[0076] -In the simulation system, the first input, the second input, and the initial virtual contour are trained and automatically adjusted using the DQN model-free reinforcement learning model;

[0077] - calculating at least one of a first output associated with the first input, a second output associated with the second input, and a finished virtual profile associated with the initial virtual profile.

[0078] It is conceivable that the first output, the second output and the finished virtual outline are the desired training data, which can in turn be used on the physical modular product.

[0079] Embodiment 2

[0080] In this embodiment, the modular object is a modular vehicle chassis. On this basis, the first input that can be obtained on the physical hardware includes the shape, weight and moment of inertia of each component of the modular vehicle chassis, the output speed and torque of the actuator, and the second input that can be obtained includes gravity environment parameters. These obtainable parameters are determined parameters. Relatively speaking, the training data expected to be obtained by the twin modeling method of this embodiment include the damping and dynamic and static friction coefficient of the shaft, the dynamic and static friction coefficient and collision coefficient of the wheel and the ground. These expected parameters are uncertain parameters.

[0081] Reference Figure 3 The twin modeling method of the modular object (especially, vehicle chassis) production line provided in this embodiment includes:

[0082] -Install an encoder for the physical vehicle chassis to control the speed and output torque of the motor and collect various data when the vehicle chassis is running, such as the speed, acceleration, position of the chassis, the speed and torque of the motor;

[0083] -Use the reinforcement learning system to solve the relationship between the state s and the dynamic operation a of the digital / virtual twin model in the simulation system. This relationship represents the optimal control strategy of the digital / virtual twin model, so that the motion performance of the digital / virtual twin model is consistent with that of the physical vehicle chassis. The specific solution process can be described as:

[0084] S1. Add and sample data in the experience replay pool; the data part needs to sample valuable data, for example, under the normal working condition of the vehicle chassis, set multiple sets of different (first / second) input values ​​to obtain different states s, such as data under different conditions such as straight, backward, and turning;

[0085] S2. Define the number of layers N of the Q network, which is set according to the number of unknown parameters Num. The number of layers N = lnNum, 32 neurons are set in each layer, and the ReLU activation function is used, that is, f(x) = max(0,x), so that when the input value x is greater than 0, the ReLU function remains x; when the input value x is less than or equal to 0, the ReLU activation function outputs 0;

[0086] S3, initialization parameters, Q(s,a), learning rate α = 0.001, discount factor γ = 0.95; the final mean square error threshold is [0, 1]; the number of data groups buffer_size = 1000, the number of data in each group batch_size is 64, the probability of the greedy algorithm ∈ = 1 / t; the target network is updated every 10 times;

[0087] S4, select operation: according to the current state s, use the greedy algorithm strategy to select the operation;

[0088] S5. Perform actions in the simulation environment and observe rewards: take action a, interact with the environment, and observe the next state

[0089] S ′ and immediate return r;

[0090] S6, storage experience: (s, a, r, S ′ ) is stored in the experience replay pool;

[0091] S7, randomly sample a batch of data from the experience replay pool;

[0092] S8. Calculate the target Q value: Use the target network to calculate the target Q value, that is,

[0093] S9, update the main network: update the model parameters according to the loss function L;

[0094] S10, update target network: regularly update the parameters of the target network, and the update cycle set here is every 10 steps;

[0095] S11, repeat steps S2-S8 until the termination condition is reached, that is, the mean square error threshold is [0, 1].

[0096] Embodiment 3

[0097] Reference Figure 4 , which shows a schematic diagram of a twin modeling device for a modular object production line according to the present invention. In this embodiment, a twin modeling device for a modular object production line is provided, which includes:

[0098] An input module 100 configured to obtain characteristic parameters of each component of the modular object as a first input, and to obtain environmental parameters of a target working environment of the modular object as a second input;

[0099] A creation module 200 configured to create an initial virtual outline of a modular object based on at least a portion of the first input;

[0100] An import module 300 configured to import the first input, the second input, and the initial virtual profile into the simulation system;

[0101] A training module 400 is configured to train the first input, the second input, and the initial virtual profile using a model-free reinforcement learning model in a simulation system and automatically adjust the first input, the second input, and the initial virtual profile at the same time;

[0102] The calculation module 500 is configured to calculate at least one of a first output associated with the first input, a second output associated with the second input, and a finished virtual contour associated with the initial virtual contour.

[0103] Embodiment 4

[0104] The present application also provides an electronic device, which includes a memory storing non-transitory computer-readable instructions; and also includes at least one processor configured to, when executing the non-transitory computer-readable instructions, perform at least one step of the twin modeling method of the modular object production line described in the first aspect of the present invention.

[0105] The technical solutions provided by the embodiments of the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the embodiments of the present invention. The description of the above embodiments is only applicable to help understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, according to the embodiments of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A twin modeling method for a modular object production line, characterized in that: The following steps are involved: - obtaining characteristic parameters of each component of the modular object as a first input, and obtaining environmental parameters of a target working environment of the modular object as a second input; - creating an initial virtual outline of the modular object based on at least a part of the first input; - importing the first input, the second input and the initial virtual profile into a simulation system; - in the simulation system, using a model-free reinforcement learning model to train the first input, the second input, and the initial virtual profile and automatically adjust them simultaneously; - calculating at least one of a first output associated with the first input, a second output associated with the second input, and a finished virtual profile associated with the initial virtual profile.

2. The twin modeling method of modular object production line according to claim 1, characterized in that: The model-free reinforcement learning model is in the form of a deep Q network, where the Q function in the deep Q network is: Among them, α represents the learning rate, γ represents the discount factor of the Markov reward process, s and S ′ Represent the current and next environment states, a and a respectively. ′ Represent the current and next executable operations respectively, and r represents the reward obtained in the current state.

3. The twin modeling method of modular object production line according to claim 2 is characterized in that: The Q function is updated in the following way: Where ω represents the loss function and N represents the total number of states / executions.

4. The twin modeling method of modular object production line according to claim 3 is characterized in that: It further includes: -Select any executable operation with probability ∈ as the exploration operation; - Select the executable action with the maximum expected reward estimate from past experience as the exploitation action with probability 1-∈; - Balance the exploration operation and the exploitation operation to avoid local optimal operation a t : in, Represents the set of all operations.

5. The twin modeling method of modular object production line according to claim 4, characterized in that: The probability ∈ is set to decay over time t: Wherein, t represents time, and its value range is t>0.

6. The twin modeling method of modular object production line according to claim 1, characterized in that: The first input includes at least one of geometric characteristic parameters, material characteristic parameters, mass characteristic parameters, connection methods, constraints, actuator types, output characteristic parameters and dynamic characteristic parameters; the second input includes at least one or more of gravity environment parameters, friction environment parameters, collision environment parameters and vibration environment parameters.

7. The twin modeling method of modular object production line according to claim 6, characterized in that: The simulation system has dynamics calculation and kinematics calculation, and the dynamics calculation and kinematics calculation are based on ODE physics engine.

8. The twin modeling method of modular object production line according to claim 7, characterized in that: The simulation system is compatible with the ROS2 operating system.

9. A twin modeling device for a modular object production line, characterized in that: include: An input module configured to obtain characteristic parameters of each component of the modular object as a first input, and to obtain environmental parameters of a target working environment of the modular object as a second input; a creation module configured to create an initial virtual outline of the modular object based on at least a portion of the first input; an import module configured to import the first input, the second input and the initial virtual profile into a simulation system; A training module configured to train the first input, the second input, and the initial virtual profile using a model-free reinforcement learning model in the simulation system and automatically adjust the first input, the second input, and the initial virtual profile at the same time; A calculation module is configured to calculate at least one of a first output associated with the first input, a second output associated with the second input, and a finished virtual profile associated with the initial virtual profile.

10. An electronic device, characterized in that: include: a memory storing non-transitory computer-readable instructions; At least one processor is configured to, when executing the non-transitory computer-readable instructions, perform at least one step of the twin modeling method for a modular object production line according to any one of claims 1-8.

Images