Method for manipulating physical objects by means of a machine, and machine controller
The method employs a graphical neural network to coordinate control agents based on object neighborhoods, addressing the challenges of central and decentralized control in complex systems by optimizing communication and resource allocation, resulting in enhanced machine efficiency and performance.
Patent Information
- Application Number
- PCT/EP2024/060554
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-23
- Filing Date
- 2024-04-18
- Publication Date
- 2024-11-28
Smart Images

Figure EP2024060554_28112024_PF_FP_ABST
Abstract
Description
[0001] Description
[0002] Method for manipulating physical objects by a machine and machine control
[0003] Data-driven machine learning methods are increasingly being used to control complex machines, such as robots, logistics systems, manufacturing plants, or machine tools. In particular, artificial neural networks are trained using reinforcement learning methods to generate a state-specific control action for controlling the machine for a given state of the machine, thereby optimizing its performance. Such a control agent optimized for controlling a machine is often referred to as a policy or, for short, an agent.
[0004] In many cases, multiple components of a machine must be controlled in a coordinated manner in order to manipulate a large number of objects, e.g., in a parcel sorting system. For this purpose, each object can be assigned a control agent that controls the manipulation of that object. The control agents should then, if possible, interact in such a way that the global, i.e., cross-object, performance of the machine, e.g., object throughput, resource consumption, or wear, is optimized.
[0005] Known methods for optimizing cooperating control agents include so-called multi-agent reinforcement learning (MARL) methods. In the first variant, the control agents are controlled by a central controller. However, this often leads to a significant increase in the dimensions of the state spaces or action spaces to be considered. In a second variant, however, the control agents act largely independently or decentrally. In this variant, however, a lack of coordination between different control agents can lead to global performance losses.
[0006] Further methods for coordinating control agents can be found in the publication "Graph Neural Networks for Decentralized Multi-Robot Path Planning" by Qingbiao Li, Fernando Gama, Alejandro Ribeiro, and Amanda Prorok at https: / / arxiv.org / pdf / 1912.06095.pdf (accessed April 25, 2023) and the publication "Graph Policy Gradients for Large Scale Robot Control" by Arbaaz Khan, Ekaterina Tolstaya, Alejandro Ribeiro, and Vijay Kumar at https: / / arxiv.org / pdf / 1907.03822.pdf (accessed April 25, 2023).
[0007] It is an object of the present invention to provide a method for manipulating physical objects by a machine and a machine control for controlling the machine, which allow better coordinated manipulations of the objects and / or more efficient control of the machine.
[0008] This object is achieved by a method having the features of patent claim 1, by a machine control having the features of patent claim 13, by a computer program product having the features of patent claim 14 and by a computer-readable storage medium having the features of patent claim 15.
[0009] In order for a machine to manipulate physical objects, a large number of objects are fed to the machine, either in real life or in dynamic simulation. The manipulation can in particular comprise machining, processing, positioning or steering the objects. The objects can be, for example, workpieces, components, piece goods or process goods. Furthermore, sensors or simulations are used to continuously detect which of the objects are currently adjacent to one another according to a predetermined proximity criterion. In addition, a control agent for controlling the machine is assigned to each object, and the current status data of the respective object is fed into this control agent. Furthermore, neurons of a neural network are continuously linked according to the currently detected proximity of the objects. The neural network controls the transmission of transfer data sets between the control agents.Furthermore, the control agents control the manipulation of the objects, either in real life or in simulation, based on the input state data and the transmitted transfer data sets, and the corresponding performance of the machine is determined. This trains the neural network to optimize performance. Finally, the machine is controlled by means of the trained neural network and the control agents to manipulate the physical objects.
[0010] To carry out the method according to the invention, a corresponding machine control, a computer program product and a computer-readable, preferably non-volatile storage medium are provided.
[0011] The method according to the invention and the machine control according to the invention can be carried out or implemented, for example, by means of one or more computers, processors, application-specific integrated circuits (AS IC), digital signal processors (DSP) and / or so-called "field programmable gate arrays" (FPGA). Furthermore, the method according to the invention can be carried out at least partially in a cloud and / or in an edge computing environment.
[0012] A particular advantage of the invention is that communication between the control agents can be optimized with regard to global machine performance by training the neural network. In this way, the control agents can be coordinated more efficiently, the objects can be manipulated more efficiently, and ultimately the machine can be operated more efficiently. Central control of the control agents is not necessary in many cases. Furthermore, a method according to the invention can also be used to efficiently coordinate control agents for a changing number of objects.
[0013] A further advantage of the invention is that by detecting object neighborhoods and linking neurons of the neural network accordingly, communication between the control agents and / or optimization of this communication can be restricted to objects whose mutual influence is relatively likely.
[0014] Advantageous embodiments and further developments of the invention are specified in the dependent claims.
[0015] According to an advantageous embodiment of the invention, the control agents can be trained together with the neural network to optimize performance. In this way, learning-based control agents can also be used whose control capabilities are optimized in cooperation with a communication controller of the neural network. Furthermore, an instance of a common control agent can each be assigned to the objects. The common control agent can then be trained together with the neural network to optimize performance. This means that in many cases, control agent training can be limited to training a single common control agent. In this way, training effort can generally be considerably reduced.Furthermore, when the number of objects changes, a new object can simply be assigned another instance of the common, trained or to-be-trained control agent.
[0016] According to a further advantageous embodiment of the invention, during the transmission of the transfer data sets, the latter can be propagated by the neural network.
[0017] The transfer data sets can be transmitted, in particular, along links of the neural network. In this way, the neighborhood-specific links of the neural network can directly control which control agent transmits transfer data sets to which control agent. If the transfer data sets each contain multiple data elements, the neural network can also include data-element-specific links, to which data-element-specific neural weights may be assigned. This also allows the neural network to control which data elements of a transfer data set are transmitted.
[0018] According to a particularly advantageous embodiment of the invention, the neural network can be a graph neural network. The latter is often also referred to as a "graph neural network" or abbreviated to GNN. Properties, implementations, and training of such graph neural networks are described, for example, in the online encyclopedia Wikipedia at https: / / en.wikipedia.org / wiki / Graph_neural_network (accessed on April 26, 2023) and in the references cited therein. The content of the above Wikipedia page is hereby incorporated into this application. Graphic neural networks allow data transmission between neighboring nodes of a graph to be mapped to the graphic neural network in a natural manner. In a single evaluation step of a graphic neural network, the nodes of the graph are each updated with data received from neighboring nodes.By repeatedly evaluating the data, it is possible to propagate it successively across multiple, neighboring nodes. By training a graphical neural network, communication between nodes of a graph can be optimized according to predefined criteria. A variety of well-known optimization methods are available for performing such training.
[0019] According to a further advantageous embodiment of the invention, a distance measure can be used as the proximity criterion. In particular, a respective current distance between the objects can be determined, for example, using a camera, a proximity sensor, a field strength sensor, or by simulation. In this case, objects can be detected as currently adjacent to one another if their current distance is less than a predetermined maximum distance. Alternatively or additionally, a predetermined number of those objects that are currently closest to a respective object can also be detected as currently adjacent to it.
[0020] As a rule, manipulating objects at a relatively short distance requires more coordination effort than manipulating objects at a relatively large distance. By limiting communication and / or optimizing this communication to objects at a short distance, communication and optimization resources can be saved in many cases without significantly impairing the coordination of the control agents.
[0021] In addition, other proximity criteria may also be provided. Thus, objects can generally be recorded as adjacent if they can influence each other, if such influence appears likely, and / or if these objects interact indirectly. An example of the latter case are objects that require a common resource, such as workpieces that are to be machined by the same tool and therefore cannot be manipulated independently of one another.
[0022] According to a further advantageous embodiment of the invention, a dynamic graph can be generated from the currently detected neighborhoods, the nodes of which are each assigned to a control agent or an object. Nodes that are currently assigned to neighboring objects can be connected by an edge. Nodes connected by an edge are referred to below as neighboring nodes. Accordingly, an edge can be deleted if the objects assigned to the nodes connected by it are no longer neighboring. The neurons of the neural network, in particular of a graphic neural network, can then be linked according to the edges of the dynamic graph. A link structure of the dynamic graph, i.e. the connections of its nodes by edges, can be represented in particular by a dynamic, i.e. time-varying, adj acency matrix.Such an adj acency matrix is often also called an adj acency matrix or a neighborhood matrix . An adj acency matrix has a row and a column for each node . If an nth node is connected to an mth node by an edge, the nth row of the mth column of the adj acency matrix usually contains a non-vanishing value , e.g. a one , otherwise a zero .
[0023] In addition, status data of the assigned object, control-relevant data of the assigned control agent and / or transfer data records received from neighboring nodes of the respective node can be stored in a respective node. This allows the data relevant for manipulating the objects and / or for controlling the machine to be stored in a uniform manner and according to their transmission paths in the dynamic graph. If the data stored in a respective node comprises several data elements, a separate, data element-specific edge to a neighboring node of the respective node can be provided for each data element. In this way, data exchange between neighboring nodes can be controlled by the neural network in a data element-specific manner.
[0024] According to a further advantageous embodiment of the invention, a statistical distribution of control signal values can each be output by the control agents. The neural network and / or the control agents can then be trained based on the determined performance and on gradients of the statistical distributions. In this way, the neural network can be trained efficiently even in cases in which the determined performance cannot be differentiated, or can only be differentiated imprecisely, according to the learning parameters of the neural network to be optimized. In such a case, instead of gradients of the performance, the gradients of the statistical distributions according to the learning parameters and the determined performance can be used for a backpropagation process.
[0025] Preferably, a respective statistical distribution can be specified by a control signal value and its spread. In such a case, a respective gradient can be determined by deriving a Gaussian distribution with the respective control signal value as the mean and the respective spread as the standard deviation.
[0026] Alternatively or additionally, a gradient-free optimization method can be used to train the neural network and / or the control agents. In this case, it is not necessary to determine performance gradients. A genetic optimization method or particle swarm optimization, for example, can be used as a gradient-free optimization method.
[0027] Furthermore, a machine learning module can be trained to predict a resulting performance based on control signals from the control agents. The trained machine learning module can then be used to train the neural network and / or the control agents to predict the performance, with the training being carried out based on gradients of the predicted performance. In particular, the machine learning module can implement a so-called Q-learning method and / or include a so-called transition model.
[0028] An embodiment of the invention is explained in more detail below with reference to the drawing. Each of the drawings is a schematic representation of: Figure 1 shows a machine control system for controlling a machine for manipulating fed objects,
[0029] Figure 2 shows a parcel sorting system as an example of such a machine, and
[0030] Figure 3 a training of the machine control.
[0031] Insofar as the same or corresponding reference symbols are used in different figures, these reference symbols designate the same or corresponding entities which can be implemented or configured in particular as described in connection with the relevant figure.
[0032] Figure 1 illustrates a machine control CTL when controlling a machine M for manipulating physical objects 01, 02, ... supplied to the machine M. The manipulation can in particular comprise machining, processing, positioning or steering these objects 01, 02, ... .
[0033] The machine control CTL is shown in Figure 1 as external to the machine M and coupled to it. Alternatively, the machine control CTL can also be fully or partially integrated into the machine M. The machine control CTL has one or more processors for executing process steps of the machine control CTL as well as one or more memories coupled to the processor for storing the data to be processed by the machine control CTL.
[0034] The machine M can be, for example, a robot, a logistics system, a conveyor belt, a production plant, a machine tool, or a parcel sorting system. The machine M comprises, in particular, components for the controlled and individual manipulation of the supplied objects 01, 02, ..., for example, individually controllable conveyor belt segments, sorting elements, tools, or a gripper arm. The objects 01, 02, ... to be manipulated can be workpieces, components, piece goods, process goods, or packages and, in particular, can occur in a variable number over time.
[0035] For each object 01 or 02, ... currently supplied to machine M, the machine control system CTL generates an object-specific control agent Pl or P2, ... to control machine M and assigns it to the respective object 01 or 02, .... Each control agent Pl or P2, ... specifically controls those components of machine M through which the assigned object 01 or 02, ... can currently be manipulated. Each control agent Pl or P2, ... thus controls the manipulation of the object 01 or 02, ... assigned to it.
[0036] As an example of such a machine M, Figure 2 illustrates a parcel sorting system. The latter has a conveyor belt comprising individually controllable conveyor belt segments FS. For reasons of clarity, only four of the conveyor belt segments FS are explicitly identified by a reference symbol in Figure 2. By individually controlling each conveyor belt segment FS, its running speed can be adjusted individually and independently of the other conveyor belt segments FS.
[0037] Let us assume that several fed packages 01, ..., 05 are currently being transported as objects by the conveyor belt. The packages 01, ..., 05 can obviously be manipulated by specifically controlling the conveyor belt segments FS on which they are currently resting. Thus, packages 01, 05 resting on different FS can be brought closer to or separated from each other by varying the speeds of the respective conveyor belt segments FS, for example, to achieve a uniform package throughput. Furthermore, a package 01, ... or 05 resting on different conveyor belt segments FS can be rotated by temporarily operating the respective conveyor belt segments FS at different running speeds.
[0038] To the extent that manipulations of currently neighboring objects can influence each other, e.g., if two packages are currently resting on a shared conveyor belt segment FS, control agents assigned to neighboring objects should coordinate their control. For this purpose, data exchange between the control agents P1, P2, ... is provided. According to the invention, this data exchange is optimized through machine learning. A corresponding training of the machine control system CTL is explained in more detail below.
[0039] As Figure 1 further illustrates, the machine controller CTL has a graphic neural network GNN for controlling communication between different control agents Pl, P2, ....
[0040] In addition, a video camera C is connected to the machine control system CTL. This camera is used to detect objects 01, 02, ... fed to the machine M and to determine which of these objects are adjacent to each other. The video camera C is aimed at the machine M or at the objects 01, 02, ... fed to the machine M and continuously records current images of these objects 01, 02, ...
[0041] The images recorded by the video camera C are transmitted in the form of video data VD to an optical evaluation device OPT of the machine control system CTL. The optical evaluation device OPT continuously evaluates the video data using image recognition in order to detect objects currently fed to the machine M and manipulated by it, here O1, O2, .... In addition, the optical evaluation device OPT continuously determines the position and, if applicable, the size, orientation, and / or speed of each detected object O1 or O2, .... From this, the optical evaluation device OPT continuously deduces which of the detected objects O1, O2, ... are currently adjacent to one another.
[0042] In the present embodiment, a distance measure is used as the criterion for proximity. Based on the determined positions and, if applicable, the sizes and / or orientations of objects 01, 02, ..., the optical evaluation device OPT determines a respective distance between objects 01, 02, ... and compares this with a predetermined maximum distance. Objects whose distance is less than the maximum distance are considered adjacent, while objects with a greater distance are considered non-adjacent.
[0043] From the determined neighborhoods of objects 01, 02, ..., the optical evaluation device OPT continuously generates and / or updates a dynamic graph G. The latter is time-varying and contains, for each currently manipulable object 01 or 02, ..., a node assigned to this object, with precisely the nodes of neighboring objects being connected by an edge. Accordingly, an edge between two previously neighboring nodes is deleted as soon as the associated objects are no longer recognized as neighboring by the optical evaluation device OPT. The dynamic graph G thus encodes, in a sense, the current, sensor-detected neighborhoods of objects 01, 02, ..., especially when the number of objects changes.
[0044] Based on the dynamic graph G, the machine control system CTL continuously connects or breaks the connections between neurons of the graphic neural network (GNN) according to the neighborhoods encoded in the dynamic graph G. The graphic neural network (GNN) thus structurally maps the current nodes and edges of the dynamic graph G and thus the current neighborhoods of the objects O1, O2, ...
[0045] Furthermore, the optical evaluation device OPT continuously outputs the detected position and, if applicable, the size, orientation, or speed of each manipulable object 01, 02, ... in the form of an object-specific status data set SI or S2, ... Here, S1 is the status data set specific to object 01, S2 is the status data set specific to object 01, etc.
[0046] In addition, the machine M also continuously records current status data of the objects 01, 02, ... and / or status data of components of the machine M responsible for the objects 01, 02, ... in an object-specific manner.
[0047] The object-specific status data is also transmitted in the form of object-specific status data records from the machine M to the machine control CTL. For reasons of clarity, these object-specific status data records are also designated in Figure 1 with the reference symbols SI, S2, .... Here, too, S1 is the status data record specific for object 01, S2 is the status data record specific for object 01, etc. In the case of a parcel sorting system, in particular the current speeds of conveyor belt segments FS on which a respective object 01 or 02, ... is currently resting can be transmitted as the respective object-specific status data record SI or S2, ... from the machine M to the machine control CTL.
[0048] In general, such status data sets may include measurement data, sensor data, environmental data or other data arising during the operation of the machine M or influencing the operation, in particular data on actuator positions, forces occurring, power, pressure, temperature, valve positions, emissions and / or resource consumption of the machine M or one of its components.
[0049] In the present embodiment, the state data sets SI, S2, ... are preferably each quantified by a numerical state vector.
[0050] As soon as the optical evaluation device OPT detects a new object 01 or 02, ..., the machine control CTL is prompted to generate a new object-specific control agent Pl or P2, ... and assign it to the newly detected object 01 or 02, .... The new control agent Pl or P2, ... is generated as a new instance of a common control agent P.
[0051] In a previous step, the shared control agent P was trained using a reinforcement learning method together with the graphical neural network (GNN) to optimize the performance of the machine M. Such training is explained in more detail below. Preferably, the shared control agent P is implemented by an artificial neural network. An object-specific instance of the shared control agent P can be created, for example, by copying it along with its trained learning parameters and exclusively allocating an object-specific memory to the copy, i.e., the instance P1 or P2, .... Alternatively or additionally, the instances can correspond to various calls or evaluations of routines by means of which the shared control agent P is implemented.
[0052] In the present exemplary embodiment, the specific memory for a respective object O1 or O2, ... is created in the node of the graphical neural network GNN assigned to this object O1 or O2, ... The respective control agent P1 or P2, ... is then assigned to this node. In this node, state data of the respective object O1 or O2, ... and other data relevant for its manipulation are stored, updated, and / or retrieved by the assigned control agent P1 or P2, .... In this way, the state data and other control-relevant data, as well as their neighborhood relationships, are embedded in the graphical neural network GNN. Such embedding of feature data is often referred to as embedding in the field of machine learning.
[0053] To control the machine M, the object-specific state data sets SI, S2, ... are continuously fed to the control agent Pl or P2, ... responsible for the respective object O1 or O2, ... The respective control agent Pl or P2, ... stores the state data sets SI or S2, ... as object-specific feature data in the node of the graphical neural network GNN assigned to it.
[0054] Furthermore, to coordinate the control agents PI, P2, ..., transfer data sets TD are continuously transmitted between the control agents PI, P2, ... The transmission is controlled by the trained graphic neural network (GNN) in a performance-optimized manner. For this purpose, the feature data stored in the nodes of the graphic neural network (GNN) are propagated along its edges through the graphic neural network (GNN). Which feature data is transported in the form of a transfer data set TD is determined by neural weights of the graphic neural network (GNN). At each propagation step, the data to be transmitted is transported one edge or to a neighboring node.Through the training process described below, the neural weights were trained in such a way that those data elements that are most important for efficient coordination are transported to other control agents.
[0055] The transfer data transported to a node continuously updates the transfer data sets TD already stored in that node. The updated transfer data sets TD are continuously combined with the object-specific and other control agent-specific feature data of the respective node to form a current feature vector or aggregated in another way.
[0056] The resulting current feature vector of the respective node is then fed into the respectively assigned control agent P1 or P2, ..., which derives therefrom an object-specific action data record A1 or A2, .... A respective action data record A1 or A2, ... specifies a respective control action for controlling the machine M, in particular for manipulating the assigned object O1 or O2, .... Such action data records can be used, in particular, to specify manipulated variables or manipulated variable changes of the machine M, e.g., for setting individual running speeds of the conveyor belt segments FS.
[0057] The action data sets A1, A2, ... derived from the control agents P1, P2, ... are transmitted to the machine M, which executes the control actions specified by the action data sets A1, A2, ... . As a result, the objects O1, O2, ... currently accessible to the machine M are manipulated in a performance-optimized manner.
[0058] Figure 3 illustrates a data-driven training of the machine control CTL by joint training of the graphic neural network GNN and the joint control agent P .
[0059] Training is generally understood as the optimization of a mapping from the input data of a machine learning module, here the graphical neural network GNN and the joint control agent P, to its output data. This mapping is optimized according to predetermined criteria during a training phase. The success of control actions can be used as a criterion, particularly in control models. Through training, for example, the network structures of neurons in a neural network and / or the weights of connections between the neurons can be adjusted or optimized so that the predetermined criteria are met as closely as possible. Training can therefore be viewed as an optimization problem.For such optimization problems in the field of machine learning, a variety of efficient optimization methods are available, in particular gradient-based optimization methods, gradient-free optimization methods, backpropagation methods, particle swarm optimizations, genetic optimization methods and / or population-based optimization methods.
[0060] In particular, artificial neural networks, recurrent neural networks, convolutional neural networks, perceptrons, Bayesian neural networks, autoencoders, variational autoencoders, Gaussian processes, deep learning architectures, support vector machines, data-driven regression models, K-nearest neighbor classifiers, physical models or decision trees can be trained in this way.
[0061] As mentioned above, the graphical neural network (GNN) and the joint control agent (P) in the present embodiment are each implemented as an artificial neural network. Both networks (GNN and P) are trained together using a reinforcement learning method.
[0062] To provide training data, a simulator SIM is used, which dynamically simulates the machine M and its manipulation of virtual objects O1, O2, ... over time. For reasons of clarity, the same reference numerals O1, O2, ... are used for the virtual objects in Figure 3 as for the real objects in Figure 1. A variety of efficient simulators are available for such simulations. In particular, a so-called digital twin of the machine M can be used as a simulator. Alternatively or additionally, the training data can also be measured or otherwise recorded during operation of the machine M or a similar machine.
[0063] The simulator SIM comprises an object generator GEN for continuously generating a plurality of virtual objects 01, 02, .... The latter are continuously fed to the machine M at least partially randomly and / or according to a predetermined, as realistic as possible distribution in simulation.
[0064] As a result, the simulator SIM continuously determines or updates a dynamic graph G, which, as described above, specifies the current virtual objects 01, 02, ... and their current neighborhoods. The dynamic graph G is transmitted from the simulator SIM to the machine controller CTL.
[0065] In addition, the SIM simulator continuously generates object-specific state data sets SI, S2, ..., each of which specifies the current state of the generated virtual object 01 or 02, ..., as well as other feature data relevant for its manipulation. The current state data sets SI, S2, ... are transmitted from the SIM simulator to the CTL machine control system.
[0066] The machine controller CTL continuously checks the dynamic graph G to determine whether a new virtual object is specified therein. As soon as a new virtual object O1 or O2, ... is detected in this way, the machine controller CTL generates a new object-specific control agent P1 or P2, ... and assigns it to the respective new virtual object O1 or O2, ... The new control agent P1 or P2, ... is generated as a new instance of the common control agent P, as described above.
[0067] Furthermore, neurons of the graphic neural network (GNN) are continuously linked or their links broken by the machine control system (CTL) according to the neighborhoods encoded in the dynamic graph (G), as described above. The state data sets SI, S2, ... are continuously fed to the control agent P1 or P2, ... responsible for the respective virtual object O1 or O2, ..., as well as to a performance evaluator PEV of the machine control system (CTL). Furthermore, transfer data sets TD are continuously transmitted between the control agents P1, P2, .... The transmission is controlled by the graphic neural network (GNN) as described above.
[0068] The control agents P1, P2, ... each derive an object-specific action data set A1 or A2, ... as described above based on a supplied state data set SI or S2, ... and on received transfer data sets TD. The action data sets A1, A2, ... derived by the control agents P1, P2, ... are transmitted to the simulator SIM, which subsequently simulates the control actions specified by the action data sets A1, A2, ... and their effects. As a result of the simulation, the simulator SIM generates, in particular, current state data sets SI, S2, ... that specify the new states of the virtual objects O1, O2, ... and / or the machine M induced by the control actions.
[0069] The current state data sets SI, S2, ... are fed, among other things, into the performance evaluator PEV of the machine controller CTL. The performance evaluator PEV serves the purpose of determining a performance value RET based on state data from the machine M and / or objects manipulated by the machine M. This value quantifies the performance, particularly the cross-object performance, of the machine M. An evaluation function that returns such a performance value is often referred to as a reward function, objective function, cost function, or loss function in the context of machine learning.The performance to be evaluated can, in particular, relate to an object throughput, resource consumption, wear, power, yield, speed, runtime, precision, error rate, efficiency, yield, product quality, pollutant emissions, stability, a constraint to be met, or other target variables of the machine M or one of its components. In particular, a measure that indicates how efficiently the objects 01, 02, ... are manipulated by the machine M can be used as a performance measure.
[0070] Preferably, a cumulative performance over several time steps is determined. In this way, the induced control actions can also be evaluated with regard to longer-term performance goals. Such cumulative performance is often referred to as return in the context of reinforcement learning. To calculate the return, individual performance values determined for future time steps can be discounted, i.e., assigned weights that decrease for each time step. For the present embodiment, it is assumed that the performance value RET determined by the performance evaluator PEV is such a cumulative performance value.
[0071] To determine performance values RET, the performance evaluator PEV can, for example, have an additional machine learning module (not shown) trained to predict a resulting performance based on state data sets and / or action data sets. Standard supervised learning methods can be used to train such a predictive machine learning module. Alternatively or additionally, the performance evaluator PEV can also be trained using a Q-learning method to determine cumulative performance over a future period. A variety of efficient numerical optimization methods are available for implementing Q-learning methods.
[0072] Based on the performance values RET determined by the performance evaluator PEV, the joint control agent P and the graphic neural network GNN are jointly trained using a reinforcement learning method to optimize the performance of the machine M. For this purpose, the performance values RET are fed back to both the joint control agent P and the graphic neural network GNN, as indicated by dashed arrows in Figure 3. Depending on this, model parameters of the joint control agent P and the graphic neural network GNN to be optimized are set within the framework of an optimization process such that the performance values RET are optimized, at least on average, during the optimization process. In this way, the joint control agent P, in particular, is trained to control the manipulation of objects O1, O2, ... in a decentralized manner using object-specific instances P1, P2, ...A variety of efficient standard procedures can be used to conduct such training.
[0073] A concrete optimization procedure can be implemented as follows:
[0074] Let V n (t) , n=l,..., N(t) , a set of nodes of the dynamic graph G that changes with time t and represents a current set of control agents Pl, P2, ..., PN and objects O1, O2, ..., ON. N(t) indicates a current number of nodes. Let E(t) be a dynamic quadratic adj acency matrix of dimension N(t) , which contains an edge, i.e., a matrix element E nm (t)<>0 if the current distance between the nth object and the mth object is less than a specified maximum distance. In this case, the nth and mth objects are considered adjacent.
[0075] The graphical neural network GNN acts on the feature data of nodes Vn , where the current links of the nodes V n by the current adj acency matrix E(t), i.e., by the currently detected neighborhoods. Training the graphical neural network (GNN) then optimizes matrices Ak, k=l,...,K, which each map a feature vector of a respective node to data elements of a transfer data set TD to be transferred. Several propagation steps, here K, can be provided. Such propagation steps are often referred to as hops. The matrices Ak, indexed by the index k, are then step-specific and are optimized step-specifically.
[0076] By optimizing a respective matrix Ak, it is ensured that, in the k-th propagation step, those data elements are transferred that are most important for the efficient coordination of the control agents PI, ..., PN. The matrices Ak have Z rows and S columns, where S corresponds to the dimension of the feature vectors and Z to a dimension of a respective transfer data vector TD.
[0077] The feature vectors of the different nodes are combined row by row to form a feature matrix X, which accordingly has N rows and S columns. From the feature matrix X, a new feature matrix Y is formed according to Y = f( k E k *X*A T k ), where A kis the matrix transposed to Ak and Ek = E(t) *Ek i with EQ = E(t) , describes cross-hop neighborhoods. f is a nonlinear function, e.g., a sigmoid function or a tanh function, acting matrix-element-wise. The new feature matrix Y accordingly has N rows and Z columns.
[0078] The above calculation can be repeated several times, whereby the new feature matrix Y replaces the previous feature matrix X. Due to the possibly changed number of columns of the new feature matrix Y, the number of columns and possibly the number of rows of the matrices Ak also change in the next iteration step. Such an iteration step is often referred to as a layer in the context of graphical neural networks.
[0079] By training the graphic neural network (GNN) and the joint control agent (P), the matrices Ak of all layers, together with the learning parameters of the joint control agent (P), are adjusted to optimize performance as described above. In this way, the transmission of the transfer data sets TD by the graphic neural network (GNN) and the decentralized control efficiency of the control agents (P1, P2, ...) can be optimized with respect to the global performance of the machine (M).
Claims
Patent claims 1. Computer-implemented method for manipulating physical objects (01, 02) by a machine (M), wherein a) a plurality of objects (01, 02) are supplied to the machine (M) in real life or in dynamic simulation, b) it is continuously detected by sensors or simulation which of the objects (01, 02) are currently adjacent to one another according to a predetermined neighborhood criterion, c) a control agent (Pl, P2) for controlling the machine (M) is assigned to each of the objects (01, 02), into which control agent current state data (S1, S2) of the respective object (01, 02) are fed, d) neurons of a neural network (GNN) are continuously linked according to the currently detected neighborhoods of the objects (01, 02), e) a transmission of transfer data sets (TD) between the control agents (Pl, P2) is controlled by the neural network (GNN), f) by the Control agents (Pl, P2) based on the input status data (SI,S2) and the transmitted transfer data sets (TD), a manipulation of the objects (01, 02) is controlled in real life or in simulation, wherein a related performance (RET) of the machine (M) is determined, g) the neural network (GNN) is trained to optimize the performance (RET), and h) the machine (M) is controlled by means of the trained neural network (GNN) and the control agents (P1, P2) in order to manipulate the physical objects (01, 02).
2. Method according to claim 1, characterized in that the control agents (Pl, P2) are trained together with the neural network (GNN) to optimize the performance (RET).
3. Method according to one of the preceding claims, characterized in that an instance (P1, P2) of a common control agent (P) is assigned to each of the objects (01, 02), and in that the common control agent (P) is trained together with the neural network (GNN) to optimize the performance (RET).
4. Method according to one of the preceding claims, characterized in that during the transmission of the transfer data sets (TD), the transfer data sets (TD) are propagated by the neural network (GNN).
5. Method according to one of the preceding claims, characterized in that the transfer data sets (TD) are transmitted along links of the neural network (GNN) 6. Method according to one of the preceding claims, characterized in that the neural network (GNN) is a graphic neural network.
7. Method according to one of the preceding claims, characterized in that a distance measure is used as the proximity criterion.
8. Method according to one of the preceding claims, characterized in that a dynamic graph (G) is generated from the currently detected neighborhoods, the nodes of which are each assigned to a Control agents (Pl, P2) or an object (01, 02), wherein nodes currently assigned to adjacent objects (01, 02) are connected by an edge, and wherein an edge is deleted when the objects (01, 02) assigned to the nodes connected thereby are no longer adjacent, and that the neurons of the neural network (GNN) are linked according to the edges of the dynamic graph (G).
9. The method according to claim 8, characterized in that in a respective node, status data (SI, S2) of the assigned object (01, 02), control-relevant data of the assigned control agent (Pl, P2) and / or transfer data records (TD) received from neighboring nodes of the respective node are stored.
10. Method according to one of the preceding claims, characterized in that a statistical distribution of control signal values is output by the control agents (P1, P2) in each case, and in that the neural network and / or the control agents (P1, P2) are trained on the basis of the determined performance (RET) and on the basis of gradients of the statistical distributions.
11. Method according to one of the preceding claims, characterized in that a gradient-free optimization method is used to train the neural network (GNN) and / or the control agents (P1, P2).
12. Method according to one of the preceding claims, characterized in that that a machine learning module is trained to predict a resulting performance based on control signals (A1, A2) of the control agents (Pl, P2), and that for training the neural network (GNN) and / or the control agents (Pl, P2), the trained machine learning module is used to predict the performance (RET), wherein the training is carried out based on gradients of the predicted performance.
13. Machine control (CTL) for controlling a machine (M), with means for carrying out the steps of a method according to one of the preceding claims.
14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 12.
15. A computer-readable storage medium comprising a computer program product according to claim 14.
Citation Information
Patent Citations
Method for configuring a control agent for a technical system and control device
EP3940596A1
Method and control device for controlling a technical system
WO2023072528A1
