Generative flow model training method and apparatus, and action prediction method and apparatus

Abstract

Provided in the present application are a generative flow model training method and apparatus, and an action prediction method and apparatus, which can realize generative flow model training of a directed acyclic graph or a cyclic graph, and can be more widely applied to various artificial intelligence scenarios. The generative flow model training method comprises: acquiring a plurality of state nodes in a generative flow model; then, acquiring the value of a loss function on the basis of the plurality of state nodes, wherein the value of the loss function comprises a fused value of the value of a flow matching function and the value of a constraint function, each state node has an input flow and an output flow, the value of the flow matching function comprises a value calculated on the basis of the input flow and the output flow of each state node among the plurality of state nodes, the value of the constraint function comprises a value calculated on the basis of evaluation values corresponding to the output flows of the plurality of state nodes, and the value of a constraint function corresponding to an output flow of a state node in a cyclic path is different from the value of a constraint function corresponding to an output flow of a state node in an acyclic path; and updating the generative flow model on the basis of the value of the loss function to obtain an updated generative flow model.

Description

A generative flow model training method, action prediction method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 28, 2023, with application number 202311292420.7 and application name “A generative flow model training method, action prediction method and device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to a generative flow model training method, an action prediction method, and an apparatus. Background Art

[0003] Generative flow models construct composite structures through sequential decision-making. Commonly used generative flow models are typically constructed using directed acyclic graphs (DAGs). This means that each state node can have one or more parent nodes, unlike a tree structure where each state node has only one parent node. Generative flow models have a single initial node and multiple terminal nodes. Starting from the initial node, the model samples and generates a sequence of actions, completing transitions between states until it reaches the terminal node, where sampling ends. The terminal node corresponds to the generated composite structure.

[0004] However, for the training of generative flow models, existing solutions can usually only be applied to models built on directed acyclic graphs. If the model is built on a cyclic graph, the edge traffic of the ring will accumulate infinitely, making model training impossible.

[0005] Summary of the Invention

[0006] The present application provides a generative flow model training method, an action prediction method, and a device, which can realize generative flow model training for directed acyclic graphs or cyclic graphs, and can be more widely applied to various artificial intelligence scenarios.

[0007] In view of this, in the first aspect, the present application provides a training method for a generative flow model, the method comprising: first, obtaining a plurality of state nodes in the generative flow model, the plurality of state nodes can be used to represent the state of an intelligent agent, and the state nodes can be transferred by executing actions; then obtaining the value of a loss function according to the plurality of state nodes, the value of the loss function including the value of a flow matching function and the value of a constraint function fused together, the plurality of state nodes in the generative flow model can be connected by a flow, the flow can be used to represent the probability of selecting each action when the current state node transfers to the next state node, each state node has an input flow and an output flow, the value of the flow matching function includes a value based on a plurality of state nodes, and the loss ... The values ​​​​of the constraint function include the values ​​​​of the evaluation values ​​​​corresponding to the output streams of multiple state nodes, wherein the value of the constraint function corresponding to the output stream of the state node in the cyclic path is different from the value of the constraint function corresponding to the output stream of the state node in the acyclic path. The cyclic path includes the path forming a loop between the multiple state nodes, and the acyclic path is the path that does not include the loop formed between the multiple state nodes. The starting point and end point of the path formed between the multiple states are usually determined according to the task performed by the generative flow model; then the generative flow model is updated according to the value of the loss function to obtain the updated generative flow model.

[0008] In the implementation manner of the present application, a constraint term, i.e., a constraint function, is set in the loss function, and the size of the output value of the loss function is constrained by the constraint function, thereby constraining the overall output of the generative flow model to converge to a non-circular flow. For example, in the loss function part, the reward value set for the loop flow can be increased so that the loss of the loop flow is greater than 0, thereby constraining the reward of the loop flow to converge to close to 0, making the flow of the cyclic part closer to 0, and reducing the impact of the loop on the model output. Therefore, the method provided by the present application can be applied to generative flow models with and without loops, and therefore can be applied to generative flow model training in more scenarios, with stronger generalization ability.

[0009] In one possible implementation, the aforementioned acquisition of the value of the loss function based on multiple state nodes may include: determining the probability distribution corresponding to the multiple state nodes based on the output results of the policy network, and the probability distribution can be used to determine the probability of at least one action to be selected when selecting the next action based on each state node; and then calculating the value of the loss function based on the probability distribution and the multiple state nodes.

[0010] That is, the method provided in this application can be applied to reinforcement learning, and the probability distribution corresponding to each state node can be learned through the policy network. The aforementioned training process can be one of the iterative processes in the iterative training. In the current iterative process, the probability distribution can be determined by the output of the policy network after the previous training, so as to calculate the value of the loss function based on the probability distribution and the possible transfer paths between multiple state nodes in the generated flow network, and thus update the policy network based on the value of the loss function, thereby updating the generated flow network based on the output result of the policy network.

[0011] In a possible implementation, the aforementioned determination of the probability distribution corresponding to multiple state nodes based on the policy network may include: outputting a measurement value of the flow transmitted between multiple state nodes in the measurement space based on the policy network, the measurement value may be used to represent the density of the flow transmitted between multiple state nodes in the measurement space; and then determining the probability distribution based on the measurement value between the multiple state nodes, that is, the probability of selection corresponding to each action to be selected when each state node performs a state transition selection action. In the implementation of the present application, the density in the measurement space has a corresponding relationship with the probability, so the probability of selecting each action when performing a state transition can be determined by the measurement value output by the policy network, thereby more intuitively measuring the input flow and output flow of each state node and the corresponding probability through the measurement space.

[0012] In one possible implementation, the aforementioned calculation of the value of the loss function based on the probability distribution and multiple state nodes may include: determining at least one path between multiple state nodes based on the probability distribution, the starting point and end point of each path being determined by the tasks that need to be performed according to the generation flow model. For example, in an autonomous driving agent, the state node may represent the current autonomous driving state (such as a vehicle speed of 5 km / h), which is the starting point of the path, and the end point may be the target driving state that requires autonomous driving (such as increasing the vehicle speed to 60 km / h). The transfer actions between state nodes may include acceleration intervals [5,15], [15,20], etc.; calculating the value of the loss function based on at least one path and the input flow and output flow of each state node.

[0013] In the implementation mode of the present application, after determining the probability distribution in the measurement space, one or more possible state transition paths can be determined based on the probability distribution, thereby calculating the value of the loss function based on the one or more possible state transition paths to achieve the update of the generated flow network.

[0014] In one possible implementation, the aforementioned constraint function may include a function based on the norm of an edge flow matrix. The edge flow matrix is ​​a matrix formed by edge flows between intermediate state nodes in a plurality of state nodes. An edge flow is a flow transmitted between any pair of nodes in a plurality of state nodes. In the implementation of the present application, the norm of the edge flow matrix can be used to constrain the edge flows, which can reduce sampling time and accelerate model convergence.

[0015] In one possible implementation, the aforementioned loss function includes a function formed by the divergence between the flow matching function and the constraint function. Therefore, in the implementation of the present application, the loss value is calculated by generalized divergence, thereby obtaining a more stable loss value, which can accelerate model convergence and improve training efficiency.

[0016] In one possible implementation, the aforementioned loss function includes a function based on node flow matching loss, a function based on trajectory balance loss, a function based on node density loss in continuous space, or a function based on trajectory density loss in continuous space. Therefore, in the implementation of this application, loss values ​​can be calculated from multiple dimensions, which has strong generalization.

[0017] In a second aspect, the present application provides an action prediction method, comprising: obtaining state information, the state information being used to indicate the current state of an agent; using the state information as input to generate a flow model to obtain a probability value corresponding to at least one action, the probability value corresponding to the at least one action being used to determine the action performed by the agent; wherein the generation model is obtained by training using a method such as the aforementioned first aspect or any optional implementation method of the first aspect.

[0018] The effects achieved by the second aspect can be found in the description of the first aspect or any optional implementation manner of the first aspect, and will not be repeated here.

[0019] In a third aspect, the present application provides an action prediction method, comprising: first obtaining multiple state nodes in a generative flow model; then obtaining measure values ​​of the multiple state nodes, the measure values ​​representing the density of paths formed by the multiple state nodes when performing state transitions in the measure space, the measure values ​​being used to determine a probability distribution, and the probability distribution being used to determine the probability of at least one action to be selected when selecting the next action based on each state node in the multiple state nodes; determining at least one path based on the measure values ​​of the multiple state nodes, the at least one path including a path formed by transitions between multiple state nodes based on actions selected based on the probability distribution, wherein the starting point and end point of each path are determined based on the task performed according to the generative flow model.

[0020] Therefore, in the embodiments of the present application, action prediction can be performed in a measurable measurement space, so that a more accurate next action can be predicted.

[0021] In one possible implementation, the aforementioned method may further include: calculating a loss value based on at least one path; and updating the generative flow model based on the loss value to obtain an updated generative flow model. Therefore, the method for action prediction in a measurement space provided by this application can also be applied to the training phase, and has strong generalization capabilities.

[0022] In a fourth aspect, the present application provides a training device for generating a flow model, the device comprising:

[0023] An acquisition module is used to acquire multiple state nodes in the generated flow model;

[0024] a calculation module for obtaining a value of a loss function based on multiple state nodes, the value of the loss function including a fusion value of a flow matching function and a constraint function, the value of the flow matching function including a value calculated based on an input flow and an output flow of each state node in the multiple state nodes, the value of the constraint function including a value calculated based on an evaluation value corresponding to an output flow of the multiple state nodes, wherein the value of the constraint function corresponding to the output flow in a cyclic path is different from the value of the constraint function corresponding to the output flow in an acyclic path, the cyclic path includes a path forming a loop between state nodes, and the acyclic path is a path formed between state nodes that does not include a loop;

[0025] The updating module is used to update the generative flow model according to the value of the loss function to obtain an updated generative flow model.

[0026] Among them, the effects achieved by the fourth aspect and any optional implementation of the fourth aspect can be found in the description of the aforementioned first aspect or any optional implementation of the first aspect, and will not be repeated here.

[0027] In one possible implementation, the aforementioned calculation module is specifically used to: determine the probability distribution corresponding to multiple state nodes based on the policy network, the probability distribution is used to determine the probability of at least one action to be selected when selecting the next action based on each state node; calculate the value of the loss function based on the probability distribution and multiple state nodes.

[0028] In one possible implementation, the aforementioned calculation module is specifically used to: output a measurement value of a flow transmitted between multiple state nodes in a measurement space according to a policy network, where the measurement value represents the density of the flow transmitted between multiple state nodes in the measurement space; and determine a probability distribution based on the measurement value between multiple state nodes.

[0029] In one possible embodiment, the aforementioned computing module is specifically used to: determine at least one path between multiple state nodes based on a probability distribution, where the starting point and end point of the at least one path are determined by the task performed according to the generation flow model; and calculate the value of the loss function based on the at least one path and the input flow and output flow of each state node.

[0030] In a possible implementation, the aforementioned constraint function includes a function based on the norm of an edge flow matrix, where the edge flow matrix is ​​a matrix formed by edge flows between intermediate state nodes in a plurality of state nodes, the edge flow.

[0031] In a possible implementation, the aforementioned loss function includes a function formed by the divergence between the flow matching function and the constraint function.

[0032] In a possible implementation, the aforementioned loss function includes a function based on node flow matching loss, trajectory balance loss, node density loss in continuous space, or trajectory density loss in continuous space.

[0033] In a fifth aspect, the present application provides an action prediction device, comprising:

[0034] The acquisition module is used to obtain state information, which is used to indicate the current state of the agent;

[0035] A generation module, configured to use the state information as an input to generate a flow model to obtain a probability value corresponding to at least one action, wherein the probability value corresponding to the at least one action is used to determine an action to be performed by the agent;

[0036] The generative model is obtained by training through the method of the aforementioned first aspect or any optional implementation method of the first aspect.

[0037] In a sixth aspect, the present application provides an action prediction device, comprising:

[0038] An acquisition module is used to acquire multiple state nodes in the generated flow model;

[0039] a measurement module, configured to obtain measurement values ​​of a plurality of state nodes, the measurement values ​​representing the density of paths formed by the plurality of state nodes when performing state transitions in a measurement space, the measurement values ​​being used to determine a probability distribution, the probability distribution being used to determine a probability of at least one action being selected when selecting a next action based on each of the plurality of state nodes;

[0040] A determination module is used to determine at least one path based on the measurement values ​​of multiple state nodes, where the at least one path includes a path formed by transferring actions between multiple state nodes based on probability distribution, and the starting point and end point of the at least one path are determined by the tasks executed according to the generation flow model.

[0041] In a possible implementation, the apparatus further includes an updating module configured to: calculate a loss value according to at least one path; and update the generated flow model according to the loss value to obtain an updated generated flow model.

[0042] In a seventh aspect, embodiments of the present application provide a training device for a generative flow model, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor invokes program code in the memory to execute processing-related functions in the training method for a generative flow model described in any of the first aspects above. Optionally, the data quantization device may be a chip.

[0043] In an eighth aspect, embodiments of the present application provide an action prediction device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor invokes program code in the memory to execute the processing-related functions of the action prediction method described in any one of the second aspects above. Optionally, the data quantization device may be a chip.

[0044] In a ninth aspect, embodiments of the present application provide an action prediction device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor invokes program code in the memory to execute the processing-related functions of the action prediction method described in any one of the third aspects. Optionally, the data quantization device may be a chip.

[0045] In the tenth aspect, an embodiment of the present application provides a digital processing chip or chip, the chip includes a processing unit and a communication interface, the processing unit obtains program instructions through the communication interface, the program instructions are executed by the processing unit, and the processing unit is used to perform processing-related functions in any optional implementation of the first to third aspects above.

[0046] In an eleventh aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the method in any optional implementation of the first to third aspects above.

[0047] In the twelfth aspect, an embodiment of the present application provides a computer program product comprising a computer program / instructions, which, when executed by a processor, enables the processor to execute a method in any optional implementation of the above-mentioned first to third aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG1 is a schematic diagram of a reinforcement learning process provided by this application;

[0049] FIG2 is a schematic diagram of the structure of a generation flow model provided by the present application;

[0050] FIG3 is a flow chart of a training method for generating a flow model provided by the present application;

[0051] FIG4 is a flow chart of another training method for generating a flow model provided by the present application;

[0052] FIG5 is a flow chart of another training method for generating a flow model provided by the present application;

[0053] FIG6 is a flow chart of an action prediction method provided by the present application;

[0054] FIG7 is a flow chart of another motion prediction method provided by the present application;

[0055] FIG8 is a schematic structural diagram of a training device for generating a flow model provided by the present application;

[0056] FIG9 is a schematic structural diagram of an action prediction device provided by the present application;

[0057] FIG10 is a schematic structural diagram of another motion prediction device provided by the present application;

[0058] FIG11 is a schematic structural diagram of another training device for generating a flow model provided by the present application;

[0059] FIG12 is a schematic diagram of the structure of another motion prediction device provided by the present application;

[0060] FIG13 is a schematic diagram of a chip structure provided in this application. DETAILED DESCRIPTION

[0061] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0062] First, the method provided in this application involves concepts related to neural networks. For ease of understanding, the relevant concepts of the neural network involved are introduced below.

[0063] (1) Neural Network

[0064] A neural network can be composed of neural units, which can be represented by x s The output of the arithmetic unit can be:

[0065] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x sThe weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to perform nonlinear transformation on the features obtained in the neural network and convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0066] (2) Convolutional Neural Network

[0067] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which is composed of a rectangular arrangement of neurons. Neurons in the same feature plane can share weights, which are referred to as convolution kernels. Shared weights can be understood as position-independent feature extraction. Convolution kernels can be formalized as matrices of random size, and during CNN training, the kernels can be learned to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0068] (3) Loss function

[0069] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function (loss function) or objective function (objective function), which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function can usually include loss functions such as mean square error, cross entropy, logarithm, exponential, etc. For example, the mean square error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0070] (4) Graph

[0071] A graph is a data structure consisting of at least one vertex and at least one edge. In some scenarios, vertices in a graph can be mapped to entities, and edges can be mapped to relationships between entities. A graph can be directed or undirected. Of course, a graph can also include data other than vertices and edges, such as vertex and edge labels.

[0072] (5) Directed acyclic graph (DAG)

[0073] A directed acyclic graph is a directed graph without loops. For example, in a non-directed acyclic graph, if a node A goes from point B to point C and then back to point A, forming a loop, then changing the direction of the edge from point C to point A to point A to point C will turn it into a directed acyclic graph. The number of spanning trees in a directed acyclic graph is equal to the in-degree product of the nodes with non-zero in-degree.

[0074] (6) Cyclic graph

[0075] Similar to the aforementioned DAG, the difference is that a cyclic graph may include a ring structure formed by connections between multiple nodes.

[0076] (7) Reinforcement Learning

[0077] Reinforcement learning (RL), also known as reinforcement learning, evaluation learning, or enhanced learning, is used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment.

[0078] Reinforcement learning involves an agent learning through trial and error. Rewards earned through interactions with the environment guide the agent's behavior, with the goal of maximizing the reward. Reinforcement learning typically requires no training dataset, or can generate training data through interaction with the environment. In reinforcement learning, the reinforcement signal (i.e., reward) provided by the environment evaluates the quality of the action rather than instructing the reinforcement learning system on how to perform the correct action. Because the external environment provides little information, the agent must rely on its own experience to learn. In this way, the agent acquires knowledge in an action-reward (i.e., reward) environment and refines its action plan to adapt to the environment.

[0079] Reinforcement learning is a machine learning paradigm used to solve problems in which intelligent agents learn strategies to maximize rewards or complete specific machine learning tasks during their interactions with the environment. The agent learns through trial and error, earning rewards through interaction with the environment and using them to guide action selection. It finds the optimal strategy for the current state and selects appropriate actions based on that strategy to maximize its rewards.

[0080] Furthermore, there are two crucial concepts in reinforcement learning: exploitation and exploration, sometimes also called utilization and trial. In reinforcement learning, exploitation refers to the agent selecting the optimal action from a known distribution of binary (state-action) pairs, based on the principle of maximizing the value of that action. In other words, when the agent chooses from a known set of actions, it is called exploitation. Exploration, on the other hand, refers to the agent selecting unknown actions outside of the known distribution of binary (state-action) pairs.

[0081] For example, Figure 1 is a diagram of the reinforcement learning training process. As shown in Figure 1, reinforcement learning mainly includes multiple elements: an agent, an environment, a state, an action, and a reward. The agent's input is called a state, and its output is called an action.

[0082] For example, the training process for reinforcement learning involves multiple interactions between the agent and the environment, obtaining the action, state, and reward for each interaction. These multiple sets of data (action, state, reward, and next-moment state) are used as training data for the agent to train once. The agent is trained again using this process until convergence conditions are met.

[0083] The process of obtaining the action, state, and reward for an interaction is shown in Figure 1. The current state of the environment, s(t), is input to the agent, and the agent's output action, a(t), is obtained. The reward r(t) for this interaction is calculated based on the relevant performance indicators of the environment under the influence of action a(t). At this point, the state s(t), action a(t), and reward r(t) for this interaction are obtained. The state s(t), action a(t), and reward r(t) of this interaction are recorded for subsequent training of the agent. The next state s(t+1) of the environment under the influence of action a(t) is also recorded to facilitate the next interaction between the agent and the environment.

[0084] An agent is an entity capable of thinking and interacting with its environment. For example, an agent can be a computer system or part of a computer system in a specific environment. Based on its own perception of the environment, the agent can follow instructions or learn autonomously, and communicate and collaborate with other agents to autonomously achieve its goals within its environment. An agent can be software or a combination of software and hardware. In the federated learning process described in this application, the agent can be a client.

[0085] Decision model: This refers to the behavioral rules used by an agent in reinforcement learning. For example, during learning, an agent can output actions based on the state and use these actions to explore the environment and update the state. Decision model updates rely on policy gradients (PGs). Decision models are typically Q networks or policy networks, such as multilayer perceptrons (MLPs) or other deep reinforcement learning models (deep Q networks, DQNs).

[0086] The Markov decision process (MDP) is a common model in reinforcement learning. It is a mathematical model for analyzing decision problems based on discrete-time stochastic control. It assumes that the environment has the Markov property (the conditional probability distribution of the environment's future state depends only on the current state). The decision maker periodically observes the state of the environment, makes decisions (also called actions) based on the current state, and interacts with the environment to obtain the next state and reward. In other words, at each time t, the state s(t) observed by the decision maker, under the influence of the action a(t), will transition to the next state s(t+1) and feedback reward r(t). Here, s(t) represents the state function, a(t) represents the action function, r(t) represents the reward, and t represents time.

[0087] Markov chain: Assume a random process of transition from one state to another. For state s, the next state depends on the current state and is independent of the state before it. That is, s(t+1) is only related to s(t). Therefore, when state transitions are performed using a Markov chain, a state transition path with a circular structure may occur. The paths formed by multiple state nodes in the method provided herein can include paths formed based on Markov chains.

[0088] State transition matrix: In a Markov decision process, each state transitions to the next state with a certain probability. The transition probabilities between multiple state nodes can be represented as a matrix. For example, in the method provided in this application, the transition probabilities between states can be learned through a policy network. That is, the policy network can output a state transition matrix, and learning is completed after the transition probabilities converge.

[0089] Forward and backward probabilities: Since each state generates an observation variable, there will be t observation variables generated at time t. The probability of being in state i at time t is the forward probability α. At time t, under the condition that the state is s, the probability of being in the observation variable O is the backward probability β.

[0090] (8) Measure space

[0091] Assuming that (X, R) is a measurable space and μ is a measure on R, then (X, R, μ) is called a measure space. Vectors in the measure space are usually non-negative, normalized, and additivity. In the embodiments mentioned below in this application, the state nodes can be projected into the measurable space, and the measures in the measure space can be used to measure the flows transmitted between the state nodes.

[0092] (9) Generative Model

[0093] A generative model is a model that generates observations randomly, typically given certain implicit parameters. It assigns a joint probability distribution to a sequence of observations and labeled data. In machine learning, generative models can be used to directly model data (e.g., sampling data based on the probability density function of a variable) or to construct conditional probability distributions across variables. Conditional probability distributions can be formed from generative models using Bayes' theorem.

[0094] (10) Generate flow model

[0095] The generative flow model (or flow-based generative flow model or generative flow model) is a flow model that samples in a sequential decision-making manner and constructs a composite structure, so that the probability of generating a structure is proportional to the reward return value of the structure. The generative flow model constructs a composite structure in a sequential decision-making manner. It constructs the flow model in a directed acyclic graph, that is, each state node has multiple parent nodes, which is different from the tree structure where each state node has only one parent node. Among them, the directed acyclic graph has a unique initial node and multiple terminal nodes. The generative flow model starts sampling and generating an action sequence from the initial node of the directed acyclic graph, completes the transfer between states, and ends sampling until it reaches the terminal node. The terminal node corresponds to a corresponding composite structure.

[0096] The initial node can include output flow, the intermediate nodes can include input flow and output flow (or reward value), and the terminal node can include input flow and reward value. Imagine the flow model as a water pipe. The water flow of the initial node is the total inflow of the model, and the sum of the water flow of all terminal nodes (such as) is the total outflow of the model. For each intermediate node, the inflow is equal to the outflow. The inflow and outflow of the node are predicted using a neural network (that is, the generative flow can be specifically constructed using a neural network). By optimizing the flow matching constraint as the objective function, the generative flow model can learn a strategy (that is, optimize the neural network) so that the probability of sampling and generating a composite structure is proportional to its reward value. Structures with higher reward values ​​are more likely to be sampled. In this way, the generative flow model can sample a series of structures with high reward values.

[0097] An exemplary diagram of the generation flow model can be referred to FIG2 , as shown in FIG2 , where s i Represents the state, x j Represents a composite structure. s0 is the initial node, s3, s9, s 10 , s 11 are terminal nodes because they correspond to the composite structures x5, x9, x 10 , x 11Each state node in Figure 2 can be regarded as the state of the agent, the edge between state nodes can be regarded as the action performed from the previous state node to the next state node, and the value of the edge between state nodes (i.e. the outflow / inflow of the node mentioned above) is used to represent the probability of executing the action.

[0098] However, during the construction or training of some existing generative flow models, they are typically only able to train generative flow models with directed acyclic graph structures. For generative flow models that may contain cyclic graphs, this may lead to infinite accumulation of edge flows, resulting in low convergence efficiency or even failure of model training. Furthermore, this may lead to unstable model structures.

[0099] Therefore, the present application provides a generative flow model training method, which can realize a generative flow model based on a graph structure, can realize the generative flow model training of a directed acyclic graph or a cyclic graph, and can be more widely applied to various artificial intelligence scenarios.

[0100] Taking the flow model shown in Figure 2 as an example, each state node in Figure 2 can be considered the state of the agent, the edges between state nodes can be considered the actions performed to transfer from the previous state node to the next state node, and the values ​​of the edges between state nodes (i.e., the outflow / inflow of the node mentioned above) are used to represent the probability of executing the action. In the process of training the generative flow model, the loss function is mainly obtained by constraining the inflow and outflow of the generative flow model for a certain state node, that is, the inflow and outflow of the state node are equal.

[0101] Simply put, for any state node, the state node may have multiple parent nodes and multiple child nodes. The value of the edge between the state node and the multiple parent nodes is the inflow corresponding to the state node, and the value of the edge between the state node and the multiple child nodes is the outflow corresponding to the state node. In the method provided by the present application, during the training phase, a flow matching function and a constraint function are set in the loss function. The flow matching function is the loss value between the inflow and outflow of each state node, and the constraint function is the function for performing eye calculations on the output flow of each state node, such as calculating an evaluation value, a reward value, or a penalty value based on the output flow. Therefore, even if there is a circular path in the generative flow model, the constraint term can be used to reduce the impact of the circular path on the output result of the generative flow model, thereby avoiding the formation of an infinite loop flow within the generative flow, and realizing training for a generative flow model with a loop.

[0102] Specifically, the training method for generating a flow model provided in the embodiments of the present application can be applied to electronic devices. For example, the electronic devices can be servers, robots, smartphones, personal computers (PCs), laptops, wireless electronic devices in industrial control, wireless electronic devices in self-driving, wireless electronic devices in smart grids, wireless electronic devices in logistics and warehousing, wireless electronic devices in transportation safety, wireless electronic devices in smart cities, etc.

[0103] The above introduces the equipment used for the training method of the generative flow model provided in the embodiment of the present application. The following introduces the scenario to which the training method of the generative flow model provided in the embodiment of the present application is applied. Specifically, the training method of the generative flow model provided in the embodiment of the present application can be applied to train a generative flow model that performs one or more tasks. Among them, the tasks performed by the generative flow model may include but are not limited to autonomous driving tasks, robot control tasks, game character control tasks, recommendation tasks, parameter optimization tasks, molecular structure search or construction tasks, or protein structure search tasks, etc.

[0104] For example, in an autonomous driving task, each intelligent vehicle can be considered an intelligent agent, and each intelligent vehicle can obtain information about itself and other intelligent vehicles through sensors, such as its position on the road, its speed, the distance between it and other intelligent vehicles, and the speed and acceleration of other intelligent vehicles. The information obtained by each intelligent vehicle about itself and other intelligent vehicles can be considered its observation of the environmental state, and can also be referred to as state information corresponding to the state of the intelligent vehicle. Based on its state, the intelligent vehicle needs to determine its own actions, such as acceleration, deceleration, or lane change, to maximize the positive feedback it receives from the environment, i.e., maximize the safety factor of the intelligent vehicle or minimize the travel time it takes to reach its destination. The action space within which intelligent vehicles can perform actions is typically continuous. For example, the action space specifically includes: accelerating to 80-200 km / h, decelerating to 0-80 km / h, and adjusting a turning angle between 0-40 degrees. Based on the actual application scenario, the intelligent vehicle can choose to perform any action in the action space.

[0105] For another example, in robotic control tasks, each robot is considered an intelligent agent, and multiple robots may collaborate to complete a specific task. For example, in the logistics and warehousing field, multiple robots collaborate to transport designated goods to a specific location. Each robot can also obtain information about itself and other robots through sensors, such as its own speed, the rotation angle of its own robotic arm, the distance between itself and other robots, the current actions of other robots, and the operating speed of other robots. The information obtained by each robot about other robots can be regarded as its observation of the environmental state. For robots, based on their observations of the environmental state, they need to determine their actions, such as movement direction, rotation angle, and pauses, to maximize the positive feedback they receive from the environment. This means that the robots can complete the transport task in the shortest possible time while avoiding collisions. Similarly, the action space within which robots can perform actions is continuous. For example, the action space specifically includes movement direction adjustment from 0 to 360 degrees and arm rotation angles from 0 to 180 degrees.

[0106] For example, in a game character control task, each character unit is considered an intelligent agent, and multiple character units may collaborate to complete a specific combat task. For example, in a large-scale real-time strategy game, multiple character units of the same race work together to perform tasks against character units of other races. Each character unit can obtain information about itself and other character units of the same race, such as its location, the attack targets of itself and other character units, attack methods, and movement directions. The information obtained by each character unit about itself and other character units can be regarded as the character unit's observation of the environmental state. Each character unit in the game needs to determine its own actions based on its observation of the environmental state, such as switching attack targets, changing attack methods, and changing movement routes, so that the positive feedback the character unit receives from the environment is as high as possible, that is, the character unit can complete the combat task with as little loss as possible.

[0107] For recommendation tasks, these tasks can be, for example, social recommendation tasks, product recommendation tasks, or advertising recommendation tasks. Taking the product recommendation task as an example, the server that recommends products to users can be considered an intelligent agent. The states of the recommendation environment in which the agent resides, such as when a user clicks on or closes a product recommended by the server, can be considered. The server's recommendation of a specific product to a user can be considered an action performed by the server. Because the server can recommend a vast number of products, the action space in which the server can perform actions is continuous, meaning that the action space can be considered to include an infinite number of actions. The server needs to determine its own actions based on observations of the recommendation environment, such as recommending certain types of products or products within certain price ranges. This ensures that the server receives as much positive feedback from the recommendation environment as possible, meaning that users click on and browse the product information recommended by the server as frequently as possible.

[0108] For parameter optimization tasks, these tasks can include optimizing the hyperparameters of a neural network during training or optimizing communication parameters in a wireless network scenario. Taking the optimization of hyperparameters during neural network training as an example, the server used to train the neural network can be considered an intelligent agent. The performance of the neural network after adjusting the training hyperparameters can be considered the state of the parameter optimization environment in which the intelligent agent resides; the way the server adjusts the hyperparameters of the neural network during training can be considered the actions performed by the server. Since hyperparameters can typically take values ​​within one or more intervals, the action space of the actions that the server can perform is also continuous, that is, the action space can be considered to include an infinite number of actions. The server needs to determine its own actions based on the observed values ​​of the state of the parameter optimization environment, such as adjusting the hyperparameters to a specific value, so that the server receives as much positive feedback from the parameter optimization environment as possible, that is, the performance of the trained neural network after adjusting the hyperparameters is as high as possible.

[0109] The following describes the method flow provided by this application in conjunction with the aforementioned application scenarios.

[0110] The method provided in this application can be divided into a training phase and an inference phase. The training phase is the phase of training the generative flow model, and the inference phase is the phase of using the trained generative flow model to predict the next action based on the current state. Among them, the training phase and the inference phase can be deployed in the same device or in different devices. For example, the training phase can be deployed in the cloud or on a server, and the inference phase can be deployed in an intelligent terminal, such as a mobile phone, tablet, smart car, robot, or wearable device.

[0111] For ease of understanding, this application introduces the processes of the training phase and the inference phase respectively.

[0112] 1. Training Phase

[0113] Referring to FIG3 , a flow chart of a training method for generating a flow model provided in the present application is shown as follows.

[0114] It should be noted that the training process of the model can usually be divided into multiple iterations or multiple executions. This application exemplarily introduces one of the iterations or execution processes as an example. The steps mentioned below can be repeated and will not be repeated below.

[0115] 301. Obtain multiple state nodes in the generated flow model.

[0116] The generation flow model can be updated during each iteration, so the generation flow model mentioned in this step can be the generation flow model updated in the previous iteration, and each state node has an input flow and an output flow.

[0117] Typically, a node has one or more parent nodes. The input stream is the flow from these one or more parent nodes to the current state node. The current state node may also have one or more child nodes, and the output stream is the flow from the current node to these one or more child nodes. The output stream can be used to calculate the probability of each action being selected when the current node selects the next action from the one or more actions.

[0118] In addition, if the current state node is connected to multiple child nodes, the flow transmitted between the current state node and one of the child nodes can be called an edge flow or a sub-flow. In other words, the flow between a pair of state nodes can be called an edge flow or a sub-flow.

[0119] Furthermore, during the training phase, the policy network can select the next action based on the probability distribution of each action when the generative flow model selects the next action at each state node. For example, referring to the reinforcement learning process described in Figure 1 above, an iteration can be understood as the interaction between the agent and the environment. In each iteration, the state information fed back by the environment in the previous iteration is used as the input of the agent, or as the input of the generative flow model. Combined with the probability distribution output by the policy network, an output stream is output for each state node in the current iteration. This output stream can be understood as a probability stream, which can be used to determine the probability of each action being selected at the current state node.

[0120] 302. Calculate the value of the loss function according to each state node, where the loss function includes a flow matching function and a constraint function.

[0121] The flow matching function can be used to calculate the loss value based on the difference between the input flow and the output flow of each state node, and the constraint function can be calculated based on the evaluation value corresponding to each state node. The constraint function can be understood as a constraint term in the loss function, especially for the loop part, and can be used to constrain the flow of the loop part. It can be convenient to constrain the reward of the overall flow transmitted by the loop path to be closer to 0, reducing the impact of the loop in the generated flow network on the output results of the generated flow model. Among them, the loop path is a path including a loop formed between multiple state nodes, and the acyclic path is a path without a loop formed between multiple nodes.

[0122] Optionally, during the update of the generative flow network, the policy network can be used to determine the probability distribution corresponding to each state node. Specifically, when the generative flow model selects the next action based on each state node, the probability of each action to be selected is determined. The loss function is then calculated based on this probability distribution. Therefore, in the embodiments of the present application, a more reasonable probability distribution for each state node can be learned through the policy network.

[0123] In a possible implementation, in order to facilitate measurement of input / output flows between various state nodes, possible probability distributions between various state nodes may also be determined in the measurement space.

[0124] Alternatively, each state node can be mapped to a point in a measure space. The policy network can output a measure value between each state node in the measure space. This measure value can be used to represent the density of flows between each state node in the measure space. Subsequently, the probability distribution of the state node can be determined based on the measure value of each node. For example, the transition probability between each state node can be determined based on the density of edge flows formed between each state, thereby determining the probability distribution.

[0125] Optionally, at least one path between multiple state nodes, i.e., a path for transitions between state nodes, can be determined based on the calculated probability distribution. The starting and ending points of each path can be determined based on the task performed by the generated flow model. For example, based on the probability distribution, an action with a probability value greater than a preset value or the action with the largest probability value can be selected as the next action for each state node, thereby generating one or more transition paths. The value of the loss function is then calculated based on the at least one path and the input and output flows of each state node.

[0126] For example, for a vehicle driving scenario, the current driving state of the vehicle can be determined as a state node. The probability of each action to be selected when the current node selects the next action from one or more actions is determined based on the probability distribution. After selecting the next action, the next driving state, that is, the next state node, can be entered.

[0127] Furthermore, the loss function can be calculated in multiple dimensions or in multiple ways, such as node flow matching loss, trajectory balance loss, or loss based on the density of nodes or trajectories in continuous space. Therefore, the appropriate loss function calculation method can be selected based on training efficiency or computing power suitable for the actual application scenario, which provides strong generalization capabilities.

[0128] Optionally, when calculating the loss function, the flow matching function and the constraint function can be weightedly fused, or the fusion can be performed by calculating the divergence between the flow matching function and the constraint function. The specific fusion algorithm can be selected based on the actual application scenario, and this application does not limit this. For example, in order to improve the stability of the loss function and thus improve the stability of the model training effect, the fusion can be performed by calculating the divergence between the matching function and the constraint function, thereby improving the stability of the model training.

[0129] 303. Update the generative flow model according to the value of the loss function to obtain an updated generative flow model.

[0130] After calculating the value of the loss function, the generative flow model can be reversely updated based on the value of the loss function to obtain an updated generative flow model.

[0131] Specifically, the method provided in this application can be trained using reinforcement learning during the training phase, that is, during the update, the policy network can be updated so that the policy network can output a more accurate probability distribution and update the generation flow model based on the probability distribution.

[0132] In the implementation of this application, during the update phase of the generative flow model, in addition to the flow matching function based on the input and output flows of each node, a constraint function is added to the loss function. This is equivalent to a regularization constraint on the size of the total edge flow, which helps weaken or terminate loops, thereby reducing the impact of loops on the model output. Therefore, the method provided in this application can be used to train both acyclic and cyclic generative flow models, achieving faster convergence, improving model performance, and having very strong generalization capabilities.

[0133] The method provided in this application is introduced below in conjunction with more specific application scenarios.

[0134] Specifically, the system architecture of the training phase can be shown in FIG4 .

[0135] The system architecture is based on reinforcement learning. For the state generated by the environment, the policy network determines the probability distribution, and then determines the next state node of the current state node based on the probability distribution until the task is completed, forming an acyclic or cyclic graph.

[0136] The loss function may include multiple parts such as a flow matching function and a constraint function as shown in Figure 4. That is, the loss value may be calculated based on the flow matching function and the constraint function.

[0137] The flow matching loss is a loss set for the difference between a node's input and output flows. The constraint function is a constraint set for the looped portion. It can be calculated based on the evaluation value of the output flow of each state node. This evaluation value can include the reward value based on the output flow of each state node in reinforcement learning, as well as the constraint value set for the output flow of each state node. This is equivalent to setting a regularization term in the loss function for the looped portion, thereby regularizing the overall flow, making the model training results more inclined to refer to the output results of the acyclic path, and reducing the size of the loop flow.

[0138] Specifically, in the method provided by this application, the process of calculating the loss function can be divided into multiple parts. For example, the probability distribution corresponding to each state node can be first calculated in the measurement space, and then the value of the loss function can be calculated based on the probability distribution of each state node. For ease of understanding, the embodiment of this application is divided into the following two parts, namely the measurement space projection and the loss function calculation part.

[0139] (1) Measure space projection

[0140] Typically, when running a generative flow model, the transition paths of each state node need to be determined based on a probability distribution. During the training phase, a policy network can be used to generate this probability distribution. To facilitate measuring the output flows of each state node, the probability distribution can be calculated in the measurement space. This means that the density dimension of the measurement space can be used to more intuitively measure the transition status of each state node.

[0141] For example, the state nodes projected into the measure space can be shown in Figure 5, where S0 is the starting point, S f is the endpoint, and A, B, and C are intermediate nodes. The data transmitted through the nodes is called a flow. For example, the output flow of S0 includes three sub-flows as shown in Figure 5, which are flows transmitted between nodes A, B, and C respectively, and can also be called edge flows.

[0142] In the implementation manner of the present application, during the training phase, the policy learning of the policy network can be performed in the measurement space. The policy network can be used to output the measurement value of the flow transmitted between each state node. The measurement value can be used to represent the path formed between the state nodes or the density of the flow in the measurement space, and the values ​​in the measurement space are all non-negative metrics. Subsequently, the probability distribution corresponding to each state node can be determined based on the measurement value, that is, when the state transfer is performed based on each state node, the probability of selecting each action in the action space. In the implementation manner of the present application, the probability distribution can be determined in the measurement space, and the measurement value of the flow between each state node can be output by the policy network, so that the probability of the flow between each state node can be represented by the density of the measurement space. For example, the flow with a larger density value represents a greater probability of transferring to the corresponding state node after selecting the action.

[0143] For example, the policy network can output the measure of transitions between states. In the measure space, the measure of transitions between states can be represented by density. Therefore, in the measure space, this method can be used to determine the possible transition paths between state nodes, or the probability of transitions between states.

[0144] For ease of distinction, edge flows in looped paths can be referred to as zero flows. During reinforcement learning, the environment's reward for zero flows is typically 0. Typically, in a continuous measurement space, cycles are negligible for paths between nodes. For flows with rewards approaching 0, their corresponding density is also relatively low, so the loop is weakened in the measurement space, allowing subsequent termination of the looped flow.

[0145] For example, the measure space is represented as (S, S, F), where F can be understood as the measure in the measure space, i.e., the flow between state nodes is the measure in the measure space. In the measure space, the output flow to A can be defined as: F out (A→S), S∈A, the input flow is represented as: F out (S→A). The constraints for each flow (i.e., the constraints corresponding to the aforementioned constraint function) can be expressed as: R=F(·→sf), and the flow matching constraint can be expressed as: S*Fin=S*Fout. Edge flows in a loop are usually constrained to R=0.

[0146] The probability of transition between state nodes can be determined by using forward or backward Markov chains. For example, the forward probability can be expressed as Among them, F(·→A) represents the flow transferred from the current state node to node A, and F(·→S) represents all output flows of the current node (or any state node). The backward probability can be expressed as: Among them, F(A→·) represents the flow transferred from state node A to the current state node, and F(S→·) represents all flows flowing to the current node.

[0147] (2) Loss function calculation

[0148] After determining the probability distribution of each state node, the probability of state transitions between states can be determined. Therefore, at least one path for transitions between state nodes can be determined based on the probability distribution. A loss function is calculated based on the flows or paths formed between the nodes, allowing for subsequent updates based on the loss function value.

[0149] In this embodiment, the loss function not only calculates the loss value based on the input and output flows of a node, but also adds a constraint term to constrain the loss value of loop paths. This is equivalent to constraining the size of the overall flow, which can prevent overfitting during training and limit the parameters that need to be optimized.

[0150] For example, the loss function can be expressed as: Lα(F)=L(F)+αR(F)

[0151] Among them, F represents the total flow between state nodes, R() is the constraint function, L() is the flow matching function, and α is the weight coefficient.

[0152] Specifically, the constraint function may include a reward for the flow based on the environment, a penalty item set for a flow with a loop, or a value obtained by performing other extended calculations on each flow.

[0153] For example, take the constraint function as an example based on the reward value calculation. If the environment's reward for the flow is higher, the value of the corresponding constraint function will be smaller. If the environment's reward for the flow is lower, the value of the corresponding constraint function will be larger. The penalty term is similar to this. The higher the reward, the smaller the penalty term corresponding to the flow, and the lower the reward, the larger the penalty term corresponding to the flow. It can be understood that the environment usually rewards less for flows with loops, so the greater the loss value set in the loss function, that is, the greater the difference between the output path and the expected output path, indicating that the interaction result between the screened state transition path and the environment is poor. Therefore, the embodiment of the present application constrains the cyclic path so that the final output result is more inclined to be acyclic, thereby reducing the impact of the loop on the model output result.

[0154] Optionally, the constraint term can adopt a regularization constraint, such as constraining the norm of the edge flow matrix, as expressed as:

[0155] This regularizes the overall flow size between state nodes s, so that the final flow converges to a loop-free flow.

[0156] Usually, in order to make the loss value stable, we can define L(F1+F0,…,F p +F0)≥L(F1,…,F p ), where edge flow F0 is defined as a sub-flow of the total edge flow F. That is, if 0 flow does not reduce the loss value, it means that based on (F1,…,F p ) The loss value calculated by the flow is stable. Therefore, for each F i When including 0 flow, the sufficient condition for the loss value to be stable can be expressed as

[0157] Therefore, in order to stabilize the loss value, the weight coefficient α can be set to 0, for 0 flow: Let α n →0 + , and (F n ) n∈N yes The sequence of edge flows when minimizing. If F n Converges to (F n ) n∈N , then F n is an acyclic flow, that is, a flow in a path without a loop.

[0158] Therefore, in the implementation mode of the present application, by setting the constraint item R, the loss value for the 0 flow is not 0, so that even if there is a loop in the path, the loop part can be constrained by the set constraint item, so that the influence of the loop on the output result of the model can be realized, so that the model can finally output a loop-free path.

[0159] Optionally, when calculating the loss function, it can be calculated in a variety of ways, such as calculating the loss value based on node flow matching, based on trajectory balance, or based on the density loss of nodes or trajectories in a continuous space. The specific method can be selected according to the actual application, and this application does not limit it. Among them, based on node flow matching, that is, based on the loss value between the input flow and output flow of each state node, the loss value is calculated based on trajectory balance, that is, based on the trajectory formed by multiple edge flows between each state node, and based on the density of nodes or trajectories in a continuous space, that is, in a continuous space, the loss value can be calculated based on the input flow and output flow of the node or based on the trajectory formed by multiple edge flows between each state node.

[0160] Optionally, in order to further improve the stability of the loss function and accelerate the convergence of the model, the divergence between the flow matching function and the constraint function can be set in the loss function when calculating the loss value.

[0161] For example, the way to calculate the loss function based on divergence can be expressed as: f,g,v(α,β):=∫ x∈A f(α(x)-β(x))g(α(x),β(x))dv(x)

[0162] Among them, in order to make the loss value stable, we can constrain: f≥0, g≥1, f and g are continuously differentiable functions,

[0163] In the embodiments of this application, the training loss can be calculated using a generalized divergence approach, and a regularization constraint can be used to constrain the zero-flow cycle loss. The flow matching loss and the zero-flow cycle loss can be combined to obtain an overall loss value. The policy network can then be updated using a backpropagation algorithm to converge the policy network and obtain a more accurate probability distribution of state transitions for each state node.

[0164] Some possible loss functions are introduced below as examples.

[0165] Example 1: Node flow matching

[0166] That is, the loss value is calculated based on the difference between the input flow and output flow of each state node.

[0167] For the generalized divergence calculation function, you can choose f(x) = log(1+ε|x| α ), g(x,y)=(1+η(x+y)) β , then the loss function can be expressed as:

[0168] ε, η, α, β are all greater than 0, τ represents the trajectory, s t Represents one of the flows in the trajectory. F in Represents the input flow of the state node, F out Represents the output flow of the state node. The trajectory consists of multiple edge flows, which can be understood as a path from s0 to s f Non-negative metrics in the measure space. For example, To represent the trajectory flow, that is, F out (s0) and from s0 to s f Based on π f The result of merging the obtained probability distributions.

[0169] Generally, when sampling a transfer path, if R is not 0, the sampling time for the path τ is limited, and the sampling distribution is usually positively correlated with R. This can be expressed as:

[0170] and

[0171] Therefore, by constraining the sampling duration of the transfer path, it is possible to avoid falling into an infinite sampling loop.

[0172] Therefore, in this example, the loss value can be calculated at the node granularity, thereby realizing the calculation of the loss function based on the generalized divergence, and combined with the regularized constraint term, the overall flow of the path is constrained in units of nodes, so that the flow of the final updated path converges to a non-cyclic flow, realizing the generative flow model training for the cyclic path.

[0173] Example 2: Trajectory Balance Loss

[0174] Compared with the node flow matching mentioned above, Example 2 is equivalent to calculating the loss value based on the path granularity. It can be expressed as:

[0175] Among them, when the forward flow and reverse flow are parameterized by the forward probability strategy and the reverse probability strategy respectively: F f (s t →s t+1 )=F out (s t )×π f (s t →s t+1 ) F b (s t →s t+1 )=F out (s t+1 )×π b (s t →s t+1 )

[0176] Therefore, in the implementation of this application, the balance loss can be calculated at the trajectory granularity to achieve the calculation of the loss value. This allows the calculation of the loss function based on the generalized divergence. In combination with the regularized constraint term, the overall flow of the path is constrained based on the flow in the path, so that the flow of the final updated path converges to a non-cyclic flow, realizing the generative flow model training for the path with cycles.

[0177] Example 3: Based on the density of nodes or trajectories in continuous space

[0178] The continuous space can be the aforementioned measurement space, or other geometric spaces (such as Euclidean space), etc. The appropriate space can be selected according to the actual application scenario, and this application does not limit this.

[0179] The loss function can be expressed as:

[0180] That is, in the continuous space, the input and output flows in the two previous examples are replaced by the measurement density f i and fo , where s t A node can represent a state in a path or a path.

[0181] Therefore, in the method provided in this application, the value of the loss function can be calculated from multiple dimensions, and the stability of the loss function can be improved by generalized divergence. By adding regularization constraints to constrain the size of the overall flow, it is beneficial to weaken the influence of the loop and realize the training of the loop generation flow model.

[0182] 2. Reasoning Stage

[0183] After model training, the generative flow model obtained in the training phase can be used to achieve action prediction.

[0184] See Figure 6, which is a flowchart of an action prediction method of the present application.

[0185] 601. Get status information.

[0186] In the reasoning stage, the intelligent agent can be understood as the device deployed by the generative flow model, or it can be directly understood as the ontology of the task to which the generative flow model is applied, that is, a learner or decision maker.

[0187] For example, when the generative flow model is applied to process autonomous driving tasks, the intelligent agent is the autonomous driving vehicle that deploys the generative flow model; for another example, when the generative flow model is applied to process robot control tasks, the intelligent agent is the robot that deploys the generative flow model; for example, when the generative flow model is applied to process game character control tasks, the intelligent agent is the computer that deploys the generative flow model, and the computer is used to control the game character.

[0188] As the agent performs different actions, it transitions to different states. For example, in an autonomous driving task, after accelerating, the autonomous vehicle moves to a different position and gets closer to the obstacle ahead. Another example is in a robotic control task, after rotating its arm, the arm can be in different positions, and the distance between the arm and the cargo can also change.

[0189] Generally speaking, state information is information used to indicate the state of an agent, where the agent's state can be any state. For example, in an autonomous driving task, state information can include information such as the position of the autonomous vehicle in a map, the distance between the autonomous vehicle and other vehicles, and the distance between the autonomous vehicle and obstacles. Another example is a robotic control task, where state information can include information such as the position of the robot in a warehouse, the distance between the robot and other robots, and the distance between the robot's arm and the cargo.

[0190] Optionally, the state information is obtained based on a state sequence, which indicates a plurality of states in an ordered sequence. In the state sequence, a transition from a previous state to a next state is achieved by performing an action in the previous state. In other words, the state sequence records the plurality of states that the agent transitions to in an ordered sequence.

[0191] Optionally, in a state sequence, a state among multiple states is obtained based on a previous state and an execution action corresponding to the previous state, the execution action corresponding to the previous state is selected from multiple actions sampled in the action space, and the selection probability of the multiple actions is obtained based on the predicted value output by the generative flow model.

[0192] Specifically, the state sequence can be generated by the electronic device based on a generative flow model. In the process of generating the state sequence, the electronic device first randomly samples K actions from the action space corresponding to the current state, and then inputs the state information corresponding to the current state into the generative flow model to obtain K output values ​​of the generative flow model for the K actions in the current state, that is, the predicted value of the generative flow model selecting the K actions in the current state. Secondly, the K output values ​​output by the generative flow model are normalized to calculate the probability value of selecting each of the K actions in the current state. Then, based on the probability value of the K actions, one of the actions is selected, and the next state is transferred based on the current state and the selected action. Similarly, for each state, the electronic device samples multiple actions in the continuous action space, calculates the selection probability of each action in the multiple actions through the generative flow model, and selects an action based on the selection probability of each action to execute the action to obtain the next state. In this way, the electronic device can achieve continuous transition between states by continuously selecting corresponding actions in different states, thereby obtaining a state sequence including multiple states.

[0193] 602. Use the state information as input to generate a flow model to obtain a probability value corresponding to at least one action pair.

[0194] In this embodiment, the generative flow model is used to predict the probability of selecting each action based on the input state information. Simply put, after inputting the first state information into the generative flow model, the generative flow model can predict the predicted values ​​when each given action is selected in the first state. These predicted values ​​can be converted into probabilities for selecting each action.

[0195] For example, assuming that the multiple first actions are action 1-action 5, and the multiple first output values ​​obtained by the generative flow model for action 1-action 5 are 1, 2, 4, 5, and 6 respectively, then the probability values ​​of the intelligent agent selecting action 1-action in the first state can be 1 / (1+2+4+5+6), 2 / (1+2+4+5+6), 4 / (1+2+4+5+6), 5 / (1+2+4+5+6), and 6 / (1+2+4+5+6).

[0196] Among them, the multiple output values ​​obtained by the generative flow model based on the input state information correspond to the multiple actions one-to-one, that is, the multiple output values ​​are used to represent the predicted values ​​when the corresponding action is selected in the state. In addition, the multiple actions corresponding to the multiple output values ​​are sampled in a continuous action space. The actions included in the continuous action space are continuous variables, that is, the multiple actions can be multiple actions sampled within a certain action interval. For example, in an autonomous driving scenario, assuming that the action space where the multiple actions are located is acceleration to 0-100km / h, then by randomly sampling in the action space, a certain number of actions (that is, the above-mentioned multiple actions) can be sampled. For example, assuming that the number of action samples is 5, the multiple actions sampled are acceleration to 5km / h, acceleration to 11km / h, acceleration to 23km / h, acceleration to 55km / h, and acceleration to 88km / h.

[0197] In this embodiment, the number of multiple output values ​​is determined by the number of multiple actions obtained by sampling. The number of multiple actions can be determined based on the capabilities and accuracy requirements of the electronic device for training and generating the flow model. For example, the number of multiple actions is a value such as 1000, 5000 or 10000. This embodiment does not specifically limit the number of multiple actions.

[0198] The training process of the generative flow model can be found in the introduction to the training phase above and will not be repeated here.

[0199] Therefore, in the implementation manner of the present application, the generative flow model can be applied based on both cyclic graphs and acyclic graphs, and has very strong generalization.

[0200] In addition, the present application also provides an action prediction method that can be applied to both the training stage and the inference stage. Specifically, the present application also provides an action prediction method based on the measure space, which can be used in the measure space.

[0201] Please refer to FIG7 , which is a flowchart of another motion prediction method provided by the present application.

[0202] 701. Obtain multiple state nodes in the generated flow model.

[0203] For step 701, please refer to the description of step 301, which will not be repeated here.

[0204] 702. Obtain a measure value for state transition between multiple state nodes in the measure space.

[0205] Each state node can be mapped to a point in the measure space, and the measure value can be used to calculate the probability of each state node in the measure space selecting each action when performing state transition, or the measure value can be used to represent the probability of each state node in the measure space selecting each action when performing state transition.

[0206] In a measure space, the measure value can be used to represent the density of paths formed by state nodes during state transitions. Typically, an action space can be pre-set, which can include multiple actions. For example, in an autonomous driving scenario, the action space can be set to discrete or continuous actions for accelerating to 0-100 km / h, or discrete or continuous actions for turning 0-90 degrees. The measure value can then be used to determine the probability of selecting each action.

[0207] 703. Generate at least one path according to the measurement value.

[0208] In the measure space, the size of the flow between nodes, or density, is related to the measure value of each node. For example, the larger the measure value, the greater the density in the measure space. Therefore, based on the density of paths between state nodes, at least one path can be generated, i.e., the path for state transitions between multiple state nodes.

[0209] If the method provided in the embodiment of the present application is applied to the training stage, the probability distribution can be determined based on the measurement value output by the policy network. The probability distribution can be used to determine the probability of at least one action to be selected when selecting the next action based on each state node in multiple state nodes, that is, the probability of selecting each action when the state is transferred based on each state node, and the loss is calculated in the subsequent stage, so as to update the generation flow model based on the results of the policy network.

[0210] If the method provided in this application is applied to the reasoning stage, at least one path corresponding to the specific task output can be generated and subsequent actions can be executed, thereby realizing action prediction of state transition in the measurement space.

[0211] Therefore, in the embodiments of the present application, the prediction of the state transition path can be achieved in the measurement space, so that based on the measurable space, the state transition path can be determined more accurately and efficiently.

[0212] The above describes the process of the method provided in this application. The following describes the structure of the device for executing the method provided in this application.

[0213] Refer to FIG8 , which is a schematic structural diagram of a training device for generating a model provided in the present application, including the following.

[0214] An acquisition module 801 is used to acquire multiple state nodes in a generated flow model;

[0215] A calculation module 802 is configured to obtain a value of a loss function based on multiple state nodes, where the value of the loss function includes a fusion value of a flow matching function and a constraint function, the value of the flow matching function includes a value calculated based on the input flow and output flow of each state node in the multiple state nodes, and the value of the constraint function includes a value calculated based on the evaluation values ​​corresponding to the output flows of the multiple state nodes, wherein the value of the constraint function corresponding to the output flow in a cyclic path is different from the value of the constraint function corresponding to the output flow in an acyclic path, the cyclic path includes a path forming a loop between state nodes, and the acyclic path is a path formed between state nodes that does not include a loop;

[0216] The updating module 803 is configured to update the generative flow model according to the value of the loss function to obtain an updated generative flow model.

[0217] In one possible implementation, the aforementioned calculation module 802 is specifically used to: determine the probability distribution corresponding to multiple state nodes based on the policy network, the probability distribution is used to determine the probability of at least one action to be selected when selecting the next action based on each state node; calculate the value of the loss function based on the probability distribution and multiple state nodes.

[0218] In one possible implementation, the aforementioned calculation module 802 is specifically used to: output a measurement value of a flow transmitted between multiple state nodes in a measurement space according to a policy network, wherein the measurement value represents the density of a path formed by multiple state nodes in the measurement space; and determine a probability distribution based on the measurement value between multiple state nodes.

[0219] In one possible implementation, the aforementioned calculation module 802 is specifically used to: determine at least one path between multiple state nodes based on a probability distribution, where the starting point and end point of the at least one path are determined by the task performed according to the generation flow model; and calculate the value of the loss function based on the at least one path and the input flow and output flow of each state node.

[0220] In one possible implementation, the aforementioned constraint function includes a function based on the norm of an edge flow matrix, where the edge flow matrix is ​​a matrix formed by edge flows between intermediate state nodes among multiple state nodes, and the edge flow is the flow transmitted between a pair of state nodes.

[0221] In a possible implementation, the aforementioned loss function includes a function formed by the divergence between the flow matching function and the constraint function.

[0222] In a possible implementation, the aforementioned loss function includes a function based on node flow matching loss, a function based on trajectory balance loss, a function based on node density loss in a continuous space, or a function based on trajectory density loss in a continuous space.

[0223] Referring to FIG9 , the present application provides a schematic structural diagram of an action prediction device, including the following:

[0224] Acquisition module 901, used to obtain state information, the state information is used to indicate the current state of the agent;

[0225] A generation module 902 is configured to use the state information as input to generate a flow model to obtain a probability value corresponding to at least one action, wherein the probability value corresponding to the at least one action is used to determine an action to be performed by the agent;

[0226] The generative model is obtained by training using the method described in the aforementioned training phase.

[0227] Referring to FIG10 , the present application provides a schematic structural diagram of an action prediction device, including the following:

[0228] An acquisition module 1001 is used to acquire multiple state nodes in a generated flow model;

[0229] A measurement module 1002 is configured to obtain measurement values ​​for a plurality of state nodes, the measurement values ​​representing the density of paths formed by the plurality of state nodes during state transitions in a measurement space, and the measurement values ​​are used to determine a probability distribution, which is used to determine the probability of at least one action being selected when selecting the next action based on each of the plurality of state nodes.

[0230] Determination module 1003 is used to determine at least one path based on the measurement values ​​of multiple state nodes, where the at least one path includes a path formed by transferring actions between multiple state nodes based on probability distribution, where the starting point and end point of each path are determined by the tasks executed according to the generation flow model.

[0231] In a possible implementation, the apparatus further includes an updating module 1004 configured to: calculate a loss value according to at least one path; and update the generated flow model according to the loss value to obtain an updated generated flow model.

[0232] As shown in Figure 11, it is a schematic diagram of the hardware structure of a training device 110 for generating a flow model provided by an embodiment of the present application. The training device 110 for generating a flow model can be used to implement the steps of the method in the aforementioned training phase.

[0233] The training apparatus 110 for generating a flow model shown in FIG11 may include a processor 1101 , a memory 1102 , a communication interface 1103 , and a bus 1104 . The processor 1101 , the memory 1102 , and the communication interface 1103 may be connected via the bus 1104 .

[0234] Processor 1101 is the control center of the training device 110 for generating the flow model. It can be a general-purpose central processing unit (CPU) or other general-purpose processor. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adaptively configured according to the actual application scenario.

[0235] As an example, the processor 1101 may include one or more CPUs, and may also include other processors, such as the CPU, NPU, or GPU shown in Figure 11.

[0236] The memory 1102 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0237] In one possible implementation, the memory 1102 may exist independently of the processor 1101. The memory 1102 may be connected to the processor 1101 via a bus 1104 and used to store data, instructions, or program codes. When the processor 1101 calls and executes the instructions or program codes stored in the memory 1102, the method provided in the embodiment of the present application can be implemented, for example, the training method for generating a flow model shown in the training phase.

[0238] In another possible implementation, the memory 1102 may also be integrated with the processor 1101 .

[0239] Communication interface 1103 is used to connect the training device 110 for generating a flow model to other devices via a communication network, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 1103 may include a receiving unit for receiving data and a sending unit for sending data.

[0240] Bus 1104 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, FIG11 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0241] It should be pointed out that the structure shown in Figure 11 does not constitute a limitation on the training device 110 for generating a flow model. In addition to the components shown in Figure 11, the training device 110 for generating a flow model may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0242] Figure 12 is a schematic diagram of the hardware structure of an action prediction device 120 provided in an embodiment of the present application. The action prediction device 120 can be used to implement the steps of the method in the aforementioned reasoning stage.

[0243] The motion prediction device 120 shown in FIG12 may include a processor 1201 , a memory 1202 , a communication interface 1203 , and a bus 1204 . The processor 1201 , the memory 1202 , and the communication interface 1203 may be connected via the bus 1204 .

[0244] The processor 1201 is the control center of the motion prediction device 120 and can be a general-purpose central processing unit (CPU) or other general-purpose processor. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adaptively configured according to the actual application scenario.

[0245] As an example, the processor 1201 may include one or more CPUs, and may also include other processors, such as the CPU, NPU, or GPU shown in Figure 12.

[0246] The memory 1202 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0247] In one possible implementation, memory 1202 may exist independently of processor 1201. Memory 1202 may be connected to processor 1201 via bus 1204 and used to store data, instructions, or program code. When processor 1201 calls and executes the instructions or program code stored in memory 1202, the methods provided in the embodiments of the present application, such as the action prediction method shown in the inference stage, can be implemented.

[0248] In another possible implementation, the memory 1202 may also be integrated with the processor 1201 .

[0249] Communication interface 1203 is used to connect the motion prediction device 120 to other devices via a communication network. The communication network can be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. Communication interface 1203 can include a receiving unit for receiving data and a sending unit for sending data.

[0250] Bus 1204 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, FIG12 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0251] It should be pointed out that the structure shown in Figure 12 does not constitute a limitation on the action prediction device 120. In addition to the components shown in Figure 12, the action prediction device 120 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0252] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., and includes a number of instructions for enabling a data quantization device (which can be a personal computer, server, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0253] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0254] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for training a model or performing an inference task. When the program is run on a computer, the computer executes all or part of the steps in the method described in the embodiments shown in Figures 4 to 12 above.

[0255] The present application also provides a digital processing chip. The digital processing chip integrates circuitry and one or more interfaces for implementing the aforementioned processor or processor functions. When the digital processing chip integrates memory, it can perform the method steps of any one or more of the aforementioned embodiments. When the digital processing chip does not integrate memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the aforementioned embodiments based on program code stored in the external memory.

[0256] A computer program product is also provided in the embodiment of the present application, and the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0257] The target sensing device or target sensing device provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the server executes the method described in the embodiments shown in Figures 4 to 7 above. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device end, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0258] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0259] For example, see Figure 13, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 130. NPU 130 is mounted on the host CPU (host CPU) as a coprocessor and assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1303, which is controlled by controller 1304 to extract matrix data from memory and perform multiplication operations.

[0260] In some implementations, arithmetic circuit 1303 includes multiple processing engines (PEs). In some implementations, arithmetic circuit 1303 is a two-dimensional systolic array. Arithmetic circuit 1303 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1303 is a general-purpose matrix processor.

[0261] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 1302 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 1301 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 1308.

[0262] Unified memory 1306 is used to store input and output data. Weight data is directly transferred to weight memory 1302 via direct memory access controller (DMAC) 1305. Input data is also transferred to unified memory 1306 via DMAC.

[0263] The bus interface unit (BIU) 1310 is used for interaction between the AXI bus, the DMAC, and the instruction fetch buffer (IFB) 1309 .

[0264] The bus interface unit 1310 (BIU) is used for the instruction fetch memory 1309 to obtain instructions from the external memory, and is also used for the storage unit access controller 1305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0265] DMAC is mainly used to move input data in the external memory DDR to the unified memory 1306 or to move weight data to the weight memory 1302 or to move input data to the input memory 1301.

[0266] The vector calculation unit 1307 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0267] In some implementations, the vector calculation unit 1307 can store the processed output vector to the unified memory 1306. For example, the vector calculation unit 1307 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1303, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1307 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1303, such as for use in subsequent layers in a neural network.

[0268] An instruction fetch buffer 1309 connected to the controller 1304 is used to store instructions used by the controller 1304;

[0269] Unified memory 1306, input memory 1301, weight memory 1302, and instruction fetch memory 1309 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0270] The operations of each layer in the recurrent neural network can be performed by the operation circuit 1303 or the vector calculation unit 1307.

[0271] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the program of the method of Figures 4 to 7 above.

[0272] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0273] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0274] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0275] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0276] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when implementing in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some ports, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application.

Claims

1. A training method for generating a flow model, characterized in that: The method comprises: Get multiple state nodes in the generated flow model; Obtain a value of a loss function according to the multiple state nodes, the value of the loss function includes a value fused from a value of a flow matching function and a value of a constraint function, the value of the flow matching function includes a value calculated based on an input flow and an output flow of each state node in the multiple state nodes, the value of the constraint function includes a value calculated from an evaluation value corresponding to an output flow of the multiple state nodes, wherein a value of a constraint function corresponding to an output flow of a state node in a cyclic path is different from a value of a constraint function corresponding to an output flow of a state node in an acyclic path, the cyclic path includes a path forming a loop between the multiple state nodes, and the acyclic path is a path formed between the multiple state nodes that does not include a loop; The generative flow model is updated according to the value of the loss function to obtain an updated generative flow model.

2. The method according to claim 1, characterized in that The obtaining the value of the loss function according to the multiple state nodes includes: Determine, according to the policy network, probability distributions corresponding to the plurality of state nodes, wherein the probability distributions are used to determine a probability of at least one action to be selected being selected when selecting a next action based on each state node; The value of the loss function is calculated according to the probability distribution and the multiple state nodes.

3. The method according to claim 2, characterized in that The determining, according to the policy network, probability distributions corresponding to the plurality of state nodes comprises: Outputting a measurement value of a flow transmitted between the plurality of state nodes in a measurement space according to the strategy network, wherein the measurement value represents a density of the flow transmitted between the plurality of state nodes in the measurement space; The probability distribution is determined according to the measurement values ​​between the multiple state nodes.

4. The method according to claim 3, characterized in that: The calculating the value of the loss function according to the probability distribution and the multiple state nodes includes: Determine at least one path between the plurality of state nodes according to the probability distribution, wherein a starting point and an end point of the at least one path are determined according to a task executed according to the generation flow model; The value of the loss function is calculated according to the at least one path and the input flow and the output flow of each state node.

5. The method according to any one of claims 1 to 4, characterized in that The constraint function includes a function composed of norms based on an edge flow matrix, wherein the edge flow matrix is ​​a matrix formed by edge flows between intermediate state nodes among the multiple state nodes, and the edge flow is a flow transmitted between any pair of nodes among the multiple state nodes.

6. The method according to any one of claims 1 to 5, characterized in that The loss function includes a function formed by a divergence between the flow matching function and the constraint function.

7. The method according to any one of claims 1 to 6, characterized in that The loss function includes a function based on node flow matching loss, a function based on trajectory balance loss, a function based on density loss of nodes in continuous space, or a function based on density loss of trajectories in continuous space.

8. A method for motion prediction, characterized in that: include: Acquire state information, where the state information is used to indicate the current state of the agent; Using the state information as an input of a generative flow model to obtain a probability value corresponding to at least one action, wherein the probability value corresponding to the at least one action is used to determine an action performed by the agent; Wherein, the generative flow model is obtained by training through the method described in any one of claims 1-7.

9. A method for motion prediction, characterized in that: include: Get multiple state nodes in the generated flow model; Obtaining measurement values ​​of the multiple state nodes, the measurement values ​​representing the density of paths formed by the multiple state nodes when performing state transitions in the measurement space, the measurement values ​​being used to determine a probability distribution, the probability distribution being used to determine the probability of at least one action to be selected being selected when selecting the next action based on each state node in the multiple state nodes; At least one path is generated according to the measurement values ​​of the multiple state nodes, and the at least one path includes a path formed by transferring between the multiple state nodes based on the actions selected based on the probability distribution. The starting point and the end point of the at least one path are determined by the tasks executed according to the generated flow model.

10. The method according to claim 9, characterized in that The method further comprises: Calculating a loss value according to the at least one path; The generative flow model is updated according to the loss value to obtain an updated generative flow model.

11. A training device for generating a flow model, characterized in that: The device comprises: An acquisition module, used for acquiring multiple state nodes in the generated flow model; A calculation module, used for obtaining a value of a loss function according to the multiple state nodes, the value of the loss function includes a value fused from a flow matching function and a constraint function, the value of the flow matching function includes a value calculated based on an input flow and an output flow of each state node in the multiple state nodes, the value of the constraint function includes a value calculated from an evaluation value corresponding to an output flow of the multiple state nodes, wherein a value of the constraint function corresponding to an output flow of a state node in a cyclic path is different from a value of the constraint function corresponding to an output flow of a state node in an acyclic path, the cyclic path includes a path forming a loop between the multiple state nodes, and the acyclic path is a path formed between the multiple state nodes that does not include a loop; An updating module is used to update the generative flow model according to the value of the loss function to obtain an updated generative flow model.

12. The device according to claim 11, characterized in that The computing module is specifically used for: Determine, according to the policy network, probability distributions corresponding to the plurality of state nodes, wherein the probability distributions are used to determine a probability of at least one action to be selected being selected when selecting a next action based on each state node; The value of the loss function is calculated according to the probability distribution and the multiple state nodes.

13. The device according to claim 12, characterized in that The computing module is specifically used for: Outputting a measurement value of a flow transmitted between the plurality of state nodes in a measurement space according to the strategy network, wherein the measurement value represents a density of the flow transmitted between the plurality of state nodes in the measurement space; The probability distribution is determined according to the measurement values ​​between the multiple state nodes.

14. The device according to claim 13, characterized in that The computing module is specifically used for: Determine at least one path between the plurality of state nodes according to the probability distribution, wherein a starting point and an end point of the at least one path are determined according to a task executed according to the generation flow model; The value of the loss function is calculated according to the at least one path and the input flow and the output flow of each state node.

15. The device according to any one of claims 11 to 14, characterized in that The constraint function includes a function composed of norms based on an edge flow matrix, wherein the edge flow matrix is ​​a matrix formed by edge flows between intermediate state nodes among the multiple state nodes, and the edge flow is a flow transmitted between any pair of nodes among the multiple state nodes.

16. The device according to any one of claims 11 to 15, characterized in that The loss function includes a function formed by a divergence between the flow matching function and the constraint function.

17. The device according to any one of claims 11 to 16, characterized in that The loss function includes a function based on node flow matching loss, based on trajectory balance loss, based on density loss of nodes in continuous space, or based on density loss of trajectories of nodes in continuous space.

18. An action prediction device, characterized in that: include: An acquisition module, used to acquire state information, wherein the state information is used to indicate the current state of the agent; A generation module, used to use the state information as an input of a generation flow model to obtain a probability value corresponding to at least one action, wherein the probability value corresponding to the at least one action is used to determine an action performed by the agent; Wherein, the generation model is obtained by training through the method according to any one of claims 1-7.

19. An action prediction device, characterized in that: include: An acquisition module, used for acquiring multiple state nodes in the generated flow model; A measurement module, used to obtain measurement values ​​of the multiple state nodes, the measurement values ​​representing the density of paths formed by the multiple state nodes when performing state transitions in the measurement space, the measurement values ​​being used to determine a probability distribution, and the probability distribution being used to determine a probability of at least one action to be selected being selected when selecting the next action based on each state node in the multiple state nodes; A determination module is used to determine at least one path based on the measurement values ​​of the multiple state nodes, the at least one path includes a path formed by transferring between the multiple state nodes based on the actions selected based on the probability distribution, and the starting point and end point of the at least one path are determined by the tasks executed according to the generation flow model.

20. The device according to claim 19, characterized in that The device also includes an updating module, which is used to: Calculating a loss value according to the at least one path; The generative flow model is updated according to the loss value to obtain an updated generative flow model.

21. A training device for generating a flow model, characterized in that: The device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 1 to 7.

22. A training device for generating a flow model, characterized in that: The device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 8 to 10.

23. A computer storage medium, characterized in that The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 10.

24. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 10.

Images