Model training method and related device thereof

Abstract

The application discloses a model training method and a related device, which are used for training a generative flow model in an offline training mode, so that the generative flow model has better performance. The method comprises the following steps: when a to-be-trained model needs to be trained, first information of an agent is acquired from a preset offline data set, and the first information is used for indicating that the agent is in a target state. Then, the first information is input into the to-be-trained model, so that the first information is processed by the to-be-trained model, and a probability of occurrence of a first action of the agent is obtained, the first action of the agent being used for enabling the agent to enter a next state of the target state from the target state. Finally, the to-be-trained model is trained based on the probability of occurrence of the first action of the agent and a real probability of occurrence of the first action derived from the offline data set, so that a generative flow model is obtained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model training method and related equipment. Background Technology

[0002] With the rapid development of AI technology, generative flow models are widely used to describe and solve the action strategy selection of intelligent agents in the process of interacting with the environment, so that the intelligent agents can maximize the reward or achieve specific goals after performing corresponding actions.

[0003] Currently, the generative flow models provided by related technologies, after determining that an agent is in a target state, can process information associated with that target state to predict the probability of one or more actions taken by the agent. These actions are used to propel the agent from the target state to one or more subsequent states. In this way, the agent can execute the action with the highest probability predicted by the neural network model, thereby entering a subsequent state of the target state.

[0004] The aforementioned generative flow models typically employ online training. This means that during model training, for any given state of the agent, the model can apply the predicted action to that state within an environment simulator, thereby randomly generating the agent's next state. While this training method allows the model to learn as many states as possible, some states may not accurately reflect the agent's actual environment, resulting in relatively mediocre performance of the trained generative flow model. Summary of the Invention

[0005] This application provides a model training method and related equipment, which trains a generative stream model in an offline training mode, thereby enabling the generative stream model to have better performance.

[0006] A first aspect of this application provides a model training method, the method comprising:

[0007] When it is necessary to train the model to be trained, a pre-set offline dataset can be obtained first, and the first information can be extracted from the offline dataset. The first information is used to indicate that the agent is in the target state.

[0008] After obtaining the initial information, it can be input into the model to be trained. The model processes the initial information to obtain the (predicted) probability of the agent's first action. This first action is used to move the agent from the target state to the next state. At this point, the model to be trained has completed the action prediction for the target state. In one possible implementation, when obtaining the probability of the agent's first action, the model to be trained can, as far as possible, adhere to the following constraint: the difference between the probability of the agent's first action and the actual probability of the agent's first action is within a preset range, where the actual probability of the agent's first action can be extracted from an offline dataset.

[0009] After obtaining the probability of the agent's first action, the model to be trained can be trained based on the probability of the agent's first action until the model training conditions are met, thereby obtaining the generative flow model.

[0010] As can be seen from the above method, when training the model to be trained, the first information of the agent can be obtained from a pre-set offline dataset. This first information indicates that the agent is in the target state. Then, the first information can be input into the model to be trained, allowing the model to process it and obtain the probability of the agent's first action. This first action causes the agent to transition from the target state to the next state. Finally, based on the probability of the agent's first action and its actual probability of occurrence, the model to be trained is trained to obtain a generative flow model. The actual probability of the first action originates from the offline dataset. In the aforementioned process, the probability of the agent's first action can be called the predicted action policy of the model to be trained for the target state, and the actual probability of the agent's first action can be called the actual action policy of the target state in the offline database. This allows the predicted action policy for the target state to be as close as possible to the actual action policy for the target state. The actual action policy for the target state determines the actual probability of the agent entering the next state from the target state. Therefore, the model to be trained can not only learn as many next states of the target state as possible, but also the learned states are sufficiently consistent with the actual environment in which the agent is located (because the data in the offline dataset are all pre-set based on the actual environment in which the agent is located). Thus, the generative stream model trained in the offline training mode can have better performance.

[0011] In one possible implementation, training the model to be trained based on the probability of the first action to obtain a generative flow model includes: correcting the probability of the agent's second action based on an offline dataset to obtain a corrected probability of the second action, which is used to cause the agent to enter the target state from the previous state; correcting the reward value corresponding to the target state based on the offline dataset to obtain a corrected reward value corresponding to the target state; and training the model to be trained based on the probability of the first action, the corrected probability of the second action, and the corrected reward value corresponding to the target state to obtain the generative flow model. In the aforementioned implementation, after obtaining the probability of the agent's first action, the probability of the agent's second action can also be obtained. The agent's second action is used to cause the agent to enter the target state from the previous state. It should be noted that since the model to be trained has already completed the action prediction for the previous state of the target state, the probability of the agent's second action can be obtained directly. Therefore, some data in the offline dataset can be used to correct the probability of the agent's second action to obtain the corrected probability of the agent's second action. After obtaining the probability of the agent's first action, the reward value of the target state object can be obtained from the offline dataset. This reward value can then be corrected using data from the offline dataset, resulting in a corrected reward value for the target state. Once the corrected probability of the agent's second action and the corrected reward value for the target state are obtained, the model to be trained can be built using these factors, thus producing a generative flow model.

[0012] In one possible implementation, the offline dataset includes M first candidate information and M second candidate information. The i-th first candidate information indicates that the agent is in the i-th candidate state, and the i-th second candidate information indicates that the agent is in the previous state of the i-th candidate state. The M first candidate information includes first information, and the M second candidate information includes second information. The second information indicates that the agent is in the previous state of the target state, and the M candidate states include the target state, where M ≥ 1. Based on the offline dataset, the probability of the agent's second action is corrected to obtain the corrected probability of the second action. This correction involves adjusting the probability of the agent's second action based on the first information, the second information, the M first candidate information, and the M second candidate information. In the aforementioned implementation, the offline dataset contains M data groups. The first data group contains the first first candidate information, the first second candidate information, the first third candidate information, the reward value corresponding to the first candidate state, and the first real action policy. Similarly, the Mth data set contains the Mth first candidate information, the Mth second candidate information, the Mth third candidate information, the reward value corresponding to the Mth candidate state, and the Mth real action policy. We can then select one of the M data sets, designating the first candidate information in that set as the first information, the candidate state indicated by the first candidate information as the target state, and the second candidate information as the second information. Thus, the first information indicates that the agent is in the target state, the second information indicates that the agent is in the state preceding the target state, and the actual probability of the agent's first action is known (derived from the corresponding real action policy). We can then extract M first candidate information and M second candidate information from the offline database, and calculate the new transition value for the target state using the first information, the second information, and the M first and M second candidate information. Finally, we use this new transition value for the target state to correct the probability of the agent's second action, thus obtaining the corrected probability of the second action.

[0013] In one possible implementation, the offline dataset also includes reward values ​​corresponding to M candidate states. Based on the offline dataset, the reward value corresponding to the target state is corrected to obtain the corrected reward value for the target state. This involves correcting the reward value for the target state based on the first information, the M first candidate information pieces, and the reward values ​​corresponding to the M candidate states. Alternatively, in the aforementioned implementation, the M first candidate information pieces and the reward values ​​corresponding to the M candidate states can be extracted from the offline database, and calculations can be performed on these pieces to obtain the corrected reward value for the target state.

[0014] In one possible implementation, the first information is information collected when the agent is in the target state, and the information includes at least one of the following: image, video, audio, or text.

[0015] The second aspect of this application provides an action prediction method, which is implemented by a generative flow model in the first aspect or any possible implementation of the first aspect. The method includes: acquiring information of an agent, the information indicating that the agent is in a target state; processing the information through a model to be trained to obtain the probability of an action of the agent, the action being used to cause the agent to move from the target state to the next state of the target state.

[0016] As can be seen from the above method, when the agent is currently in the target state, in order to predict the action that will lead it to the next state from the target state, the agent can first collect information indicating that it is in the target state. After obtaining its own information, the agent can input this information into a generative flow model, which processes the information to obtain the probability of one or more actions. Then, the agent can select the action with the highest probability from among the one or more actions and execute that action to enter a specific next state of the target state. In the aforementioned process, because the agent has a built-in generative flow model, it can accurately complete the transitions between different states based on the completion of action prediction and action execution.

[0017] A third aspect of this application provides an action prediction method, which includes: acquiring information about an agent, the information indicating that the agent is in a target state; processing the information using a model to be trained to obtain the probability of an action occurring in the agent, the action being used to cause the agent to move from the target state to the next state of the target state; and determining the action to be executed based on the probability of the action occurring and a preset probability of the action occurring.

[0018] As can be seen from the above method, when the agent is currently in the target state, in order to predict the action that will lead it to the next state from the target state, the agent can first collect information indicating that it is in the target state. After obtaining its own information, the agent can input this information into a generative flow model, which processes the information to obtain the probability of one or more actions. Then, the agent can select the action with the highest probability from among the one or more actions and execute that action to enter a specific next state of the target state. In the aforementioned process, because the agent has a built-in generative flow model, it can accurately complete the transitions between different states based on the completion of action prediction and action execution.

[0019] In one possible implementation, the probability of an action occurring is based on the probability of a preset action occurring. Determining the action to be executed includes: among the actions and preset actions, identifying the action with the highest probability of occurrence as the action to be executed.

[0020] A fourth aspect of this application provides a model training apparatus, comprising: an acquisition module for acquiring first information of an agent from a preset offline dataset, the first information indicating that the agent is in a target state; a processing module for processing the first information using a model to be trained to obtain the probability of occurrence of a first action of the agent, the first action causing the agent to move from the target state to the next state of the target state; and a training module for training the model to be trained based on the probability of occurrence of the first action to obtain a generative flow model, wherein the actual occurrence probability is derived from the offline dataset.

[0021] As can be seen from the above apparatus, when training the model to be trained, the first information of the agent can be obtained from a pre-set offline dataset. This first information indicates that the agent is in a target state. Then, the first information can be input into the model to be trained, allowing the model to process the information and obtain the probability of the agent's first action. This first action causes the agent to transition from the target state to the next state. Finally, based on the probability of the agent's first action and its actual probability of occurrence, the model to be trained is trained to obtain a generative flow model. The actual probability of the first action originates from the offline dataset. In the aforementioned process, the probability of the agent's first action can be called the predicted action policy of the model to be trained for the target state, and the actual probability of the agent's first action can be called the actual action policy of the target state in the offline database. This allows the predicted action policy for the target state to be as close as possible to the actual action policy for the target state. The actual action policy for the target state determines the actual probability of the agent entering the next state from the target state. Therefore, the model to be trained can not only learn as many next states of the target state as possible, but also the learned states are sufficiently consistent with the actual environment in which the agent is located (because the data in the offline dataset are all pre-set based on the actual environment in which the agent is located). Thus, the generative stream model trained in the offline training mode can have better performance.

[0022] In one possible implementation, a training module is used to train the model to be trained based on the probability of the occurrence of the first action, such that the difference between the probability of the occurrence of the first action and the actual probability of the occurrence of the first action is within a preset range, thereby obtaining a generative flow model.

[0023] In one possible implementation, the training module is used to: correct the occurrence probability of the agent's second action based on an offline dataset to obtain a corrected occurrence probability of the second action, which is used to cause the agent to enter the target state from the previous state of the target state; correct the reward value corresponding to the target state based on the offline dataset to obtain a corrected reward value corresponding to the target state; and train the model to be trained based on the occurrence probability of the first action, the corrected occurrence probability of the second action, and the reward value corresponding to the target state to obtain a generative flow model.

[0024] In one possible implementation, the offline dataset includes M first candidate information and M second candidate information. The i-th first candidate information is used to indicate that the agent is in the i-th candidate state, and the i-th second candidate information is used to indicate that the agent is in the previous state of the i-th candidate state. The M first candidate information includes first information, and the M second candidate information includes second information. The second information is used to indicate that the agent is in the previous state of the target state, and the M candidate states include the target state, M≥1. The training module is used to correct the probability of the agent's second action based on the first information, the second information, the M first candidate information, and the M second candidate information to obtain the corrected probability of the second action.

[0025] In one possible implementation, the offline dataset also includes reward values ​​corresponding to M candidate states. A training module is used to correct the reward value corresponding to the target state based on the first information, the M first candidate information, and the reward values ​​corresponding to the M candidate states, so as to obtain the corrected reward value corresponding to the target state.

[0026] In one possible implementation, the first information is information collected when the agent is in the target state, and the information includes at least one of the following: image, video, audio, or text.

[0027] A fifth aspect of this application provides an action prediction device, which includes a generative flow model in the third aspect or any possible implementation of the third aspect. The device includes: an acquisition module for acquiring information of an agent, the information indicating that the agent is in a target state; and a processing module for processing the information through a model to be trained to obtain the probability of an action of the agent, the action being used to cause the agent to move from the target state to the next state of the target state.

[0028] As can be seen from the above device, when the agent is currently in the target state, in order to predict the action that will lead it to the next state from the target state, the agent can first collect information indicating that it is in the target state. After obtaining its own information, the agent can input its own information into a generative flow model, which processes the information to obtain the probability of one or more actions. Then, the agent can select the action with the highest probability from the one or more actions and execute that action to enter a certain next state of the target state. In the aforementioned process, because the agent has a built-in generative flow model, it can accurately complete the transition between different states based on the completion of action prediction and action execution.

[0029] A sixth aspect of this application provides an action prediction device, comprising: an acquisition module for acquiring information about an agent, the information indicating that the agent is in a target state; a processing module for processing the information using a model to be trained to obtain the probability of an action occurring in the agent, the action causing the agent to move from the target state to the next state of the target state; and a determination module for determining an action to be executed based on the probability of the action occurring and a preset probability of the action occurring.

[0030] As can be seen from the above device, when the agent is currently in the target state, in order to predict the action that will lead it to the next state from the target state, the agent can first collect information indicating that it is in the target state. After obtaining its own information, the agent can input its own information into a generative flow model, which processes the information to obtain the probability of one or more actions. Then, the agent can select the action with the highest probability from the one or more actions and execute that action to enter a certain next state of the target state. In the aforementioned process, because the agent has a built-in generative flow model, it can accurately complete the transition between different states based on the completion of action prediction and action execution.

[0031] In one possible implementation, the probability of an action occurring is based on the probability of a preset action occurring. Determining the action to be executed includes: among the actions and preset actions, identifying the action with the highest probability of occurrence as the action to be executed.

[0032] A seventh aspect of this application provides a model training apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training apparatus performs the method described in the first aspect or any possible implementation thereof.

[0033] An eighth aspect of this application provides an action prediction apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the action prediction apparatus performs the method described in the second aspect, the third aspect, or any possible implementation of the third aspect.

[0034] A ninth aspect of this application provides a circuit system including a processing circuit configured to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, the third aspect, or any possible implementation of the third aspect.

[0035] A tenth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory, such that the processor performs the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, the third aspect, or any possible implementation of the third aspect.

[0036] In one possible implementation, the processor is coupled to the memory via an interface.

[0037] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0038] The eleventh aspect of this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to implement the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, the third aspect, or any possible implementation of the third aspect.

[0039] The twelfth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, the third aspect, or any possible implementation of the third aspect.

[0040] In this embodiment, when training the model to be trained is required, first information about the agent can be obtained from a pre-set offline dataset. This first information indicates that the agent is in a target state. Then, the first information can be input into the model to be trained, allowing the model to process the information and obtain the probability of the agent's first action. This first action causes the agent to transition from the target state to the next state. Finally, based on the probability of the agent's first action, the model to be trained can be trained to obtain a generative stream model. In the aforementioned process, when the model to be trained acquires the probability of the agent's first action, it ensures that the difference between the probability of the agent's first action and the actual probability of the agent's first action are within a preset range. Since the probability of the agent's first action can be called the predicted action strategy of the model to be trained for the target state, and the actual probability of the agent's first action can be called the actual action strategy for the target state in the offline database, this allows the predicted action strategy for the target state to be as close as possible to the actual action strategy for the target state. The actual action strategy for the target state determines the actual probability of the agent entering the next state from the target state. Therefore, the model to be trained can not only learn as many next states as possible for the target state, but also the learned states are sufficiently consistent with the actual environment in which the agent is located (because the data in the offline dataset are all pre-set based on the actual environment in which the agent is located). Thus, the generative stream model trained in the offline training mode can have better performance. Attached Figure Description

[0041] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0042] Figure 2a A schematic diagram of the structure of the motion prediction system provided in the embodiments of this application;

[0043] Figure 2b This is another structural schematic diagram of the motion prediction system provided in the embodiments of this application;

[0044] Figure 2c A schematic diagram of a motion prediction device provided in an embodiment of this application;

[0045] Figure 3 A schematic diagram of the system 100 architecture provided in the embodiments of this application;

[0046] Figure 4 A schematic diagram of the generation flow model provided in the embodiments of this application;

[0047] Figure 5a A schematic flowchart of the modeling method provided in the embodiments of this application;

[0048] Figure 5b A schematic diagram illustrating an application example of the model training method provided in this application embodiment;

[0049] Figure 6 A flowchart illustrating the motion prediction method provided in this application embodiment;

[0050] Figure 7 This is another structural schematic diagram of the generation flow model provided in the embodiments of this application;

[0051] Figure 8 A schematic diagram of the structure of the model training apparatus provided in the embodiments of this application;

[0052] Figure 9a A schematic diagram of the motion prediction device provided in the embodiments of this application;

[0053] Figure 9b Another schematic diagram of the motion prediction device provided in the embodiments of this application;

[0054] Figure 10 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0055] Figure 11 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0056] Figure 12 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0057] This application provides a model training method and related equipment, which trains a generative stream model in an offline training mode, thereby enabling the generative stream model to have better performance.

[0058] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0059] With the rapid development of AI technology, generative flow models are widely used to describe and solve the action strategy selection of intelligent agents in the process of interacting with the environment, so that the intelligent agents can maximize the reward or achieve specific goals after performing corresponding actions.

[0060] Currently, the generative flow models provided by related technologies, after determining that an agent is in a target state, can process information associated with that target state to predict the probability of one or more actions taken by the agent. These actions are used to guide the agent from the target state to one or more subsequent states. In this way, the agent can execute the action with the highest probability predicted by the neural network model, thus entering a subsequent state of the target state. By continuously repeating this process, the agent can progress from the initial state through intermediate states to the final state. For example, consider a vehicle in an autonomous driving scenario. When the vehicle approaches an intersection, it detects a red light (e.g., the vehicle's camera captures a red light). The point where the vehicle reaches the intersection with the red light can be considered the vehicle's initial state. The vehicle can input information indicating its initial state (e.g., an image of a red light at an intersection captured by a camera) into a neural network model. The model can analyze this information to predict the probability of the action the vehicle is about to take (e.g., the probability of the vehicle stopping is 99%, and the probability of the vehicle continuing to move is 1%), so that the vehicle can take the action with the highest probability of occurrence and stop at the intersection where the red light appears. This can be considered the intermediate state in which the vehicle is located.

[0061] The aforementioned generative flow models typically employ online training. This means that during model training, for any given state of the agent, the model can apply the predicted action to that state within an environment simulator, thereby randomly generating the agent's next state. While this training method allows the model to learn as many states as possible, some states may not accurately reflect the agent's actual environment, resulting in relatively mediocre performance of the trained generative flow model.

[0062] Furthermore, during the training of the model using the online training mode, for any state of the agent, the model not only needs to determine the next state of that state, but also the previous state of that state. However, the model often has difficulty finding the correct previous state of that state, which further leads to poor performance of the trained generative flow model.

[0063] To address the aforementioned problems, this application provides an action prediction method that can be implemented in conjunction with artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. Using artificial intelligence for data processing is a common application of AI.

[0064] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 This is a structural diagram illustrating the main framework of artificial intelligence. The following explanation of the AI ​​framework is based on two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0065] (1) Infrastructure

[0066] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0067] (2) Data

[0068] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0069] (3) Data processing

[0070] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0071] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0072] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0073] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0074] (4) General ability

[0075] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0076] (5) Smart Products and Industry Applications

[0077] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0078] The following sections will introduce several application scenarios for this application.

[0079] Figure 2a This is a schematic diagram of a motion prediction system provided in an embodiment of this application. The motion prediction system includes an intelligent agent and a data processing device. The intelligent agent includes intelligent terminals such as robots, vehicle-mounted devices, or drones. The intelligent agent is the initiator of motion prediction; as the initiator of the motion prediction request, the intelligent agent can initiate the request independently.

[0080] The aforementioned data processing equipment can be cloud servers, network servers, application servers, management servers, or other devices or servers with data processing capabilities. The data processing equipment receives action prediction requests from smart terminals through an interactive interface, and then performs information processing such as machine learning, deep learning, search, reasoning, and decision-making through a storage device and a data processing processor. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0081] exist Figure 2a In the action prediction system shown, during the interaction with the environment, the agent can acquire its own state information and then send a request to the data processing device. The data processing device then performs action prediction based on the state information obtained by the agent, thereby obtaining the probability of the agent's actions occurring. For example, an agent can acquire information indicating its current state and send a processing request to the data processing device for that information. The data processing device can then call a generative flow model to process this information, thereby obtaining the probability of the agent's actions occurring. This probability is returned to the agent, and these actions allow the agent to transition from the current state to the next state. This completes the action prediction for the agent. The agent can then select the action with the highest probability of occurrence and execute it to enter the next state.

[0082] exist Figure 2a In this context, the data processing device can execute the motion prediction method of the embodiments of this application.

[0083] Figure 2b This is another structural schematic diagram of the motion prediction system provided in the embodiments of this application. Figure 2b In this system, the intelligent agent can predict actions itself. The agent can directly acquire its own state information and process it directly through its own hardware. The specific process is similar to... Figure 2a Similar to the description above, it will not be repeated here.

[0084] exist Figure 2b In the action prediction system shown, for example, an agent can obtain information indicating that it and other agents are in a certain state, and process the information to obtain the probability of the agent's actions. These actions can cause the agent to move from the current state to the next state. Thus, the action prediction for the agent is completed. The agent can then select the action with the highest probability of occurrence and execute the action to move to the next state.

[0085] exist Figure 2b In this process, the intelligent agent itself can execute the action prediction method of the embodiments of this application.

[0086] Figure 2c This is a schematic diagram of a motion prediction device provided in an embodiment of this application.

[0087] The above Figure 2a and Figure 2b The intelligent agent in the context can specifically be Figure 2c Local device 301 or local device 302 in the system. Figure 2a The data processing equipment in the middle can specifically be Figure 2cThe execution device 210 in the process includes a data storage system 250 that can store the data to be processed by the execution device 210. The data storage system 250 can be integrated into the execution device 210 or set up in the cloud or on other network servers.

[0088] Figure 2a and Figure 2b The processor in the system can perform data training / machine learning / deep learning through neural network models or other models (e.g., generative stream models), and use the data to finally train or learn the model to perform action prediction applications based on the agent's state information, thereby predicting the agent's actions.

[0089] Figure 3 A schematic diagram of the system 100 architecture provided in this application embodiment, in Figure 3 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. The client device 140 (i.e., the aforementioned intelligent agent) inputs data into the I / O interface 112. The input data may include various scheduled tasks, callable resources, and other parameters.

[0090] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0091] Finally, I / O interface 112 returns the processing result to client device 140.

[0092] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These target models / rules can then be used to achieve the aforementioned objectives or complete the aforementioned tasks, thereby providing the user with the required results. The training data can be stored in the database 130 and originates from training samples collected by the data acquisition device 160.

[0093] exist Figure 3In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0094] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this context, the data storage system 150 is an external memory relative to the execution device 110. However, in other cases, the data storage system 150 can also be placed within the execution device 110. For example... Figure 3 As shown, a neural network can be trained using training device 120.

[0095] This application also provides a chip including a neural network processor (NPU). This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rules.

[0096] The Neural Processing Unit (NPU) is a coprocessor mounted on the main central processing unit (CPU) (host CPU), where tasks are assigned by the CPU. The core of the NPU is the computation circuitry, which is controlled by a controller to retrieve data from memory (weight memory or input memory) and perform calculations.

[0097] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0098] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory and caches it in each PE (Process Equipment) of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator.

[0099] Vector computation units can further process the output of computational circuits, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, vector computation units can be used for computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0100] In some implementations, the vector computation unit can store the processed output vector into a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, for example, for use in subsequent layers of a neural network.

[0101] The unified memory is used to store input data and output data.

[0102] The weight data is directly transferred from the external memory to the input memory and / or unified memory, stored in the weight memory, and stored in the unified memory to the external memory through the direct memory access controller (DMAC).

[0103] The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC, and instruction fetch memory via a bus.

[0104] The instruction fetch buffer, connected to the controller, is used to store the instructions used by the controller.

[0105] The controller is used to invoke instructions cached in the memory to control the operation of the computing accelerator.

[0106] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, while external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDRSDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0107] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0108] (1) Neural Network

[0109] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0110]

[0111] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0112] The work of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 by +b, and operation 5 by a(). The term "space" is used here because the objects being classified are not individual things, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight vector, and each value in this vector represents the weight value of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above; that is, the weights W of each layer control how the space is transformed. The purpose of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control the transformation space, and more specifically, learning the weight matrix.

[0113] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0114] (2) Backpropagation algorithm

[0115] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0116] (3) Generative flow networks (GFlowNets)

[0117] Generative flow models typically refer to models constructed in the form of directed acyclic graphs, where each state node has at least one parent state node, unlike tree structures where each state node has only a unique parent. Generative flow models have a unique initial state node and multiple terminating state nodes. They predict actions starting from the initial state node, executing these actions to complete transitions between different states until the terminating state node is reached.

[0118] The initial state node can include output flow, intermediate state nodes can include input flow, output flow, and a preset reward value, and the terminal node can include input flow and a preset reward value. Specifically, the generative flow model can be imagined as a water pipe; the output flow of the initial state node is the total inflow of the entire generative flow model, and the sum of the input flows of all terminal state nodes is the total outflow of the entire generative flow model. For each intermediate state node, the input flow equals the output flow. The input and output flows of each intermediate state node are predicted using a neural network, ultimately allowing the prediction of the input flow of each terminal state node.

[0119] For example, such as Figure 4 As shown ( Figure 4 This is a schematic diagram of a generation flow model provided in an embodiment of this application. In the generation flow model, si represents a state node (i = 0, ..., 11), and xj represents a composite structure (j = 3, 4, 6, 10, 11). s0 is the initial state node, s10 and s11 are the final state nodes, and x3, x4, x6, x10, and x11 are composite structures, each with a reward value. The output flow of the initial state node S0 is equal to the sum of the input flows of the intermediate state node S1 and the intermediate state node S2, ..., the input flow of the final state node S10 is equal to the sum of the output flows of the intermediate state nodes S7 and S8, and the input flow of the final state node S11 is equal to the sum of the output flows of the intermediate state nodes S9.

[0120] The method provided in this application is described below from the perspectives of neural network training and neural network application.

[0121] The model training method provided in this application involves data sequence processing and can be applied to data training, machine learning, deep learning, and other methods. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (e.g., the first information of the first agent in the model training method provided in this application), ultimately obtaining a trained neural network (e.g., the generative flow model in the model training method provided in this application). Furthermore, the action prediction method provided in this application can utilize the trained neural network to input input data (e.g., the information of the first agent in the action prediction method provided in this application) into the trained neural network to obtain output data (e.g., the probability of the first agent's action occurring in the action prediction method provided in this application). It should be noted that the model training method and action prediction method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of an overall process: such as the model training stage and the model application stage.

[0122] The following text combines Figure 5a The model training method provided in the embodiments of this application will be introduced. Figure 5a A flowchart illustrating the modeling method provided in this application embodiment, as shown below. Figure 5a As shown, the method includes:

[0123] 501. Obtain the first information of the agent from the preset offline dataset. The first information is used to indicate that the agent is in the target state.

[0124] In this embodiment, when training the model to be trained (the neural network model to be trained), a pre-set offline dataset can be obtained first. The offline dataset contains M data groups (M is a positive integer greater than or equal to 1). The first data group contains the first first candidate information, the first second candidate information, the first third candidate information, the reward value corresponding to the first candidate state, and the first real action policy. Similarly, the Mth data group contains the Mth first candidate information, the Mth second candidate information, the Mth third candidate information, the reward value corresponding to the Mth candidate state, and the Mth real action policy. For ease of explanation, the following description uses the i-th data group as an example (i = 1, ..., M):

[0125] In the i-th data group, the i-th first candidate information indicates that the agent is in the i-th candidate state, the i-th second candidate information indicates that the agent is in the previous state of the i-th candidate state, and the i-th third candidate information indicates that the agent is in the next state of the i-th candidate state. It should be noted that the i-th candidate state can have one or more previous states, and similarly, the i-th candidate state can have one or more next states. Furthermore, the previous and next states of the i-th candidate state are usually the remaining candidate states from the M candidate states excluding the i-th candidate state. In other words, the transition relationships between the M candidate states have been pre-set (for example, the transition relationships between these M candidate states can be found in [reference needed]). Figure 4 (The transition relationships between the 11 states from s1 to s11).

[0126] In the i-th data group, the i-th real action policy refers to the actual probability of the action flowing out of the i-th candidate state. The action flowing out of the i-th candidate state is used to make the agent enter the next state of the i-th candidate state. It should be noted that the number of actions flowing out of the i-th candidate state is the same as the number of the next state of the i-th candidate state, and the two are in one-to-one correspondence.

[0127] In the i-th data group, the reward value corresponding to the i-th candidate state is a preset value. When the i-th candidate state is an intermediate state, its corresponding reward value is zero. When the i-th candidate state is a termination state, its corresponding reward value is not zero (the size of this reward value can be set according to actual needs, and there is no restriction here).

[0128] After obtaining the offline dataset, since all M data groups in the offline dataset can be used as training data for the model to be trained, the following description uses one of these data groups as an example. The first candidate information in this data group is called the first information, the candidate state indicated by the first candidate information in this data group is called the target state, the second candidate information in this data group is called the second information, and the third candidate information in this data group is called the third information. Thus, the first information indicates that the agent is in the target state, the second information indicates that the agent is in the previous state of the target state, and the third information indicates that the agent is in the next state of the target state. Furthermore, the actual probability of the action flowing from the target state (i.e., the first action mentioned below) occurring is known.

[0129] For example, for an offline dataset D, D includes (s″1, s1, s′R1, π1), ..., (s″1, s′R1, π1). M ,s M ,s′ M ,R M ,πM These are M data sets. Among them, s1 contains information indicating the first (candidate) state, and so on, s... M s″1 is used to indicate the Mth state. s″1 is used to indicate the state preceding the 1st state, and so on, s″ M s′1 is the information used to indicate the state preceding the M-th state. s′1 is the information used to indicate the state following the 1st state, and so on, s′... M This is information used to indicate the next state after the Mth state. R1 is the reward value corresponding to the 1st state, and so on, R... M Let π be the reward value corresponding to the Mth state. Let π1 be the policy of the first true action, π be the true probability of the action flowing from the first state, and so on, π... M Let be the policy for the Mth real action, and let be the actual probability of the action flowing out from the Mth state.

[0130] Then, we can have (s″1,s1,s′1,R1,π1), ..., (s″1) M ,s M ,s′ M ,R M ,π M In the dataset, select one of the data sets as (s″, s, s′, R, π), where s is the information used to indicate the target state, s″ is the information used to indicate the previous state of the target state, s′ is the information used to indicate the next state of the target state, R is the reward value corresponding to the target state, and π is the actual probability of the action flowing out from the target state.

[0131] It should be understood that the information mentioned in this embodiment can be presented in a variety of ways. For example, the first information used to indicate that the agent is in the target state can be an image showing the agent in the target state, a video showing the agent in the target state, an audio recording of the agent in the target state, or text describing the agent in the target state, etc.

[0132] It should also be understood that, in this embodiment, the probability of an action occurring from a certain state is the output flow of that state, and correspondingly, the probability of an action occurring flowing into a certain state is the input flow of that state, which will not be elaborated further.

[0133] 502. The first information is processed by the model to be trained to obtain the probability of the agent's first action. The first action is used to make the agent move from the target state to the next state of the target state.

[0134] After obtaining the first information, it can be input into the model to be trained. The model processes the first information to obtain the (predicted) probability of the agent's first action (i.e., the action flowing out of the target state). The first action is used to cause the agent to move from the target state to the next state. At this point, the model to be trained has completed the action prediction for the target state.

[0135] Continuing with the example above, after inputting s into the model to be trained, the model can perform a series of processes on s to obtain the predicted probability of the action flowing out from the target state.

[0136]

[0137] In the above formula, Let N be the predicted probability of the j-th action flowing out of the target state, and let N be the number of actions flowing out of the target state.

[0138] 503. Based on the probability of the first action and the actual probability of the first action, the model to be trained is trained to obtain the generative flow model. The actual probability of the first action comes from the offline dataset.

[0139] It is worth noting that when the model to be trained obtains the probability of the agent's first action, it will try its best to follow the following constraints (which can also be understood as the model to be trained using these constraints as the model training objective): the difference between the probability of the agent's first action and the actual probability of the agent's first action is within a preset range (the size of this range can be set according to the actual situation, and is not limited here).

[0140] As in the example above, the model to be trained will try its best to make The following conditions must be met:

[0141]

[0142] In the above formula, F(s,a) j ) represents the actual probability of the j-th action flowing out from the target state.

[0143] After obtaining the probability of the agent's first action, the model to be trained can be trained based on the probability of the agent's first action until the model training conditions are met, thereby obtaining the generative flow model.

[0144] Specifically, the model to be trained can be trained in the following way to obtain the generative stream model:

[0145] (1) After obtaining the probability of the agent's first action, the probability of the agent's second action (i.e., the action that leads to the target state) can also be obtained. The agent's second action is used to cause the agent to enter the target state from the previous state. It should be noted that since the model to be trained has already completed the action prediction for the previous state of the target state, the probability of the agent's second action can be obtained directly. Therefore, some data in the offline dataset can be used to correct the probability of the agent's second action, thereby obtaining the corrected probability of the agent's second action.

[0146] More specifically, the corrected probability of the agent's second action can be obtained in the following ways:

[0147] M first candidate information items and M second candidate information items are extracted from the offline database. The first information items, second information items, and the M first candidate information items are then used to calculate a new transition value for the target state. Finally, the probability of the agent's second action is adjusted using this new transition value to obtain the adjusted probability of the second action.

[0148] As in the example above, the new transition value for the target state can be obtained using the following formula.

[0149]

[0150] In the above formula, s i For information used to indicate the i-th state, s″ i This is information used to indicate the previous state of the i-th state. (The result is...) Afterwards, it can be used Predicting the probability of an action occurring in the target state. After making corrections, the predicted probability of the action flowing into the target state is obtained. Where, a′ k Let F(s,a′) be the k-th action flowing into the target state, P be the number of actions flowing into the target state, and F(s,a′) be the k-th action flowing into the target state. k ) represents the predicted probability of the k-th action occurring in the target state. The corrected predicted probability of the k-th action flowing into the target state.

[0151] (2) After obtaining the probability of the agent’s first action, the reward value of the target state object can be obtained from the offline dataset, and some data in the offline dataset can be used to correct the reward value corresponding to the target state to obtain the corrected reward value corresponding to the target state.

[0152] More specifically, the corrected reward value corresponding to the target state can be obtained in the following ways:

[0153] Extract M first candidate information and M candidate state reward values ​​from the offline database, and calculate the first information, M first candidate information and M candidate state reward values ​​to obtain the corrected reward value corresponding to the target state.

[0154] As in the example above, the corrected reward value corresponding to the target state can be obtained using the following formula.

[0155]

[0156] In the above formula, R i Let be the reward value corresponding to the i-th state.

[0157] (3) After obtaining the corrected probability of the agent's second action and the corrected reward value corresponding to the target state, the probability of the agent's first action, the corrected probability of the agent's second action, and the corrected reward value corresponding to the target state can be calculated to obtain the loss for the target state.

[0158] As in the example above, the loss L(s) for the target state can be obtained using the following formula:

[0159]

[0160] (4) After obtaining the loss for the target state, a similar operation can be performed for the remaining candidate states among the M candidate states other than the target state. Thus, the final loss for the M candidate states can be obtained. These losses are then superimposed to obtain the target loss, which is used to update the parameters of the model to be trained, resulting in the updated model. Then, the updated model can be trained using the next batch of training data until the model training conditions are met (e.g., the target loss converges, etc.), thereby obtaining the generative flow model.

[0161] In this embodiment, when training the model to be trained is required, first information about the agent can be obtained from a pre-set offline dataset. This first information indicates that the agent is in a target state. Then, the first information can be input into the model to be trained, allowing the model to process the information and obtain the probability of the agent's first action. This first action causes the agent to transition from the target state to the next state. Finally, based on the probability of the agent's first action, the model to be trained can be trained to obtain a generative stream model. In the aforementioned process, when the model to be trained acquires the probability of the agent's first action, it ensures that the difference between the probability of the agent's first action and the actual probability of the agent's first action are within a preset range. Since the probability of the agent's first action can be called the predicted action strategy of the model to be trained for the target state, and the actual probability of the agent's first action can be called the actual action strategy for the target state in the offline database, this allows the predicted action strategy for the target state to be as close as possible to the actual action strategy for the target state. The actual action strategy for the target state determines the actual probability of the agent entering the next state from the target state. Therefore, the model to be trained can not only learn as many next states as possible for the target state, but also the learned states are sufficiently consistent with the actual environment in which the agent is located (because the data in the offline dataset are all pre-set based on the actual environment in which the agent is located). Thus, the generative stream model trained in the offline training mode can have better performance.

[0162] Furthermore, during the training of the model to be trained using the offline training mode provided in this application embodiment, the transition value for the target state is corrected to obtain a new transition value for the target state. Then, based on the new transition value, the probability of the agent's second action is corrected. Since the agent's second action can be used to allow the agent to enter the target state from the previous state, the corrected probability of the agent's second action is equivalent to helping the model to be trained find the correct previous state of the target state. This can further improve the performance of the trained generative flow model.

[0163] Furthermore, during the training of the model to be trained using the offline training mode provided in this application embodiment, the reward value corresponding to the target state is corrected to obtain the corrected reward value corresponding to the target state. The loss constructed based on the corrected reward value for the target state is more accurate. Therefore, training the generative flow model based on such loss can further improve the performance of the generative flow model.

[0164] To further understand the model training method provided in the embodiments of this application, the following description is combined with... Figure 5bThis method will be further described. Figure 5b A schematic diagram illustrating an application example of the model training method provided in this application embodiment, such as... Figure 5b As shown, an offline dataset can be constructed by a third party (e.g., the developer of the agent). This involves collecting multiple goals / problems (information used to indicate multiple states), multiple corresponding answers (information used to indicate multiple actions), and labels for the answers (labelers, which can be called the true probability of occurrence of multiple answers, i.e., the true probability of occurrence of multiple actions). The reward values ​​for multiple goals are then constructed based on the labels of the multiple answers. This information can constitute the offline dataset.

[0165] For a generative flow model (GFlownets) to be trained, multiple questions from an offline dataset can be input into the model to obtain the predicted probabilities of multiple answers. Then, a target loss can be calculated based on the reward value, the actual probability of occurrence, and the predicted probability of occurrence, and the parameters of the generative flow model can be updated using this target loss. This training process is repeated multiple times until the loss converges, resulting in a trained generative flow model. This trained generative flow model possesses intelligent question-answering capabilities and can be used directly as an automatic question-answering model or as part of a larger automatic question-answering model (e.g., the ChatGPT model), thus improving and enhancing the functionality of these automatic question-answering models (e.g., increasing the diversity of answers in the ChatGPT model).

[0166] The above is a detailed description of the model training method provided in the embodiments of this application. The following will introduce the action prediction method provided in the embodiments of this application. It should be noted that the action prediction method provided in the embodiments of this application can be applied to various scenarios. The concepts of intelligent agent, action, and state involved in this method change with the application scenario. For example, in an autonomous driving scenario, a vehicle can predict its next driving action based on its current driving state in the traffic environment and execute that action to change its driving state in the traffic environment. Similarly, in a supply chain scenario, a robot can predict its next transportation direction based on its current transportation state in the workshop and move in those directions to change its transportation state in the workshop. Furthermore, in an advertising recommendation scenario, an advertiser can predict the switching between advertising content based on the advertising content currently recommended to the user and execute that switching to change the advertising content recommended to the user. Finally, in a gaming scenario, a game player can predict their next action based on their competitive state in the virtual game environment and execute that action to change their competitive state in the virtual game environment, and so on. For example, in intelligent question-answering scenarios, a machine can predict its next action based on its current conversation with the user and execute that action to generate the next conversation with the user, and so on. Figure 6 A flowchart illustrating the motion prediction method provided in this application embodiment is shown below. Figure 6 As shown, this method is executed by an agent, which has a built-in generative flow model trained in Figure 5. The method includes:

[0167] 601. Obtain information about the agent, which is used to indicate that the agent is in the target state.

[0168] In this embodiment, an intelligent agent exists in the environment. This agent continuously interacts with the environment, performing various actions to change its state within the environment. It should be noted that the agent can predict its own actions and execute them, thereby continuously altering its state within the environment.

[0169] Suppose the agent is currently in the target state. In order to predict the action that will lead it to the next state, the agent can first collect information indicating that it is in the target state. This information can be an image taken by the agent through a camera, a video taken by the agent through a camera, an audio collected by the agent through a microphone, or text generated by the agent, etc.

[0170] 602. The information is processed by the model to be trained to obtain the probability of the agent's action. The action is used to make the agent move from the target state to the next state of the target state.

[0171] After obtaining its own information, the agent can input its own information into the generative flow model set in itself, so that the generative flow model can process its own information (e.g., a series of feature extraction processes, etc.) and thus predict the probability of one or more actions of the agent.

[0172] Specifically, for any action predicted by the agent, this action is used to cause the agent to transition from the target state to a next state of the target state in the environment. Thus, the agent has completed its action prediction for the target state.

[0173] In this way, the agent can select the action with the highest probability of occurrence from one or more predicted actions and execute that action to enter a next state of the target state. Alternatively, the agent can select the action with the highest probability of occurrence from one or more predicted actions and one or more preset actions (the occurrence probabilities of these preset actions are derived from a dataset built from expert experience; these preset actions can be actions previously predicted by the agent or actions pre-set in the agent, without restriction here), and execute that action to enter a next state of the target state. At this point, the agent has completed the action execution for the target state.

[0174] For example, such as Figure 7 As shown ( Figure 7 (This is another structural diagram of the generation flow model provided in the embodiment of this application). After vehicle 1 inputs the photo it takes into the generation flow model of vehicle 1, the generation flow model can determine that vehicle 1 is in the initial state node S0 (i.e., vehicle 1 is in the initial state S0) based on the photo. Then, the generation flow model can process the photo to obtain the flow of vehicle 1's action a1 (also known as the probability of action a1 occurring) and the flow of vehicle 1's action a2.

[0175] In this context, vehicle 1's action a1 causes it to directly enter intermediate state node S1 from the initial state node S0 (or, in other words, vehicle 1's action a1 flows out of the initial state node S0 and into the intermediate state node S1). Vehicle 1's action a2 causes it to directly enter intermediate state node S2 from the initial state node S0. Furthermore, the sum of the flow rates of vehicle 1's actions a1 and a2 is the output flow rate of the initial state node S0. The flow rate of vehicle 1's actions a1 is the input flow rate of the intermediate state node S1, and the flow rate of vehicle 1's actions a2 is the input flow rate of the intermediate state node S2. If the flow rate of vehicle 1's actions a1 is greater than the flow rate of vehicle 1's actions a2, then vehicle 1 can choose action a1 and execute it. Therefore, vehicle 1 is currently in intermediate state S1.

[0176] After vehicle 1 reaches the intermediate state S1, it can take a new photo, which is used to indicate that vehicle 1 is in the intermediate state S1. Then, after vehicle 1 inputs the new photo into its generation flow model, the generation flow model can process the photo to obtain the flow of vehicle 1's action a3 (which can also be called the probability of vehicle 1's action a3 occurring).

[0177] In this context, vehicle 1's action a3 is used to directly transition vehicle 1 from intermediate state node S1 to intermediate state node S3. Furthermore, the flow of vehicle 1's action a3 is the output flow of intermediate state node S1, and the flow of vehicle 1's action a3 is the input flow of intermediate state node S3. Since the flow of vehicle 1's action a3 is the largest, vehicle 1 can execute action a3. Therefore, vehicle 1 is currently in intermediate state S3.

[0178] This process continues until vehicle 1 reaches a certain termination state node. For example, vehicle 1 continues state transition (i.e., performs action prediction and action execution) from intermediate state node S3, passes through intermediate state nodes S7 and S10, and finally reaches the termination state node S13, where it no longer performs state transition.

[0179] In this embodiment, when the agent is currently in a target state, in order to predict the action that will lead it to the next state from the target state, the agent can first collect information indicating that it is in the target state. After obtaining its own information, the agent can input its own information into a generative flow model, which processes the information to obtain the probability of one or more actions. Then, the agent can select the action with the highest probability of occurrence from among the one or more actions and execute that action to enter a certain next state of the target state. In the aforementioned process, because the agent has a built-in generative flow model, it can accurately complete the transition between different states based on the completion of action prediction and action execution.

[0180] The above is a detailed description of the model training method and action prediction method provided in the embodiments of this application. The model training device and action prediction device provided in the embodiments of this application will be described below. Figure 8 A schematic diagram of the model training apparatus provided in the embodiments of this application is shown below. Figure 8 As shown, the device includes:

[0181] The acquisition module 801 is used to acquire the first information of the agent from a preset offline dataset. The first information is used to indicate that the agent is in the target state.

[0182] The processing module 802 is used to process the first information through the model to be trained to obtain the probability of the occurrence of the first action of the agent. The first action is used to cause the agent to enter the next state of the target state from the target state.

[0183] Training module 803 is used to train the model to be trained based on the probability of the occurrence of the first action, so as to obtain the generative flow model. The actual occurrence probability comes from the offline dataset.

[0184] In this embodiment, when training the model to be trained is required, first information about the agent can be obtained from a pre-set offline dataset. This first information indicates that the agent is in a target state. Then, the first information can be input into the model to be trained, allowing the model to process the information and obtain the probability of the agent's first action. This first action causes the agent to transition from the target state to the next state. Finally, based on the probability of the agent's first action and the actual probability of the first action, the model to be trained is trained to obtain a generative flow model. The actual probability originates from the offline dataset. In the aforementioned process, the probability of the agent's first action can be called the predicted action policy of the model to be trained for the target state, and the actual probability of the agent's first action can be called the actual action policy of the target state in the offline database. This allows the predicted action policy for the target state to be as close as possible to the actual action policy for the target state. The actual action policy for the target state determines the actual probability of the agent entering the next state from the target state. Therefore, the model to be trained can not only learn as many next states of the target state as possible, but also the learned states are sufficiently consistent with the actual environment in which the agent is located (because the data in the offline dataset are all pre-set based on the actual environment in which the agent is located). Thus, the generative stream model trained in the offline training mode can have better performance.

[0185] In one possible implementation, training module 803 is used to train the model to be trained based on the probability of occurrence of the first action, so that the difference between the probability of occurrence of the first action and the actual probability of occurrence of the first action is within a preset range, thereby obtaining a generative flow model.

[0186] In one possible implementation, the training module 803 is used to: correct the occurrence probability of the agent's second action based on an offline dataset to obtain a corrected occurrence probability of the second action, the second action being used to cause the agent to enter the target state from the previous state of the target state; correct the reward value corresponding to the target state based on the offline dataset to obtain a corrected reward value corresponding to the target state; and train the model to be trained based on the occurrence probability of the first action, the corrected occurrence probability of the second action, and the reward value corresponding to the target state to obtain a generative flow model.

[0187] In one possible implementation, the offline dataset includes M first candidate information and M second candidate information. The i-th first candidate information is used to indicate that the agent is in the i-th candidate state, and the i-th second candidate information is used to indicate that the agent is in the previous state of the i-th candidate state. The M first candidate information includes first information, and the M second candidate information includes second information. The second information is used to indicate that the agent is in the previous state of the target state, and the M candidate states include the target state, M≥1. The training module 803 is used to correct the occurrence probability of the agent's second action based on the first information, the second information, the M first candidate information, and the M second candidate information to obtain the corrected occurrence probability of the second action.

[0188] In one possible implementation, the offline dataset also includes reward values ​​corresponding to M candidate states. The training module 803 is used to correct the reward value corresponding to the target state based on the first information, the M first candidate information, and the reward values ​​corresponding to the M candidate states, so as to obtain the corrected reward value corresponding to the target state.

[0189] In one possible implementation, the first information is information collected when the agent is in the target state, and the information includes at least one of the following: image, video, audio, or text.

[0190] Figure 9a A schematic diagram of the motion prediction device provided in the embodiments of this application is shown below. Figure 9a As shown, the device includes a generative stream model trained by the aforementioned model training device, and the device includes:

[0191] The acquisition module 901 is used to acquire information about the intelligent agent, which is used to indicate that the intelligent agent is in a target state.

[0192] The processing module 902 is used to process information through the model to be trained to obtain the probability of the agent's action. The action is used to cause the agent to move from the target state to the next state of the target state.

[0193] In this embodiment, when the agent is currently in a target state, in order to predict the action that will lead it to the next state from the target state, the agent can first collect information indicating that it is in the target state. After obtaining its own information, the agent can input its own information into a generative flow model, which processes the information to obtain the probability of one or more actions. Then, the agent can select the action with the highest probability of occurrence from among the one or more actions and execute that action to enter a certain next state of the target state. In the aforementioned process, because the agent has a built-in generative flow model, it can accurately complete the transition between different states based on the completion of action prediction and action execution.

[0194] Figure 9b Another schematic diagram of the motion prediction device provided in the embodiments of this application is shown below. Figure 9b As shown, the device includes a generative stream model trained by the aforementioned model training device, and the device includes:

[0195] The acquisition module 901 is used to acquire information about the intelligent agent, which is used to indicate that the intelligent agent is in a target state.

[0196] The processing module 902 is used to process information through the model to be trained to obtain the probability of the agent's action. The action is used to cause the agent to move from the target state to the next state of the target state.

[0197] The determination module 903 is used to determine the action to be executed based on the probability of the action occurring and the probability of the occurrence of a preset action.

[0198] In this embodiment, when the agent is currently in a target state, in order to predict the action that will lead it to the next state from the target state, the agent can first collect information indicating that it is in the target state. After obtaining its own information, the agent can input its own information into a generative flow model, which processes the information to obtain the probability of one or more actions. Then, the agent can select the action with the highest probability of occurrence from among the one or more actions and execute that action to enter a certain next state of the target state. In the aforementioned process, because the agent has a built-in generative flow model, it can accurately complete the transition between different states based on the completion of action prediction and action execution.

[0199] In one possible implementation, the determining module 903 is used to: determine the action with the highest probability of occurrence as the action to be executed from among the actions predicted by the generating flow model and the preset actions.

[0200] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0201] This application also relates to an execution device. Figure 10 This is a schematic diagram of the execution device provided in an embodiment of this application. Figure 10 As shown, the execution device 1000 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited here. The execution device 1000 may be equipped with the action prediction device described in the embodiment corresponding to Figure 9, for implementing... Figure 6 This corresponds to the action prediction function in the embodiment. Specifically, the execution device 1000 includes: a receiver 1001, a transmitter 1002, a processor 1003, and a memory 1004 (wherein the execution device 1000 may have one or more processors 1003). Figure 10 (Taking a processor as an example), processor 1003 may include application processor 10031 and communication processor 10032. In some embodiments of this application, receiver 1001, transmitter 1002, processor 1003 and memory 1004 may be connected via bus or other means.

[0202] Memory 1004 may include read-only memory and random access memory, and provides instructions and data to processor 1003. A portion of memory 1004 may also include non-volatile random access memory (NVRAM). Memory 1004 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0203] Processor 1003 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0204] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1003. The processor 1003 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1003 or by instructions in software form. The processor 1003 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1003 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1004. Processor 1003 reads the information in memory 1004 and, in conjunction with its hardware, completes the steps of the above method.

[0205] Receiver 1001 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1002 can be used to output digital or character information through the first interface; transmitter 1002 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1002 may also include a display device such as a display screen.

[0206] In one embodiment of this application, the processor 1003 is used to... Figure 6 The generation flow model in the corresponding embodiment predicts the behavior of the agent.

[0207] This application also relates to a training device. Figure 11 This is a schematic diagram of the structure of a training device provided in an embodiment of this application. Figure 11As shown, the training device 1100 is implemented by one or more servers. The training device 1100 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1114 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 can be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1114 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the training device 1100.

[0208] The training device 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158; or, one or more operating systems 1141, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0209] Specifically, the training device can execute the model training method in the embodiment corresponding to Figure 5.

[0210] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0211] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.

[0212] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0213] For details, please refer to Figure 12 , Figure 12 This is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1200. The NPU 1200 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1203, which is controlled by the controller 1204 to extract matrix data from the memory and perform multiplication operations.

[0214] In some implementations, the arithmetic circuit 1203 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1203 is a two-dimensional pulsating array. The arithmetic circuit 1203 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1203 is a general-purpose matrix processor.

[0215] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1202 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1201 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1208.

[0216] Unified memory 1206 is used to store input and output data. Weight data is directly transferred to weight memory 1202 via Direct Memory Access Controller (DMAC) 1205. Input data is also transferred to unified memory 1206 via DMAC.

[0217] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1209.

[0218] The Bus Interface Unit (BIU) 1213 is used by the instruction fetch memory 1209 to fetch instructions from external memory, and also by the memory access controller 1205 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0219] The DMAC is mainly used to move input data from external memory DDR to unified memory 1206, or to weight data to weight memory 1202, or to input data to input memory 1201.

[0220] The vector computation unit 1207 includes multiple processing units that further process the output of the computation circuit 1203 when needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.

[0221] In some implementations, the vector computation unit 1207 can store the processed output vector in the unified memory 1206. For example, the vector computation unit 1207 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1203, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1207 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1203, for example, for use in subsequent layers of the neural network.

[0222] The instruction fetch buffer 1209 connected to the controller 1204 is used to store the instructions used by the controller 1204;

[0223] Unified memory 1206, input memory 1201, weight memory 1202, and instruction fetch memory 1209 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0224] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0225] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0227] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0228] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A model training method, characterized in that, The method includes: First information about the agent is obtained from a pre-set offline dataset, and the first information is used to indicate that the agent is in a target state. The first information is processed by the model to be trained to obtain the probability of the occurrence of the first action of the agent. The first action is used to cause the agent to enter the next state of the target state from the target state. Based on the offline dataset, the probability of the occurrence of the agent's second action is corrected, and the reward value corresponding to the target state is corrected. The second action is used to cause the agent to enter the target state from the previous state of the target state. Based on the probability of the first action, the corrected probability of the second action, and the reward value corresponding to the corrected target state, the model to be trained is trained so that the difference between the probability of the first action and the actual probability of the first action is within a preset range, thereby obtaining a generative flow model, wherein the actual probability of the first action is derived from the offline dataset.

2. The method according to claim 1, characterized in that, The offline dataset includes M first candidate information and M second candidate information. The i-th first candidate information is used to indicate that the agent is in the i-th candidate state, and the i-th second candidate information is used to indicate that the agent is in the previous state of the i-th candidate state. The M first candidate information includes the first information, and the M second candidate information includes the second information. The second information is used to indicate that the agent is in the previous state of the target state, and the M candidate states include the target state, where M≥1. The step of correcting the occurrence probability of the agent's second action based on the offline dataset to obtain the corrected occurrence probability of the second action includes: Based on the first information, the second information, the M first candidate information and the M second candidate information, the occurrence probability of the agent's second action is corrected to obtain the corrected occurrence probability of the second action.

3. The method according to claim 2, characterized in that, The offline dataset also includes reward values ​​corresponding to the M candidate states. The step of correcting the reward value corresponding to the target state based on the offline dataset to obtain the corrected reward value for the target state includes: Based on the first information, the M first candidate information and the reward values ​​corresponding to the M candidate states, the reward value corresponding to the target state is corrected to obtain the corrected reward value corresponding to the target state.

4. The method according to any one of claims 1 to 3, characterized in that, The first information is information collected when the intelligent agent is in the target state, and the information includes at least one of the following: image, video, audio or text.

5. A motion prediction method, characterized in that, The method is implemented using the generation flow model of any one of claims 1 to 4, and the method includes: Acquire information about the intelligent agent, the information being used to indicate that the intelligent agent is in a target state; The information is processed by the generative flow model to obtain the probability of the agent's action occurring. The action is used to cause the agent to move from the target state to the next state of the target state.

6. A motion prediction method, characterized in that, The method includes: Acquire information about the intelligent agent, the information being used to indicate that the intelligent agent is in a target state; The information is processed by a generative flow model to obtain the probability of the agent's action occurring. The action is used to cause the agent to move from the target state to the next state of the target state. The generative flow model is trained by the method described in any one of claims 1 to 4. The action to be executed is determined based on the probability of the action occurring and the probability of the preset action occurring.

7. The method according to claim 6, characterized in that, The probability of the action occurring and the probability of the preset action occurring are used to determine the action to be executed, including: Among the stated actions and the preset actions, the action with the highest probability of occurrence is determined as the action to be executed.

8. A model training device, characterized in that, The device includes: The acquisition module is used to acquire first information of the agent from a preset offline dataset, wherein the first information is used to indicate that the agent is in a target state; The processing module is used to process the first information through the model to be trained to obtain the probability of the occurrence of the first action of the agent. The first action is used to cause the agent to enter the next state of the target state from the target state. The training module is used to correct the occurrence probability of the agent's second action and the reward value corresponding to the target state based on the offline dataset. The second action is used to cause the agent to enter the target state from the previous state. The module is also used to train the model to be trained based on the occurrence probability of the first action, the corrected occurrence probability of the second action, and the corrected reward value corresponding to the target state, so that the difference between the occurrence probability of the first action and the actual occurrence probability of the first action is within a preset range, thereby obtaining a generative flow model. The actual occurrence probability is derived from the offline dataset.

9. The apparatus according to claim 8, characterized in that, The offline dataset includes M first candidate information and M second candidate information. The i-th first candidate information is used to indicate that the agent is in the i-th candidate state, and the i-th second candidate information is used to indicate that the agent is in the previous state of the i-th candidate state. The M first candidate information includes the first information, and the M second candidate information includes the second information. The second information is used to indicate that the agent is in the previous state of the target state, and the M candidate states include the target state, where M≥1. The training module is used to correct the occurrence probability of the agent's second action based on the first information, the second information, the M first candidate information and the M second candidate information, so as to obtain the corrected occurrence probability of the second action.

10. The apparatus according to claim 9, characterized in that, The offline dataset also includes reward values ​​corresponding to the M candidate states. The training module is used to correct the reward value corresponding to the target state based on the first information, the M first candidate information, and the reward values ​​corresponding to the M candidate states, so as to obtain the corrected reward value corresponding to the target state.

11. The apparatus according to any one of claims 8 to 10, characterized in that, The first information is information collected when the intelligent agent is in the target state, and the information includes at least one of the following: image, video, audio or text.

12. A motion prediction device, characterized in that, The apparatus comprises the generation flow model of any one of claims 8 to 11, the apparatus comprising: An acquisition module is used to acquire information about the intelligent agent, the information being used to indicate that the intelligent agent is in a target state; The processing module is used to process the information through the generation flow model to obtain the probability of the agent's action occurring. The action is used to cause the agent to move from the target state to the next state of the target state.

13. A motion prediction device, characterized in that, The device includes: An acquisition module is used to acquire information about the intelligent agent, the information being used to indicate that the intelligent agent is in a target state; A processing module is used to process the information through a generative flow model to obtain the probability of the agent's action occurring. The action is used to cause the agent to move from the target state to the next state of the target state. The generative flow model is trained by the method described in any one of claims 1 to 4. The determination module is used to determine the action to be executed based on the probability of the action occurring and the probability of a preset action occurring.

14. A model training device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the model training device performs the method as described in any one of claims 1 to 4.

15. A motion prediction device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the motion prediction device performs the method as described in any one of claims 5 to 7.

16. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 7.

17. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 7.

Images