Information processing method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to an information processing method and device, equipment and a storage medium. The method comprises the steps of obtaining input information of a model; and providing content generated by the model based on the input information, where the model is trained based on: obtaining a first set of tokens generated by the model based on a first token sequence, the number of tokens of the first set of tokens not exceeding a target number, where the target number is less than the maximum number of output tokens of the model; in response to the condition that the first group of tokens meets a preset condition, constructing a second token sequence based on the first group of tokens and the first token sequence; obtaining a second set of tokens generated by the model based on the second token sequence; and training a model based at least on the generated first set of tokens and the generated second set of tokens. According to the embodiment of the invention, the information processing efficiency of the model in the training process can be improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to information processing methods, apparatuses, devices, and computer-readable storage media. Background Art

[0002] Reinforcement learning (RL) is a method that allows an agent to learn to make optimal decisions in specific scenarios through interaction between the agent and the environment. By trying different actions and observing the rewards returned by the environment, the agent gradually adjusts its strategy, ultimately learning behavioral patterns that maximize long-term accumulated rewards. However, current RL practices are inefficient. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for information processing is provided. The method includes: obtaining input information of a generative model; and providing content generated by the generative model based on the input information, wherein the generative model is trained based on the following process: obtaining a first set of tokens generated by the generative model based on a first token sequence, wherein the number of tokens in the first set of tokens does not exceed a target number, wherein the target number is less than a maximum number of output tokens of the generative model; in response to the first set of tokens satisfying a preset condition, constructing a second token sequence based on the first set of tokens and the first token sequence; obtaining a second set of tokens generated by the generative model based on the second token sequence; and training the generative model based on at least the generated first set of tokens and the generated second set of tokens.

[0004] In a second aspect of the present disclosure, a device for information processing is provided. The device includes: an acquisition module configured to acquire input information of a generative model; and a providing module configured to provide content generated by the generative model based on the input information, wherein the generative model is trained based on the following process: acquiring a first set of tokens generated by the generative model based on a first token sequence, wherein the number of tokens in the first set of tokens does not exceed a target number, wherein the target number is less than a maximum number of output tokens of the generative model; in response to the first set of tokens satisfying a preset condition, constructing a second token sequence based on the first set of tokens and the first token sequence; acquiring a second set of tokens generated by the generative model based on the second token sequence; and training the generative model based on at least the generated first set of tokens and the generated second set of tokens.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram illustrating an example environment in which embodiments according to the present disclosure may be implemented;

[0011] Figures 2A to 2B A flowchart illustrating an example process for information processing according to some embodiments of the present disclosure;

[0012] Figure 3 A first example diagram illustrating an example process of information processing according to some embodiments of the present disclosure;

[0013] Figure 4 A second example diagram illustrating an example process of information processing according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic structural block diagram showing an example apparatus for information processing according to some embodiments of the present disclosure; and

[0015] Figure 6 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0016] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0019] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects shall comply with the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware of and confirms them. Accordingly, when implementing the various embodiments of the present disclosure, the types, scope of use, and usage scenarios of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method may vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0020] If this specification and the solutions in the examples involve the processing of personal information, such processing will be done only with a legitimate basis (such as with the consent of the subject of personal information or as necessary for the performance of a contract) and only within the prescribed or agreed scope. A user's refusal to process personal information other than that required for basic functions will not affect the user's use of basic functions.

[0021] Due to its same-policy nature and the computational overhead in long trajectory scenarios, proximal policy optimization becomes time-consuming as response length increases. Because the same-policy requires collecting new interaction data at each policy iteration, a single interaction in long trajectory scenarios may require generating many state-action sequences. This causes the time cost of data collection to increase linearly with trajectory length, and the computational effort required for data storage and preprocessing also increases with trajectory length. Furthermore, since the proximal policy optimization objective relies on an advantage function, generalized advantage estimation can be used to calculate the advantage function. However, calculating the advantage function requires cumulatively summing the future time-difference residuals of the entire trajectory. For very long trajectories, this causes the total time complexity to increase linearly with trajectory length, thus affecting the model's information processing efficiency.

[0022] An embodiment of the present disclosure proposes an information processing scheme. The scheme includes: obtaining input information of a generative model; and providing content generated by the generative model based on the input information, wherein the generative model is trained based on the following process: obtaining a first set of tokens generated by the generative model based on a first token sequence, wherein the number of tokens in the first set of tokens does not exceed a target number, wherein the target number is less than a maximum number of output tokens of the generative model; in response to the first set of tokens satisfying a preset condition, constructing a second token sequence based on the first set of tokens and the first token sequence; obtaining a second set of tokens generated by the generative model based on the second token sequence; and training the generative model based on at least the generated first set of tokens and the generated second set of tokens.

[0023] In this way, embodiments of the present disclosure can truncate the first set of tokens based on a target number, allowing token generation that has not yet completed the current batch to continue processing in the next batch, significantly reducing generation time. This approach can improve the model's information processing efficiency while ensuring stable optimization of the strategy using incomplete trajectories.

[0024] Various example implementations of this solution are described in detail below in conjunction with the accompanying drawings.

[0025] Sample Environment

[0026] Figure 1 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1As shown, the example environment 100 may include an electronic device 110. In some embodiments, the electronic device 110 may obtain input information of a generative model 120; and provide content generated by the generative model 120 based on the input information, wherein the generative model 120 is trained based on the following process: obtaining a first set of tokens generated by the generative model 120 based on a first token sequence, wherein the number of tokens in the first set of tokens does not exceed a target number, wherein the target number is less than the maximum number of output tokens of the generative model 120; in response to the first set of tokens satisfying a preset condition, constructing a second token sequence based on the first set of tokens and the first token sequence; obtaining a second set of tokens generated by the generative model 120 based on the second token sequence; and training the generative model 120 based on at least the generated first set of tokens and the generated second set of tokens. The generative model 120 can be deployed on the electronic device 110, and can also be deployed on other devices, which will not be described in detail here.

[0027] In some embodiments, the electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a handheld computer, a portable game terminal, a VR / AR device, a personal communication system (Personal Communication System, PCS) device, a personal navigation device, a personal digital assistant (Personal Digital Assistant, PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the target user (such as a "wearable" circuit, etc.).

[0028] The electronic device 110 may also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The electronic device 110 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0029] It should be understood that the structures and functions of the various embedded elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0030] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0031] Example Process

[0032] The Markov Decision Process (MDP) is a core framework for reinforcement learning, used to formally describe sequential decision-making problems. By abstracting the interaction between an agent and its environment into a computable model, it solves the optimal action to take in each state while maximizing the long-term cumulative reward.

[0033] The Markov decision process can be expressed by M = {S, A, p, r, d0, γ}. At the tth time step, the state is represented by s t ={x,y 1:t-1}, x is the input information, y 1:t-1 For the response generated by the model from step 1 to step t-1, x and y 1:t-1 Splicing can get the current state s t Action a is the choice made by the agent in state s, which affects the state transition and reward. p represents the state transition probability, which is the probability of transitioning to state s' after executing action a in state s. p(s ′ |s,a):S×A×S→[0,1]. r:S×A→R represents the reward function, which is used to quantify the immediate benefit or cost obtained by the agent when performing an action in a specific state. For the tth time step, the value of the reward function is r(s t ,a t ).

[0034] d0 is the initial state distribution. In the Markov decision process, the initial state distribution defines the probability distribution of the first state of the environment or system at the beginning of the task. γ∈[0,1] is the discount factor, which is the weight used to balance the immediate reward and future rewards in the Markov decision process. The strategy π is the rule used in the Markov decision process to define the action selected by the agent in a given state. That is, given the current state, the action is selected from the probability distribution of the strategy π, a t ~π(s t ).

[0035] In the Markov decision process, the trajectory is a sequence of states, actions, etc. generated during the interaction between the agent and the environment M, which records the complete interaction process from the initial state to a certain termination time step. The trajectory can be expressed as τ = (s0, a0, ..., s t ,a t ,…,s T-1 ,a T-1), where T is the length of the trajectory. The goal of the Markov decision process is to find an optimal strategy through mathematical optimization so that the cumulative reward obtained by the agent from the long-term perspective when interacting with the environment is maximized after adjusting the discount factor. The formula can be expressed as follows:

[0036]

[0037] In the Markov decision process, the state value function is used to quantify the long-term expected return triggered from a certain state under a given strategy, V π (s t )=E τ～π [G t |s t ],in, is the cumulative return, which is used to measure the long-term cumulative reward obtained by the agent after time step t. The state-action value function is used to quantify the long-term expected return that can be obtained in the future after performing a specific action from a certain state under a given strategy, which can be expressed as Q π (s t ,a t )=E τ～π [G t |s t ,a t The key advantage function is used to quantify the degree of advantage of an action relative to the average level of the strategy in a specific state, which can be expressed as A π (s t ,a t )=Q π (s t ,a t )-V π (s t ).

[0038] Proximal Policy Optimization (PPO) is a policy optimization algorithm in reinforcement learning. Proximal Policy Optimization limits the magnitude of policy updates, thereby avoiding the problem of inconsistent generation results due to excessive policy updates in the generation task. It also ensures the stability of training, thereby achieving stable policy improvement throughout the process. The goal of proximal policy optimization is to maximize the pruned surrogate objective function, as shown in the following formula:

[0039]

[0040] π θ Represents the new strategy currently applied, represents the old strategy applied historically, is an estimator of the advantage function, and ∈ is a clipping hyperparameter used to limit the deviation range of the probability ratio of the new strategy to the old strategy.

[0041] In proximal policy optimization, the critic model is a value function estimator used to predict the expected future reward of the agent in the state it observes. Because the critic model provides a baseline by estimating the value function, it reduces the variance of the policy gradient, improves training stability, and provides a supplementary learning signal. When represented by a neural network, its training is essentially solving a nonlinear regression problem, with the goal of making the predicted value output by the model closer to the true value. The formula is as follows:

[0042]

[0043] represents the value function corresponding to the current strategy, represents the value function corresponding to the old strategy, Denotes the discounted return. The clipping operation of proximal policy optimization imposes constraints on the update of the value function through the hyperparameter ξ∈ to ensure that the updated value function Will not work with old value function The deviation is too large.

[0044] Figure 2A FIG. 2 is a flow chart illustrating an example process 200A for information processing according to some embodiments of the present disclosure. The process 200A may be implemented at the electronic device 110. Figure 1 To describe process 200A.

[0045] like Figure 2A As shown, in box 210, the electronic device 110 can obtain input information of the generative model.

[0046] For example, the electronic device 110 may obtain input instructions or context information provided to the generative model to guide the model to generate expected output.

[0047] In some embodiments, the generative model is an autoregressive model. An autoregressive model uses historical data to predict future values. The core of an autoregressive model is to gradually predict and generate the next token based on a sequence of generated or input tokens, ultimately forming a complete output sequence. Autoregressive models can include various types, such as natural language processing models, computer vision models, and time series prediction models.

[0048] In block 220 , the electronic device 110 may provide content generated by the generative model based on the input information.

[0049] Since proximal policy optimization requires calculating rewards based on complete generation results, it is necessary to wait for the generation process of all prompts in the batch to be completed before the policy gradient update can be performed. However, during the model training process, there are significant differences between the lengths of the responses generated when processing different problems, which leads to underutilization of the GPU. To effectively solve the above problem, the embodiments of the present disclosure can truncate the token generation process at a given target number l during the model training process. This method is called truncated proximal policy optimization.

[0050] Figure 2B FIG. 2 is a flow chart illustrating an example process 200B for model training according to some embodiments of the present disclosure. The process 200B may be implemented at the electronic device 110. Figure 1 To describe process 200B.

[0051] In some embodiments, the electronic device implementing process 200A and the electronic device implementing process 200B may be different electronic devices.

[0052] The training process of the generative model can be as follows:

[0053] In block 221 , the electronic device 110 may obtain a first set of tokens generated by a generative model based on a first token sequence.

[0054] In some embodiments, the number of tokens in the first set of tokens does not exceed a target number, where the target number is less than the maximum number of output tokens of the generative model. Figure 3 , the portion to the left of the dotted line represents the first token sequence 301 used as input in the current batch, which includes 8 tokens S1. The portion to the right of the dotted line represents the first group of tokens 302 generated in the current batch, which includes 10 tokens S1. The target number can be set to 10 in advance, which means that when the generative model generates 10 tokens in the current batch based on the first token sequence 301, it must stop generating tokens regardless of whether the token generation process is complete.

[0055] At block 222 , the electronic device 110 may construct a second token sequence based on the first set of tokens and the first token sequence in response to the first set of tokens satisfying a preset condition.

[0056] For example, reference Figure 3 If the first group of tokens 302 meets the preset conditions, the first group of tokens 302 and the first token sequence 301 can be spliced ​​to construct the second token sequence 303.

[0057] In some embodiments, in response to determining that the first set of tokens does not include an end token, it is determined that the first set of tokens satisfies a preset condition. An end token is a token used to explicitly indicate the end position of a sequence, specifically, Figure 3 , the end token can be EOS (End-of-Sequence) mark 305. For example, refer to Figure 3 If the first group of tokens 302 does not include the end token 305, it is determined that the first group of tokens 302 meets the preset condition. In other words, if the first group of tokens does not include the end token, it means that the token generation process has not been completed, so it can continue to be processed in the next batch.

[0058] In block 223 , the electronic device 110 may obtain a second set of tokens generated by the generative model based on the second token sequence.

[0059] For example, reference Figure 3 , the electronic device 110 may obtain a second group of tokens 304 generated by the generative model based on the second token sequence 303 , wherein the second group of tokens 304 includes an end token.

[0060] Since the model training process includes multiple training batches, the loss and gradient are calculated based on a batch of data each time the model parameters are updated. Therefore, in some embodiments, the first token sequence is included in the first batch of token sequences provided to the generative model, and the electronic device 110 can determine the second batch of token sequences based on the second token sequence, and then provide the second batch of token sequences to the generative model.

[0061] For example, reference Figure 3 , portion 306 in the figure represents the first batch, and portion 307 in the figure represents the second batch, which is the batch after the first batch. Therefore, the portion 306 to the left of the dotted line in the figure is the token sequence of the first batch provided to the generative model, the portion 306 to the right of the dotted line in the figure is the response generated based on the token sequence of the first batch, the portion 307 to the left of the dotted line in the figure is the token sequence of the second batch provided to the generative model, and the portion 307 to the right of the dotted line in the figure is the response generated based on the token sequence of the second batch.

[0062] In some embodiments, the first batch of token sequences further includes a third token sequence, and electronic device 110 may obtain a third set of tokens generated by the generative model based on the third token sequence. Furthermore, in response to the third set of tokens not meeting a preset condition, a fourth token sequence may be obtained from the training token sequence set to be added to the second batch of token sequences.

[0063] For example, reference Figure 3The first batch of token sequences also includes a third token sequence 308, which includes 4 tokens. The third group of tokens 309 generated by the generative model based on the third token sequence includes 8 S2 tokens and one end token 305. Since the third group of tokens 309 includes the end token 305, it means that the third group of tokens 309 has completed token generation during the processing of the first batch. Therefore, a fourth token sequence 310 can be obtained from the training token sequence set to be added to the token sequence of the second batch. The fourth token sequence 310 includes 6 S5 tokens.

[0064] In some embodiments, during model training, the number of token sequences processed in each batch can be a fixed value.

[0065] In some embodiments, because multiple token sequences are processed simultaneously in each batch, the model is configured to utilize different computing resources to process the first batch of token sequences in parallel. For example, when generating text, the model needs to generate tokens one by one. If the model is processing a batch of input sequences, it can use multiple GPUs for parallel computing to process different parts of the batch at the same time, thereby improving efficiency.

[0066] At block 224 , the electronic device 110 may train a generative model based on at least the generated first set of tokens and the generated second set of tokens.

[0067] Generalized advantage estimation is a method for efficiently computing advantage functions in reinforcement learning. Generalized advantage estimation can combine advantage estimates of different step lengths by applying an exponentially weighted average controlled by the parameter λ to achieve bias-variance balance, thereby more accurately evaluating the relative advantages of actions.

[0068] In some embodiments, a policy model is a component in reinforcement learning that defines how an agent chooses actions given the state of the environment. Figure 4 , the first advantage information and the second advantage information represent the value of the advantage function, and the electronic device 110 can determine the first advantage information based on the first set of tokens 401 to train the policy model associated with the generative model. The second advantage information is determined based on the second set of tokens 402 to train the policy model. For example, Figure 4 The token training policy model in the middle left diagonal shaded area.

[0069] In some embodiments, based on the target number, weight information corresponding to different tokens in the first set of tokens is determined, and then first advantage information is determined based on the weight information and the first set of tokens. For example, the formula for calculating the advantage function using the generalized advantage estimation is as follows:

[0070]

[0071] (γλ)…(γλ) l-t-1 This is the weight information corresponding to different tokens. The temporal difference residual (TD residual) is an indicator in reinforcement learning that measures the difference between the current state value function prediction and the actual reward, that is, the subsequent state value function estimate. It reflects the deviation between the prediction and the actual. Updating the value function through this deviation can make the model prediction closer to the actual, thereby improving the policy performance. The formula is as follows:

[0072] δ t =r t +γV(s t+1 )-V(s t )

[0073] r t represents the reward obtained at time step t, γ is the discount factor used to measure the importance of future rewards relative to immediate rewards, V(s t+1 ) represents the next state s t+1 The estimated value of the state value function, V(s t ) represents the current state s t The estimated value of the value function.

[0074] In some embodiments, the value model is a component in reinforcement learning that helps the agent judge "how good the current situation is" or "the potential value of a certain action" by evaluating the long-term expected reward of a state or state-action pair. The electronic device 110 can determine value information based on the first set of tokens and the second set of tokens in response to the second set of tokens not meeting the preset conditions. Then, based on the value information, the value model associated with the generative model is trained. For example, referring to Figure 4 , the first group of tokens 403 includes the end token, so the first group of tokens 403 meets the preset condition, and the second group of tokens 404 does not include the end token, so the second group of tokens 404 does not meet the preset condition, you can apply Figure 4 The token training value model in the middle right diagonal shaded area. This method allows the policy model and value model to be independent of each other but updated synchronously.

[0075] During the reinforcement learning training process, due to the limitation of the token generation process similar to the barrel effect, and the generation time is related to the maximum response length of the interaction, the overall reinforcement learning training efficiency is affected. L is the maximum number of output tokens for the generative model, which can reduce generation time by approximately k times. Furthermore, during model training, each cross-round response token is trained once by both the policy model and the value model, thus reducing training time by a factor of k. This approach significantly improves model training efficiency.

[0076] Example devices and equipment

[0077] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 5 1 shows a schematic structural block diagram of an example apparatus 500 for information processing according to some embodiments of the present disclosure. Apparatus 500 may be implemented as or included in electronic device 110. Each module / component in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0078] like Figure 5 As shown, the device 500 includes: an acquisition module 510, configured to acquire input information of a generative model; and a providing module 520, configured to provide content generated by the generative model based on the input information, wherein the generative model is trained based on the following process: acquiring a first group of tokens generated by the generative model based on a first token sequence, the number of tokens in the first group of tokens does not exceed a target number, wherein the target number is less than the maximum number of output tokens of the generative model; in response to the first group of tokens satisfying a preset condition, constructing a second token sequence based on the first group of tokens and the first token sequence; acquiring a second group of tokens generated by the generative model based on the second token sequence; and training the generative model based on at least the generated first group of tokens and the generated second group of tokens.

[0079] In some embodiments, the apparatus 500 is further configured to determine that a preset condition is satisfied in the first set of tokens in response to determining that the first set of tokens does not include an end token.

[0080] In some embodiments, the first token sequence is included in a first batch of token sequences provided to the generative model, and the apparatus 500 is further configured to determine a second batch of token sequences based on the second token sequence; and provide the second batch of token sequences to the generative model.

[0081] In some embodiments, the token sequence of the first batch also includes a third token sequence, and the device 500 is further configured to obtain a third group of tokens generated by the generative model based on the third token sequence; and in response to the third group of tokens not meeting the preset conditions, obtain a fourth token sequence from the training token sequence set to add to the token sequence of the second batch.

[0082] In some embodiments, the model is configured to process first batches of token sequences in parallel using different computing resources.

[0083] In some embodiments, providing module 520 is configured to determine first advantage information based on the first set of tokens to train a policy model associated with the generative model; and determine second advantage information based on the second set of tokens to train the policy model.

[0084] In some embodiments, the providing module 520 is configured to determine weight information corresponding to different tokens in the first set of tokens based on the target number; and determine first advantage information based on the weight information and the first set of tokens.

[0085] In some embodiments, the providing module 520 is configured to determine value information based on the first and second sets of tokens in response to the second set of tokens not satisfying a preset condition; and train a value model associated with the generative model based on the value information.

[0086] In some embodiments, the generative model is an autoregressive model.

[0087] like Figure 6 As shown, electronic device 600 is in the form of a general electronic device. Components of electronic device 600 may include, but are not limited to, at least one processor 610 or processing unit, memory 620, storage device 630, one or more communication units 660, one or more input devices 650, and one or more output devices 660. Processor 610 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multi-processor system, multiple processors execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 600.

[0088] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 600.

[0089] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0090] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0091] The input device 650 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 660 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 600, or with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0092] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0093] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0095] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0096] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0097] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An information processing method, comprising: Get the input information of the generative model; as well as Providing content generated by the generative model based on the input information, wherein the generative model is trained based on the following process: Obtaining a first set of tokens generated by the generative model based on a first token sequence, wherein the number of tokens in the first set of tokens does not exceed a target number, wherein the target number is less than a maximum number of output tokens of the generative model; In response to the first set of tokens satisfying a preset condition, constructing a second token sequence based on the first set of tokens and the first token sequence; Obtaining a second set of tokens generated by the generative model based on the second token sequence; as well as The generative model is trained based on at least the generated first set of tokens and the generated second set of tokens.

2. The method according to claim 1, further comprising: In response to determining that the first set of tokens does not include an end token, it is determined that the preset condition is satisfied in the first set of tokens.

3. The method of claim 1 , wherein the first token sequence is included in a first batch of token sequences provided to the generative model, the method further comprising: Determining a token sequence for a second batch based on the second token sequence; as well as The generative model is provided with the second batch of token sequences.

4. The method of claim 3, wherein the first batch of token sequences further comprises a third token sequence, the method further comprising: Obtaining a third set of tokens generated by the generative model based on the third token sequence; as well as In response to the third group of tokens not satisfying the preset condition, a fourth token sequence is obtained from the training token sequence set to be added to the second batch of token sequences.

5. The method of claim 3, wherein the model is configured to utilize different computing resources to process the first batch of token sequences in parallel.

6. The method of claim 1 , wherein training the generative model based on at least the generated first set of tokens and the generated second set of tokens comprises: determining first advantage information based on the first set of tokens to train a policy model associated with the generative model; as well as Second advantage information is determined based on the second set of tokens to train the policy model.

7. The method of claim 6, wherein determining first advantage information based on the first set of tokens comprises: determining weight information corresponding to different tokens in the first set of tokens based on the target number; as well as The first advantage information is determined based on the weight information and the first set of tokens.

8. The method of claim 1 , wherein training the generative model based on at least the generated first set of tokens and the generated second set of tokens comprises: in response to the second set of tokens not satisfying the pre-set condition, determining value information based on the first set of tokens and the second set of tokens; as well as Based on the value information, a value model associated with the generative model is trained.

9. The method of claim 1, wherein the generative model is an autoregressive model.

10. An apparatus for information processing, comprising: An acquisition module, configured to obtain input information of the generative model; as well as A providing module is configured to provide content generated by the generative model based on the input information, wherein the generative model is trained based on the following process: Obtaining a first set of tokens generated by the generative model based on a first token sequence, wherein the number of tokens in the first set of tokens does not exceed a target number, wherein the target number is less than a maximum number of output tokens of the generative model; In response to the first set of tokens satisfying a preset condition, constructing a second token sequence based on the first set of tokens and the first token sequence; Obtaining a second set of tokens generated by the generative model based on the second token sequence; as well as The generative model is trained based on at least the generated first set of tokens and the generated second set of tokens.

11. An electronic device comprising: at least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor. 12 . A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions can be executed by a processor to implement the method according to claim 1 .

13. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 9.

Images