Data processing method, device, equipment, computer program product and storage medium

Abstract

Provided are a data processing method, apparatus, device, computer program product, and computer-readable storage medium. The method encodes the historical word-gram sequence of the text sequence using multiple hidden states of different categories associated with the scene category in combination with the specific scene category and requirements of the text sequence, and updates the multiple hidden states in real time based on the newly generated word-grams, thereby generating new word-grams based on the updated multiple hidden states. The method can encode the historical word-gram sequence using multiple hidden states, so that long text sequences can be efficiently generated through its data compression capability, reducing video memory requirements and computing overhead, while avoiding the problem of reduced generation quality due to excessive data compression. By encoding multiple hidden states in combination with the specific scene category and requirements of the text sequence, the model can improve its understanding of the structure and logical relationship of the text data, thereby achieving more accurate and reliable model reasoning.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a data processing method, apparatus, device, computer program product, and storage medium. Background Art

[0002] In recent years, large language models based on the Transformer architecture have attracted widespread attention and application in the field of natural language processing. With their powerful understanding and generation capabilities, these models have achieved outstanding results in tasks such as machine translation, question-answering systems, and text summarization. However, when faced with the task of generating long sequences of text, these models face significant challenges.

[0003] Specifically, the language model based on the Transformer architecture needs to encode and store each historical token and calculate the attention weight on each historical token. This mechanism causes the model to process a large amount of historical information when generating long texts, resulting in huge video memory requirements and computing overhead, which is proportional to the number of historical tokens. Taking machine translation as an example, when generating a longer sentence, the model needs to remember and process all previously generated words, which puts tremendous pressure on video memory and computing resources. The same problem also exists in other tasks that require the generation of long texts, such as text summarization, dialogue systems, etc.

[0004] Therefore, an efficient data processing method is needed to accurately and reliably generate long text sequences while reducing video memory requirements and computational overhead. Summary of the invention

[0005] In order to solve the above problems, the present disclosure proposes a text sequence generation method, which combines the scenario requirements of the text sequence, calculates the hidden states of multiple dimensions respectively according to the data structure of the text sequence, so as to encode the historical word sequence, thereby reducing the video memory requirements and computing overhead, and accurately and reliably generates long text sequences.

[0006] Embodiments of the present disclosure provide a data processing method, apparatus, device, computer program product, and computer-readable storage medium.

[0007] An embodiment of the present disclosure provides a data processing method, comprising: obtaining a current generated word-gram of a text sequence, wherein the current generated word-gram is the latest generated word-gram in the text sequence; obtaining previous states of a plurality of hidden states associated with the text sequence, wherein each hidden state of the plurality of hidden states is used to represent text information related to different categories of the text sequence, and the previous states of the plurality of hidden states are used to represent text information related to different categories of words generated before the current generated word-gram of the text sequence, and the different categories are associated with scene categories of the text sequence; based on the current generated word-gram of the text sequence, for each hidden state of the plurality of hidden states, encoding the previous state of the hidden state using the text information related to the category corresponding to the current generated word-gram to generate the current state of the hidden state, the current state of the hidden state being used to represent text information related to the category of words of the text sequence including the current generated word-gram; based on the current state of each hidden state of the plurality of hidden states, generating the next generated word-gram of the text sequence.

[0008] According to an embodiment of the present disclosure, based on the current generated word-gram of the text sequence, for each hidden state in the multiple hidden states, using the text information of the current generated word-gram related to the category corresponding to the hidden state, the previous state of the hidden state is encoded to generate the current state of the hidden state, including: through a state space model, using the text information of the current generated word-gram related to the category corresponding to the hidden state, the previous state of the hidden state is encoded to generate the current state of the multiple hidden states; based on the current state of each hidden state in the multiple hidden states, generating the next generated word-gram of the text sequence includes: through the state space model, based on the combination of the current state of each hidden state in the multiple hidden states, generating the next generated word-gram of the text sequence; wherein the state space model takes the current generated word-gram of the text sequence as input and the next generated word-gram of the text sequence as output.

[0009] According to an embodiment of the present disclosure, the data processing method further includes: using a pre-trained large language model as a teacher model, the state space model as a student model, and training the state space model through model distillation and multi-task learning.

[0010] According to an embodiment of the present disclosure, training the state-space model through model distillation and multi-task learning includes: training the state-space model by minimizing the difference between the predicted probability distribution of the pre-trained large language model and the state-space model for generating the next generated word unit, and maximizing the probability of the state-space model generating the next generated word unit in an autoregressive manner.

[0011] According to an embodiment of the present disclosure, generating the next generated word-gram of the text sequence based on the current state of each hidden state in a plurality of hidden states includes: determining a predicted probability distribution for generating the next generated word-gram through the state space model based on a combination of the current state of each hidden state in a plurality of hidden states; and generating the next generated word-gram of the text sequence based on the determined predicted probability distribution for generating the next generated word-gram.

[0012] According to an embodiment of the present disclosure, the data processing method also includes: determining the scene category of the text sequence based on the input text corresponding to the text sequence; determining the multiple hidden states associated with the text sequence based on the scene category of the text sequence; and determining the initial state of the multiple hidden states based on the input text.

[0013] According to an embodiment of the present disclosure, the scene category of the text sequence is a medical scene; wherein the multiple hidden states associated with the text sequence include at least one of the following: a hidden state of a prompt part in an input text corresponding to the text sequence; a hidden state of a patient's admission record; and a hidden state of a patient's disease course management.

[0014] An embodiment of the present disclosure provides a data processing device, comprising: a data acquisition module, configured to acquire a current generated word-gram of a text sequence, and acquire previous states of a plurality of hidden states associated with the text sequence, wherein the current generated word-gram is the most recently generated word-gram in the text sequence, each of the plurality of hidden states is used to represent text information related to different categories of the text sequence, and the previous states of the plurality of hidden states are used to represent text information related to different categories of the word-gram generated before the current generated word-gram of the text sequence, and the different categories are related to the scene category of the text sequence. Association; a state updating module, configured to, based on the current generated word-gram of the text sequence, for each hidden state in the multiple hidden states, encode the previous state of the hidden state using the text information related to the category corresponding to the current generated word-gram to generate a current state of the hidden state, wherein the current state of the hidden state is used to represent the text information related to the category of the word-gram of the text sequence including the current generated word-gram; and a word-gram generating module, configured to generate the next generated word-gram of the text sequence based on the current state of each hidden state in the multiple hidden states.

[0015] An embodiment of the present disclosure provides a data processing device, including: one or more processors; and one or more memories, wherein a computer executable program is stored in the one or more memories, and when the computer executable program is executed by the processor, the data processing method as described above is performed.

[0016] An embodiment of the present disclosure provides a computer-readable storage medium on which computer-executable instructions are stored. When the instructions are executed by a processor, the instructions are used to implement the data processing method as described above.

[0017] The embodiments of the present disclosure provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data processing method according to the embodiments of the present disclosure.

[0018] The method provided by the embodiment of the present disclosure combines the specific scene categories and requirements of the text sequence, utilizes multiple hidden states of different categories associated with the scene category to encode the historical word-gram sequence of the text sequence, and updates the multiple hidden states in real time based on the newly generated word-grams, thereby generating new word-grams based on the updated multiple hidden states. The method of the embodiment of the present disclosure can utilize multiple hidden states to encode the historical word-gram sequence, so that long text sequences can be efficiently generated through its data compression capabilities, reducing video memory requirements and computing overhead, while avoiding the problem of reduced generation quality due to excessive data compression. In addition, by encoding multiple hidden states in combination with the specific scene categories and requirements of the text sequence, the model can improve its understanding of the structure and logical relationships of the text data, thereby achieving more accurate and reliable model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0020] Figure 1 is a flow chart showing a data processing method according to an embodiment of the present disclosure;

[0021] Figure 2 is a schematic diagram illustrating a data processing method according to an embodiment of the present disclosure;

[0022] Figure 3 is a schematic diagram illustrating a process of determining a plurality of hidden states according to an embodiment of the present disclosure;

[0023] Figure 4 is a schematic diagram showing a data processing device according to an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram showing a data processing device according to an embodiment of the present disclosure; and

[0025] Figure 6 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0027] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0028] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0030] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0031] The data processing method disclosed in the present invention may be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. By studying the design principles and implementation methods of various intelligent machines, artificial intelligence enables the data processing method disclosed in the present invention to have the function of accurately generating long text sequences in real time with lower storage requirements and computing power.

[0032] The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0033] As mentioned above, a large language model based on the Transformer architecture needs to encode and store each historical word-gram and calculate the attention weight on each historical word-gram. Historical word-grams refer to text units (such as words or subwords) that have been generated or input previously. These historical word-grams are crucial to the model's understanding and generation capabilities because they provide contextual information that can help the model understand how the text to be generated should connect with the previous text and maintain semantic coherence. When generating text, the model needs to consider historical word-grams to generate the next word-gram. By encoding and calculating attention on historical word-grams, the model can better capture the dependencies and contextual information between texts, thereby generating more reasonable and coherent text. Therefore, historical word-grams are crucial to the generation process of the model, but they also bring challenges in video memory requirements and computing power, especially when the model generates long texts, which brings huge video memory requirements and computing overhead because the model needs to process and store a large amount of historical word-gram information.

[0034] In order to address the challenges faced by large language models based on the Transformer architecture when generating long sequences of text, historical word units can be compressed into a hidden state. For example, the attention mechanism can be used to aggregate the information of historical word units and compress them into a hidden state. The compressed hidden state is used to represent the entire historical information to reduce the video memory and computing required to encode and store historical word units, thereby improving generation efficiency.

[0035] However, compressing historical tokens into a hidden state may result in the loss of important historical information, which may affect the quality of long sequence text generation. Because in the process of compressing historical information, due to the compression and loss of information, the hidden state may not be able to fully retain the details of all historical information, and some key contextual information may be discarded or blurred, resulting in the model lacking sufficient contextual understanding when generating text, thus affecting the coherence and semantic accuracy of the text.

[0036] Although the above method provides an effective way to reduce the computational burden, it may sacrifice a certain amount of generation quality when processing long sequence text generation due to the possible loss of important information. Therefore, in practical applications, it is necessary to balance the balance between computational efficiency and generation quality and determine the method that suits the needs of specific task scenarios.

[0037] Based on this, the present disclosure proposes a text sequence generation method, which combines the scenario requirements of the text sequence and calculates the hidden states of multiple dimensions according to the data structure of the text sequence to encode the historical word sequence, thereby reducing the video memory requirements and computing overhead, and accurately and reliably generates long text sequences.

[0038] The method provided by the embodiment of the present disclosure combines the specific scene categories and requirements of the text sequence, utilizes multiple hidden states of different categories associated with the scene category to encode the historical word-gram sequence of the text sequence, and updates the multiple hidden states in real time based on the newly generated word-grams, thereby generating new word-grams based on the updated multiple hidden states. The method of the embodiment of the present disclosure can utilize multiple hidden states to encode the historical word-gram sequence, so that long text sequences can be efficiently generated through its data compression capabilities, reducing video memory requirements and computing overhead, while avoiding the problem of reduced generation quality due to excessive data compression. In addition, by encoding multiple hidden states in combination with the specific scene categories and requirements of the text sequence, the model can improve its understanding of the structure and logical relationships of the text data, thereby achieving more accurate and reliable model reasoning.

[0039] Figure 1 is a flow chart illustrating a data processing method 100 according to an embodiment of the present disclosure. Figure 2 is a schematic diagram illustrating a data processing method according to an embodiment of the present disclosure.

[0040] In step S101, a currently generated word-gram of a text sequence may be obtained, wherein the currently generated word-gram is the most recently generated word-gram in the text sequence.

[0041] For example, the currently generated word-gram may be a new word-gram generated by the model at the current time step during the text generation process. Optionally, the word-gram, as the smallest language unit of the text sequence, may be a character, a word, a phrase, a sentence, or other language units. The appropriate word-gram form may be selected according to specific application requirements, and the present disclosure does not limit this.

[0042] Optionally, the most recently generated word-gram (i.e., the currently generated word-gram) can be used for subsequent text generation, i.e., the subsequent text generation depends on the contextual basis in the text sequence and the currently generated word-gram to generate a text sequence that conforms to language rules and has a specific intention.

[0043] In step S102, previous states of multiple hidden states associated with the text sequence can be obtained, wherein each hidden state of the multiple hidden states can be used to represent text information related to different categories of the text sequence, and the previous states of the multiple hidden states can be used to represent text information related to different categories of word-grams generated before the currently generated word-gram of the text sequence, and the different categories can be associated with scene categories of the text sequence.

[0044] The hidden states disclosed in the present invention can be used to characterize text information related to different categories of a text sequence to represent the historical information of the text sequence. Each hidden state can be used to capture the dynamic changes of text information under different categories in the text sequence. By using multiple hidden states, the complexity of the text sequence can be better characterized.

[0045] In an embodiment of the present disclosure, historical words are encoded and compressed through multiple hidden states to summarize meaningful information from the historical words of a text sequence, thereby reducing the amount of data that needs to be processed while retaining the most important contextual information, thereby improving the accuracy and consistency of text generation.

[0046] Furthermore, in an embodiment of the present disclosure, hidden states may be associated with scene categories, for example, multiple hidden states may be constructed based on scene categories of text sequences, so that the text generation strategy can be dynamically adjusted to generate more expected text in specific application scenarios.

[0047] Optionally, the semantic information of the text sequence under different categories can be used to perform data compression on the historical word units. Optionally, the semantic information of the text sequence under different categories can be related to the specific scene category of the text sequence. For example, the different categories of the text sequence can be determined based on the specific scene category of the text sequence, which can correspond to different data categories under the specific scene category of the text sequence.

[0048] Optionally, multiple hidden states constructed based on scene categories of text sequences can be used to encode historical word sequences to summarize and compress the information covered by historical word sequences from different categories, thereby helping the model understand the data structure of text data and the logical relationships in the data, and achieving more accurate and reliable text generation.

[0049] Optionally, the data processing method of the present disclosure can be applied to various scene categories, and various dimensions of the text sequence can be determined based on the data structure under a specific scene category, thereby determining multiple hidden states.

[0050] According to an embodiment of the present disclosure, the scene category of the text sequence can be a medical scene. Optionally, the data processing method of the present disclosure can be introduced by taking the generation of text sequences in a medical scene as an example, but it should be understood that this is only an example of the application scene of the method of the present disclosure and not a limitation.

[0051] For text sequence generation in medical scenarios, since the data in the text sequence usually has a high degree of specialization, information loss in this scenario may have an important impact on the final medical diagnosis, and since medical text data usually has a specific data structure and logical relationship, its data can have different data categories, and the medical information included in each data category may have an important impact on the final medical diagnosis. For example, the data categories of medical text data may include data categories such as diagnosis, treatment, and examination, that is, different data parts of the medical text data can correspond to different categories in these data categories. Therefore, for these different data categories, hidden states can be constructed accordingly to fully represent the corresponding features of the text sequence, so as to help the large language model understand the data structure of the text data and the logical relationship in the data, thereby achieving more accurate and reliable text generation.

[0052] According to an embodiment of the present disclosure, the multiple hidden states associated with the text sequence may include at least one of the following: a hidden state of a prompt portion in an input text corresponding to the text sequence; a hidden state of a patient's admission record; a hidden state of a patient's disease course management.

[0053] Optionally, as described above, for text sequence generation in medical scenarios, its data categories may include data categories such as diagnosis, treatment, examination, etc. Therefore, hidden states corresponding to these data categories may be constructed for text sequences in medical scenarios. For example, the hidden state of a patient's admission record and the hidden state of a patient's course of disease management, etc. In addition, the hidden state may also include prior knowledge of the input text, such as the hidden state of the prompt part in the input text.

[0054] Of course, it should be understood that the above-mentioned various hidden states are only used as examples and not as limitations in the present disclosure, and the data processing method of the present disclosure can also construct other different hidden states.

[0055] According to an embodiment of the present disclosure, the data processing method may further include: determining a scene category of the text sequence based on an input text corresponding to the text sequence; determining the multiple hidden states associated with the text sequence based on the scene category of the text sequence; and determining an initial state of the multiple hidden states based on the input text.

[0056] Optionally, the scene category of the text sequence to be generated can be determined based on the input text of the model, and then the corresponding hidden state can be constructed for the specific scene category, and the initial state of the constructed hidden state can be determined based on the information of the input text to start text generation based on the initial state. By combining the scene category and the prior knowledge of the input text to determine the hidden states of different categories, the complex semantic features of the text sequence can be better captured and the generation quality can be improved.

[0057] Next, in step S103, based on the current generated word-gram of the text sequence, for each hidden state of the multiple hidden states, the previous state of the hidden state can be encoded using the text information of the current generated word-gram related to the category corresponding to the hidden state to generate a current state of the hidden state, and the current state of the hidden state can be used to represent the text information related to the category of the word-gram of the text sequence including the current generated word-gram.

[0058] like Figure 2 As shown, the latest word-gram generated by the model (i.e., the currently generated word-gram) can be used as input, and the current generated word-gram can be used to update multiple hidden states, that is, the current state of multiple hidden states is generated based on the current generated word-gram and the previous state of multiple hidden states. Among them, each hidden state can represent the hidden state of historical information under the dimension obtained by aggregating and compressing the relevant information in the historical word-gram sequence (for example, the information under the dimension corresponding to the hidden state). Each hidden state can be represented as a corresponding vector, which can contain the encoding and summary of the relevant information in the historical word-gram sequence, that is, a compact representation containing the essence of historical related information.

[0059] Optionally, multiple hidden states can be used ( , ) to encode the historical word sequence, where , They correspond to different categories of text sequences.

[0060] As an example, for the text generation task in the medical scenario, It can correspond to prior knowledge of the input text, such as the hidden state of the prompt part in the input text, It can correspond to the hidden state of the patient's disease management. It could correspond to a hidden state of a patient's admission record, and so on.

[0061] Figure 3 is a schematic diagram illustrating a process of determining a plurality of hidden states according to an embodiment of the present disclosure.

[0062] Therefore, assuming that the current time is , correspondingly, the current generated word is , then for each hidden state given as an example above , The update can be expressed as follows:

[0063]

[0064] in, , Corresponding to each hidden state , At the last moment The state of , that is, the previous state of multiple hidden states, and , Corresponding to each hidden state , At the present moment The state of , that is, the current state of multiple hidden states, , Represent the state transformation matrix for each hidden state respectively. It can represent the influence of the current generated word on the update of each hidden state, so based on The update of the hidden state may correspond to encoding the previous state of the hidden state using the text information of the current generated word associated with the category corresponding to the hidden state. Note that for ease of understanding, for each hidden state, are shown as the same, but in fact It can be different for each hidden state. In addition, the method of encoding the previous state of the hidden state based on the text information related to the category corresponding to the hidden state using the currently generated word unit can also include other encoding methods in addition to the above-mentioned linear superposition method. This disclosure is only used as an example and not limited.

[0065] like Figure 3 As shown, only the hidden state The process of updating the hidden state and generating the next word-unit is described as an example. Specifically, the previous state of the hidden state can be updated by converting the state transition matrix and adjusting the input current generated word-unit, and the updated hidden state (i.e., the current state of the hidden state) can be iteratively used as the input of the next round of hidden state update (i.e., the previous state of the hidden state), so as to continuously update multiple hidden states based on the new word-unit generation of the model.

[0066] Finally, in step S104, the next generated word-gram of the text sequence may be generated based on the current state of each hidden state in the plurality of hidden states.

[0067] Optionally, the next generated word of the text sequence can be jointly determined based on the combination of the current states of multiple hidden states. Figure 3 As shown, the next generated word is The current state can be represented as multiple hidden states , Combinations of ,in Represents a combinatorial function.

[0068] According to an embodiment of the present disclosure, based on the current generated word-gram of the text sequence, for each hidden state in the multiple hidden states, encoding the previous state of the hidden state using the text information of the current generated word-gram related to the category corresponding to the hidden state to generate the current state of the hidden state may include: encoding the previous state of the hidden state using the text information of the current generated word-gram related to the category corresponding to the hidden state through a state space model to generate the current state of the multiple hidden states; generating the next generated word-gram of the text sequence based on the current state of each hidden state in the multiple hidden states may include: generating the next generated word-gram of the text sequence based on a combination of the current states of each hidden state in the multiple hidden states through the state space model; wherein the state space model takes the current generated word-gram of the text sequence as input and takes the next generated word-gram of the text sequence as output.

[0069] Optionally, the above steps S101-S104 of the present disclosure may be performed by a trained state space model, such as Figure 2 The state space model can take the latest word-gram generated by the model (i.e., the current generated word-gram) as input, and use the input to update multiple hidden states, so as to obtain a new word-gram to be generated (i.e., the next generated word-gram) based on the updated hidden state, and use the next generated word-gram as the output of this round of text generation, i.e., the input of the next round of text generation.

[0070] According to an embodiment of the present disclosure, the data processing method may further include: using a pre-trained large language model as a teacher model, the state space model as a student model, and training the state space model through model distillation and multi-task learning.

[0071] Optionally, the data processing method of the present disclosure may adopt a multi-task training method combined with model distillation to generate a state-space model for text sequence generation. In particular, a pre-trained large language model based on the Transformer architecture may be used as a teacher model, and the state-space model may be used as a student model, and the training of the state-space model may be optimized through multiple training objectives.

[0072] According to an embodiment of the present disclosure, training the state-space model through model distillation and multi-task learning may include: training the state-space model by minimizing the difference between the predicted probability distribution of the pre-trained large language model and the state-space model for generating the next generated word unit, and maximizing the probability of the state-space model generating the next generated word unit in an autoregressive manner.

[0073] Optionally, the above-mentioned multiple training objectives (i.e., multiple task objectives) may include making the predicted probability distribution for generating the next generated word-unit predicted by the state-space model of the present invention as similar as possible to that of the teacher model, and making the state-space model of the present invention autoregressively generate the conditional probability of the next generated word-unit.

[0074] For example, for the first training objective given as an example above, it can correspond to minimizing the difference between the predicted probability distributions of the teacher model and the student model for generating the next generated word, such as minimizing the KL (Kullback-Leibler) divergence between the predicted probability distributions, so that the prediction results of the student model are closer to the prediction results of the teacher model (which are close to the real data distribution), thereby improving the generalization ability and accuracy of the student model (i.e., the state space model of the present disclosure).

[0075] Through the above-mentioned model distillation, the prediction of the state space model of the present invention can inherit the powerful understanding and generation capabilities of the large language model based on the Transformer architecture as much as possible, while reducing the scale and complexity of the model.

[0076] For another example, for the second training objective given as an example above, it may correspond to maximizing the probability of the state-space model of the present disclosure generating the next generated word-unit in an autoregressive manner, thereby improving the autoregressive generation probability.

[0077] For multi-task training including the above two training objectives, the loss function It can be expressed as follows:

[0078]

[0079] in, can correspond to the loss function of the first training objective mentioned above, and It can correspond to the loss function of the second training objective mentioned above, represents the learning weight of the first training task, represents the learning weight of the second training task, represents the probability of the student model generating the next word through autoregression, is the probability that the teacher model generates the next word through autoregression, Indicates the current generated word Length, Represents an entire text sequence.

[0080] in, can correspond to the loss function of the first training objective mentioned above, and It can correspond to the loss function of the second training objective mentioned above, represents the learning weight of the first training task, represents the learning weight of the second training task, represents the probability of the student model generating the next word through autoregression, is the probability that the teacher model generates the next word through autoregression, Indicates the current generated word Length, Represents an entire text sequence.

[0081] Through the model compression capability of model distillation, the obtained small state-space model can be used to approximate the predictive ability of a large language model based on the Transformer architecture, thereby reducing model complexity and reducing computational and storage overhead while maintaining high predictive performance.

[0082] According to an embodiment of the present disclosure, generating the next generated word-gram of the text sequence based on the current state of each hidden state in a plurality of hidden states may include: determining a predicted probability distribution for generating the next generated word-gram through the state space model based on a combination of the current state of each hidden state in a plurality of hidden states; and generating the next generated word-gram of the text sequence based on the determined predicted probability distribution for generating the next generated word-gram.

[0083] Optionally, the state space model of the present disclosure may be used to predict the probability distribution for generating the next generated word-gram using the current states of the plurality of hidden states, and then the next word-gram may be generated based on the probability distribution.

[0084] As an example, the next word-unit can be generated by sampling based on the predicted probability distribution. For example, for each word-unit in the vocabulary symbol table , the probability of being selected as the next generated word can be expressed as follows:

[0085]

[0086] in represents a table of vocabulary symbols, Represents word embedding operation.

[0087] As above reference Figure 1-Figure 3 The data processing method disclosed in the present invention is described, which combines the specific scene categories and requirements of the text sequence, uses multiple hidden states of different categories associated with the scene category to encode the historical word-gram sequence of the text sequence, and updates the multiple hidden states in real time based on the newly generated word-gram, so as to generate new word-grams based on the updated multiple hidden states. Through this method, multiple hidden states can be used to encode the historical word-gram sequence, so that long text sequences can be efficiently generated through its data compression capability, reducing the demand for video memory and computing overhead, while avoiding the problem of reduced generation quality due to excessive data compression. In addition, by encoding multiple hidden states in combination with the specific scene categories and requirements of the text sequence, the model can improve the understanding of the structure and logical relationship of the text data, thereby achieving more accurate and reliable model reasoning. Further, by adopting model distillation in model training, the prediction of the state space model disclosed in the present invention can inherit the powerful understanding and generation capabilities of the large language model based on the Transformer architecture as much as possible, while reducing the scale and complexity of the model, and reducing the computing and storage overhead.

[0088] Figure 4 4 is a schematic diagram showing a data processing device 400 according to an embodiment of the present disclosure.

[0089] According to an embodiment of the present disclosure, the data processing device 400 may include a data acquisition module 401 , a state update module 402 and a word unit generation module 403 .

[0090] The data acquisition module 401 can be configured to acquire the current generated word-gram of the text sequence, and acquire the previous states of multiple hidden states associated with the text sequence, wherein the current generated word-gram is the most recently generated word-gram in the text sequence, each of the multiple hidden states can be used to characterize text information related to different categories of the text sequence, and the previous states of the multiple hidden states can be used to characterize text information related to different categories of the word-gram generated before the current generated word-gram of the text sequence, and the different categories can be associated with the scene category of the text sequence. Optionally, the state update module 402 can perform the operations described above with reference to steps S101 and S102.

[0091] For example, the currently generated word-gram may be a new word-gram generated by the model at the current time step during the text generation process. Optionally, the word-gram, as the smallest language unit of the text sequence, may be a character, a word, a phrase, a sentence, or other language units. The appropriate word-gram form may be selected according to specific application requirements, and the present disclosure does not limit this.

[0092] Optionally, the most recently generated word-gram (i.e., the currently generated word-gram) can be used for subsequent text generation, i.e., the subsequent text generation depends on the contextual basis in the text sequence and the currently generated word-gram to generate a text sequence that conforms to language rules and has a specific intention.

[0093] The state update module 402 can be configured to update the previous state of the multiple hidden states based on the current generated word-gram of the text sequence and the previous state of the multiple hidden states using the current generated word-gram to generate the current state of the multiple hidden states, wherein the multiple hidden states correspond to the multiple hidden states associated with the text sequence, and the previous state of the multiple hidden states is the encoding of the word-gram generated before the current generated word-gram of the text sequence under the different categories, and the different categories are associated with the scene category of the text sequence. Optionally, the state update module 402 can perform the operations described above with reference to step S102.

[0094] Optionally, the data compression of the historical word units can be performed using the semantic information of the text sequence in different categories. The semantic information of the text sequence in different categories can be related to the specific scene category of the text sequence. For example, the different categories of the text sequence can be determined based on the specific scene category of the text sequence, which can correspond to different data categories under the specific scene category of the text sequence.

[0095] Optionally, various dimensions of the text sequence may be determined based on a data structure under a specific scene category, thereby determining a plurality of hidden states.

[0096] For example, for text sequence generation in medical scenarios, its data categories may include data categories such as diagnosis, treatment, examination, etc. Therefore, hidden states corresponding to these data categories can be constructed for text sequences in medical scenarios. For example, the hidden state of the patient's admission record and the hidden state of the patient's course of disease management, etc. In addition, the hidden state may also include prior knowledge of the input text, such as the hidden state of the prompt part in the input text.

[0097] Optionally, the state update module 402 can determine the scene category of the text sequence to be generated based on the input text of the model, and then construct the corresponding hidden state for the specific scene category, and can determine the initial state of the constructed hidden state based on the information of the input text to start text generation based on the initial state. By combining the scene category and the prior knowledge of the input text to determine the hidden states of different categories, the complex semantic features of the text sequence can be better captured and the generation quality can be improved.

[0098] Optionally, the state update module 402 can utilize multiple hidden states constructed based on the scene categories of the text sequence to encode the historical word sequence, so as to summarize and compress the information covered by the historical word sequence from different categories, thereby helping the model understand the data structure of the text data and the logical relationship in the data, and achieve more accurate and reliable text generation.

[0099] Optionally, the state update module 402 can use the latest word-gram generated by the model (i.e., the current generated word-gram) as input, and use the current generated word-gram to update multiple hidden states, that is, generate the current state of multiple hidden states based on the current generated word-gram and the previous state of multiple hidden states. Each hidden state can represent the hidden state of historical information under the dimension obtained by aggregating and compressing the relevant information in the historical word-gram sequence (e.g., the information under the dimension corresponding to the hidden state). Each hidden state can be represented as a corresponding vector, which can contain the encoding and summary of the relevant information in the historical word-gram sequence, that is, a compact representation containing the essence of historical related information.

[0100] The word-gram generation module 403 may be configured to generate a next generated word-gram of the text sequence based on the current state of each hidden state in the plurality of hidden states. Optionally, the word-gram generation module 403 may perform the operations described above with reference to step S103.

[0101] Optionally, the word-gram generation module 403 may jointly determine the next generated word-gram of the text sequence based on a combination of current states of a plurality of hidden states.

[0102] According to yet another aspect of the present disclosure, a data processing device is provided. Figure 5 A schematic diagram of a data processing device 2000 according to an embodiment of the present disclosure is shown.

[0103] like Figure 5 As shown, the data processing device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores a computer readable code, and when the computer readable code is run by the one or more processors 2010, the data processing method described above may be executed.

[0104] The processor in the embodiments of the present disclosure may be an integrated circuit chip with signal processing capabilities. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., which may be an X86 architecture or an ARM architecture.

[0105] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0106] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 6 The architecture of the computing device 3000 shown in FIG. Figure 6 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the data processing method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 6 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 6 One or more components of a computing device are shown.

[0107] According to another aspect of the present disclosure, a computer-readable storage medium is also provided. Computer-readable instructions are stored on the computer storage medium. When the computer-readable instructions are executed by a processor, the data processing method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0108] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the data processing method according to the embodiments of the present disclosure.

[0109] Embodiments of the present disclosure provide a data processing method, apparatus, device, computer program product, and computer-readable storage medium.

[0110] The method provided by the embodiment of the present disclosure combines the specific scene categories and requirements of the text sequence, utilizes multiple hidden states of different categories associated with the scene category to encode the historical word-gram sequence of the text sequence, and updates the multiple hidden states in real time based on the newly generated word-grams, thereby generating new word-grams based on the updated multiple hidden states. The method of the embodiment of the present disclosure can utilize multiple hidden states to encode the historical word-gram sequence, so that long text sequences can be efficiently generated through its data compression capabilities, reducing video memory requirements and computing overhead, while avoiding the problem of reduced generation quality due to excessive data compression. In addition, by encoding multiple hidden states in combination with the specific scene categories and requirements of the text sequence, the model can improve its understanding of the structure and logical relationships of the text data, thereby achieving more accurate and reliable model reasoning.

[0111] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0113] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: Obtaining a currently generated word-element of a text sequence, wherein the currently generated word-element is a most recently generated word-element in the text sequence; Acquire a previous state of a plurality of hidden states associated with the text sequence, wherein each hidden state of the plurality of hidden states is used to represent text information of the text sequence related to a different category, and the previous state of the plurality of hidden states is used to represent text information of a word-gram generated before the currently generated word-gram of the text sequence related to a different category, and the different categories correspond to different data categories under a scene category of the text sequence; Based on the current generated word-gram of the text sequence, for each hidden state of the plurality of hidden states, using the text information of the current generated word-gram related to the category corresponding to the hidden state, a previous state of the hidden state is encoded to generate a current state of the hidden state, wherein the current state of the hidden state is used to represent the text information related to the category of the word-gram of the text sequence including the current generated word-gram; A next generated word-gram of the text sequence is generated based on a current state of each hidden state in a plurality of hidden states.

2. The method of claim 1, based on the current generated word-gram of the text sequence, for each hidden state of the plurality of hidden states, using text information of the current generated word-gram related to the category corresponding to the hidden state, encoding a previous state of the hidden state to generate a current state of the hidden state comprises: encoding a previous state of the hidden state using text information of the currently generated word-element related to the category corresponding to the hidden state to generate a current state of the plurality of hidden states through a state space model; Generating a next generated word-gram of the text sequence based on a current state of each hidden state in the plurality of hidden states comprises: Generate a next generated word-gram of the text sequence based on a combination of a current state of each hidden state in a plurality of hidden states by means of the state space model; The state space model takes the current generated word element of the text sequence as input and takes the next generated word element of the text sequence as output.

3. The method of claim 2, further comprising: A pre-trained large language model is used as a teacher model, and the state-space model is used as a student model, and the state-space model is trained through model distillation and multi-task learning.

4. The method of claim 3, wherein: Training the state-space model through model distillation and multi-task learning includes: The state-space model is trained by minimizing the difference between the predicted probability distributions of the pre-trained large language model and the state-space model for generating the next generated word-gram, and maximizing the probability of the state-space model generating the next generated word-gram in an autoregressive manner.

5. The method of claim 4, wherein: Generating a next generated word-gram of the text sequence based on a current state of each hidden state in the plurality of hidden states comprises: Determining, by the state space model, a predicted probability distribution for generating a next generated word-gram based on a combination of current states of each of the plurality of hidden states; A next generated word-gram of the text sequence is generated based on the determined prediction probability distribution for generating the next generated word-gram.

6. The method of claim 1, further comprising: Determining a scene category of the text sequence based on an input text corresponding to the text sequence; Determining the plurality of hidden states associated with the text sequence based on the scene category of the text sequence; Based on the input text, initial states of the plurality of hidden states are determined.

7. The method of claim 1, wherein: The scene category of the text sequence is a medical scene; The multiple hidden states associated with the text sequence include at least one of the following: a hidden state of a prompt portion of an input text corresponding to the text sequence; the hidden status of the patient’s admission record; The hidden state of patient management.

8. A data processing device, comprising: A data acquisition module, configured to acquire a current generated word-gram of a text sequence, and acquire a previous state of a plurality of hidden states associated with the text sequence, wherein the current generated word-gram is a newly generated word-gram in the text sequence, each hidden state in the plurality of hidden states is used to represent text information related to different categories of the text sequence, and the previous state of the plurality of hidden states is used to represent text information related to different categories of a word-gram generated before the current generated word-gram of the text sequence, and the different categories correspond to different data categories under a scene category of the text sequence; a state updating module configured to, based on the current generated word-gram of the text sequence, encode, for each hidden state of the plurality of hidden states, a previous state of the hidden state using text information of the current generated word-gram related to the category corresponding to the hidden state, so as to generate a current state of the hidden state, wherein the current state of the hidden state is used to represent text information related to the category of the word-gram of the text sequence including the current generated word-gram; and The word-gram generation module is configured to generate a next generated word-gram of the text sequence based on a current state of each hidden state in a plurality of hidden states.

9. A data processing device comprising: one or more processors; as well as One or more memories, wherein a computer executable program is stored, and when the computer executable program is executed by the processor, the method according to any one of claims 1 to 7 is performed.

10. A computer program product stored on a computer-readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 7.

11. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images