Text data reasoning method and apparatus, storage medium and electronic device

Abstract

The present application discloses a text data reasoning method and apparatus, a storage medium and an electronic device. The present application can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation and assisted driving. The method comprises: acquiring initial token sequences respectively corresponding to N pieces of text data, wherein N is an integer greater than 1, and one token in each initial token sequence represents one character in the corresponding text data; concatenating the N initial token sequences into a concatenated token sequence; and performing inference processing on the concatenated token sequence to obtain N reply token sequences, wherein the N reply token sequences are used for generating respective reply data of the N pieces of text data, and a start token in each reply token sequence is obtained by performing, in the concatenated token sequence, reasoning processing on tokens belonging to the same initial token sequence. According to the present application, GPU resources consumed by reasoning a plurality of pieces of text data can be reduced, and the reasoning performance of a device on the plurality of pieces of text data is improved.

Description

Text data reasoning method, device, storage medium and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 28, 2023, application number 202311085639.X, and application name “A method, device, storage medium and electronic device for reasoning about text data”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a text data reasoning technology. Background Art

[0003] The rapid development of high-tech has led to the emergence of large language models (LLMs) with powerful reasoning capabilities. For example, computers, leveraging the reasoning capabilities of LLMs, can infer the next sentence of the sentence "Easy to know the east wind" by invoking the vast number of model parameters within the LLMs, resulting in the inference result "A myriad of colors always blooms in spring."

[0004] During the aforementioned inference process, the computer device actually needs to call upon hundreds of billions of model parameters. For example, during the inference process of the Generative Pre-Trained Transformer 3 (GPT3) model, the model parameter count involved reaches 175 billion (175B). This results in the computer device consuming a large amount of graphics processing unit (GPU) resources during the data inference process.

[0005] In related technologies, in order to avoid inferring only one text data at a time, which would lead to inefficient GPU resource utilization, a method is proposed to treat multiple text data as the same batch for data inference processing; further, in order to adapt the inference process to the existing deep learning model framework, a character padding method is also proposed, that is, padding characters in relatively short text data to ensure that the length of each text data processed in the same batch remains consistent.

[0006] However, the padding characters added by the character padding method will also participate in the actual reasoning operation, which will cause a large amount of invalid calculations and consumption of a large amount of invalid GPU resources.

[0007] Summary of the Invention

[0008] The present application provides a method, device, storage medium and electronic device for reasoning multiple text data, which are used to reduce the GPU resources consumed by reasoning multiple text data and improve the reasoning performance of computer equipment for multiple text data.

[0009] In a first aspect, the present application provides a method for inferring text data, the method being executed by a computer device, the method comprising:

[0010] Obtaining an initial token sequence corresponding to each of N text data; wherein N is an integer greater than 1, and a token in each initial token sequence represents a character in the corresponding text data;

[0011] splicing N initial token sequences into a spliced ​​token sequence, wherein the spliced ​​token sequence is a token sequence consisting of each token in the N initial token sequences;

[0012] The concatenated token sequence is subjected to inference processing to obtain N reply token sequences; wherein the N reply token sequences are used to generate reply data for each of the N text data, and the starting token in each reply token sequence is obtained by inference processing on each token belonging to the same initial token sequence in the concatenated token sequence.

[0013] In a second aspect, the present application provides a text data reasoning device, which is deployed on a computer device and includes:

[0014] An acquisition unit is configured to acquire an initial token sequence corresponding to each of N text data; wherein N is an integer greater than 1, and a token in each initial token sequence represents a character in the corresponding text data;

[0015] a splicing unit, splicing N initial token sequences into a spliced ​​token sequence, wherein the spliced ​​token sequence is a token sequence composed of the tokens in the N initial token sequences;

[0016] An inference unit performs inference processing on the spliced ​​token sequence to obtain N reply token sequences; wherein the N reply token sequences are used to generate reply data for each of the N text data, and the starting token in each reply token sequence is obtained by performing inference processing on each token belonging to the same initial token sequence in the spliced ​​token sequence.

[0017] In a third aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the inference method for text data described in any one of the first aspects above is implemented.

[0018] In a fourth aspect, the present application provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to perform any one of the text data reasoning methods in the above-mentioned first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any one of the text data reasoning methods in the first aspect.

[0020] The beneficial effects of this application are as follows:

[0021] In an embodiment of the present application, a method for reasoning about text data is proposed. The method is an optimized splicing method for multiple text data. Specifically, a computer device obtains N (N is an integer greater than 1) initial token sequences corresponding to each of the text data, wherein a token in each of the obtained initial token sequences represents a character in the corresponding text data, and then the N initial token sequences are spliced ​​into a spliced ​​token sequence. The spliced ​​token sequence is a token sequence composed of each token in the N initial token sequences, that is, the spliced ​​token sequence contains all the tokens in the N initial token sequences, thereby splicing the N initial token sequences into a token sequence, and realizing non-redundant splicing for the N initial token sequences. The spliced ​​token sequence is then subjected to inference processing to obtain N response token sequences, where the N response token sequences are used to generate response data for each of the N text data, and the starting token in each response token sequence is: obtained by inference processing for each token belonging to the same initial token sequence in the spliced ​​token sequence. Reasoning based on the concatenated token sequence can achieve multi-text data reasoning without padding characters. Compared with the character padding method provided by related technologies, since there is no need to pad characters, there is no need to consume extra GPU resources for padding characters, which reduces the GPU resources consumed by reasoning multiple text data, thereby reducing the reasoning cost for text data and improving the reasoning performance of computer equipment for multiple text data.

[0022] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] FIG1 is a schematic diagram of an optional application scenario in an embodiment of the present application;

[0025] FIG2 is a flow chart of a method for reasoning about text data provided in an embodiment of the present application;

[0026] 3A and 3B are schematic diagrams of possible conversation scenarios in an embodiment of the present application;

[0027] FIG4 is a schematic diagram of the reasoning process based on the deep learning model framework in an embodiment of the present application;

[0028] FIG5 is a schematic diagram of completing N initial token sequences in an embodiment of the present application;

[0029] 6A to 6D are schematic diagrams of a process for obtaining a splicing token sequence in an embodiment of the present application;

[0030] FIG7 is a schematic diagram of a process of inferring and splicing token sequences based on a deep learning model framework according to an embodiment of the present application;

[0031] 8A and 8B are schematic diagrams of possible attention matrices in embodiments of the present application;

[0032] FIG9 is a schematic diagram of an inference device for text data provided in an embodiment of the present application;

[0033] FIG10 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0035] In the embodiments of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] The embodiments of the present application relate to artificial intelligence technology, primarily to natural language processing technology within artificial intelligence technology.

[0037] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results.

[0038] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. Foundational AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0039] Natural language processing (NLP) is an important field in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between humans and computers using natural language. Natural language processing involves natural language, that is, the language people use in daily life, and is closely related to linguistics research; it also involves computer science and mathematics. The pre-trained model, an important technology for model training in the field of artificial intelligence, is developed from the large language model in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies. The pre-trained model is the latest development in deep learning and integrates the above technologies.

[0040] The embodiments of the present application also involve machine learning, through which a text reasoning model for reasoning can be trained.

[0041] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0042] In an embodiment of the present application, artificial intelligence technology is applied to the field of data reasoning to reduce the GPU resources consumed by reasoning multiple text data based on text reasoning models (such as pre-trained models), and to improve the reasoning performance of computer equipment for multiple text data.

[0043] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained below.

[0044] Pretrained Model: refers to a model trained on a large-scale corpus, typically using unsupervised learning methods such as autoencoders and language models. The basic idea of ​​a pretrained model is to use a large-scale corpus and, through unsupervised learning, allow the model to learn a large amount of general knowledge and patterns, thereby serving as a foundational model for various natural language processing tasks. For example, the embodiments of the present application add adaptive training for the recommendation reason generation task to the pretrained model, allowing the resulting model to be used for generating recommendation reasons.

[0045] As an example, the text reasoning model involved in the embodiments of the present application is a pre-trained model, which can be any deep learning model; for example: a generative pre-trained transformation model (Chat Generative Pre-trained Transformer, ChatGPT), a pre-trained language model (Bidirectional Encoder Representations from Transformers, BERT), etc.

[0046] Transformer: It is a common deep learning model architecture that has been widely used in various fields such as natural language processing, computer vision (CV), and speech processing. When Transformer was first proposed, it was a sequence-to-sequence model architecture for machine translation. It consists of an encoder and a decoder. The encoder and decoder are both composed of a series of Transformer blocks with the same structure. Each Transformer block consists of at least a multi-head self-attention layer and a feedforward neural network layer. Currently, Transformer has become a common architecture in natural language processing and is often used as a pre-training model. In addition to language-related applications, Transformer has also been applied to fields such as computer vision and audio processing.

[0047] Language Model: A model used to model natural language. Its purpose is to predict the next word or character in a given text sequence. Language models can be used for a variety of natural language processing tasks, such as text semantic extraction, text generation, machine translation, speech recognition, etc. Currently, Transformer-based pre-trained language models (PLM) are more common in various natural language processing tasks and usually achieve relatively good results. For example, the more common pre-trained language models include the Transformer model based on bidirectional encoding representation (Bidirectional Encoder Representation from Transformers, BERT) and the generative pre-trained Transformer model (Generative Pre-Trained Transformer, GPT).

[0048] Large Language Model (LLM): refers to a natural language processing model with large-scale parameters and training data. The training process of a large language model usually adopts an unsupervised learning approach, that is, the model is trained through a large text corpus to learn the probability distribution and language laws of the language. During the training process, a large language model usually uses a language model as the objective function, that is, the model parameters are optimized by maximizing the predicted probability of the next word. For example, the GPT series of models based on the Transformer model structure are trained on a large corpus and can generate high-quality natural language text, such as articles and conversations.

[0049] A tokenizer is a tool that converts natural language text into a sequence of characters, words, or subwords. In the Transformer model, a tokenizer typically refers to a tool that converts natural language text into the token (segmented word) sequence required for model input. It typically uses a character- or sub-token-based tokenization method, such as Byte Pair Encoding (BPE) or Sentence Piece. These methods can split words or subwords into smaller units, allowing the model to better handle uncommon words or words not in the vocabulary.

[0050] The attention mechanism uses high-level information to weigh intermediate network features, allowing the network to focus on information that aids judgment and ignore irrelevant information. The essence of the attention mechanism is derived from human visual attention. When people perceive something, they typically don't observe the entire scene from beginning to end. Instead, they focus on specific parts based on their needs. When they notice that a certain part of a scene frequently contains something they're interested in, they learn to focus on that part in similar scenes in the future. Therefore, the attention mechanism essentially filters high-value information from a large amount of information. Different pieces of information have varying importance to the result, and this importance can be reflected by assigning different weights. In other words, the attention mechanism can be understood as a rule for assigning weights when synthesizing multiple sources. It is often used to address the difficulty of obtaining a reasonable final vector representation when a model has long input sequences. This approach involves retaining the model's intermediate results, learning from them with a new model, and associating them with the output to achieve information filtering. Attention mechanisms include attention, self-attention, single-head attention, and multi-head attention.

[0051] Token: The smallest semantic unit obtained after processing text data, also known as a segmentation, word unit, or token. A token is a basic element in text data and can be a word, a phrase, a punctuation mark, a subword, or a character. The specific token type depends on the requirements and context of the NLP task.

[0052] The following is a brief introduction to the design concept of the embodiments of this application.

[0053] At present, large language models for conversations usually use a deep learning model architecture such as Transformer. In addition, the number of model parameters of existing large language models has reached more than 7 billion. For example, the model parameters of GPT3 are as high as 17.5 billion, which leads to huge consumption of GPU resources in the model inference process.

[0054] To address the above issues and avoid inefficient GPU resource utilization caused by inferring only one text data at a time, we propose a method of treating multiple text data as a single batch for data inference. The relevant technical solutions can be summarized as follows:

[0055] Related solution 1: Before treating multiple text data as a batch for data inference, in order to ensure that the inference process can adapt to the existing deep learning framework, it is also necessary to fill in the relatively short text data with padding characters to keep the length of each text data involved in the same batch processing consistent.

[0056] However, the padding characters added by the above-mentioned character padding method will also participate in the actual reasoning operation, which will cause a large number of invalid calculations and a large amount of invalid GPU resource consumption; further, even if at least two text data with the same length are selected from the multiple text data as the processing objects of the same batch, the padding of the padding characters cannot be completely avoided, and only when there is a scenario with enough text data can two text data with the same length be found. In other words, this method is limited by the data volume and length distribution of the multiple text data, resulting in a large number of invalid calculations and a large amount of invalid GPU resource consumption.

[0057] Related Solution 2: Based on Related Solution 1, by improving the model framework of the large language model, and then based on the improved model framework, the redundant padding characters involved in Related Solution 1 are filtered out to solve the invalid calculation and invalid GPU resource consumption caused by the redundant padding characters.

[0058] However, the above-mentioned method of improving the model framework requires modifications at the model framework level and is only applicable to specific model frameworks. Secondly, it has limitations at the model application level. As far as current applications are concerned, it only supports BERT-type text understanding applications, but not large language model applications such as dialogue or generation.

[0059] In view of this, an embodiment of the present application provides an inference method for text data, which can greatly improve the inference performance of computer devices calling massive model parameters for multi-text data in inference scenarios suitable for various large language model applications; for example, for N text data with large length differences, this solution can more than double the inference performance compared to the character completion method provided by the related technology. It can not only adapt to the existing deep learning model framework, but also realize the independence of the model framework to adapt to the model framework proposed later, improve the utilization of GPU resources, and reduce the inference cost for text data.

[0060] Specifically, an embodiment of the present application provides an optimized splicing method for multiple text data sets, splicing the initial token sequences corresponding to each of the N acquired text data sets to obtain a spliced ​​token sequence that contains all the tokens in the N initial token sequences. This method, compared to the character padding method provided by related technologies, eliminates the need to consume extra GPU resources for padding characters, effectively saving GPU resources required for subsequent reasoning and thereby reducing the cost of reasoning on text data.

[0061] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios described below are only used to illustrate the embodiments of the present application and are not limiting. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0062] The solution provided by the embodiments of this application can be applied to text data reasoning scenarios, reducing the GPU resources consumed by reasoning multiple pieces of text data and improving the reasoning performance of computer devices for multiple pieces of text data. As shown in Figure 1, an application scenario provided by the embodiments of this application is shown, in which a terminal device 101 and a server 102 may be included.

[0063] The terminal device 101 can be, for example, a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, a smart TV, a smart car device, a smart wearable device, a smart TV, and an aircraft, and any other device involved in text data reasoning. The terminal device 101 can be installed with a target application, and the target application can have the functions of obtaining N text data to be reasoned input by the user object, displaying N text data, obtaining the initial token sequence corresponding to each of the N text data, splicing the N initial token sequences into splicing token sequences, obtaining splicing token sequences, obtaining and displaying the reply token sequence of each of the N text data, obtaining and displaying the reply data of each of the N text data, and the like. For example, it can be an instant messaging application, a music application, a game application, a video application, a short video application, a news application, and a shopping application. The application involved in the embodiment of the present application can be a software client, or it can be a client such as a web page or a small program. The server 102 is a server corresponding to the software or web page, small program, etc., and does not limit the specific type of the client.

[0064] It should be noted that the above-mentioned terminal device 101 is not required to perform the process of obtaining the initial token sequence corresponding to each of the N text data, splicing the N initial token sequences into a spliced ​​token sequence, and obtaining the spliced ​​token sequence. The terminal device 101 may also send the N text data to the server 102, and the server 102 may process and generate the received N text data.

[0065] The server 102 may be a backend server for the target application, configured to provide corresponding backend services, such as data inference services, etc. It may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms, but is not limited thereto.

[0066] It should be noted that the reasoning method for text data in the embodiment of the present application can be executed independently by the terminal device 101 or the server 102, or can be executed jointly by the server 102 and the terminal device 101. When executed independently by the terminal device 101 or the server 102, the process of applying the model to perform data reasoning can be implemented independently by the terminal device 101 or the server 102. For example, on the terminal device 101, the initial token sequences corresponding to the N (N is an integer greater than 1) text data to be processed can be spliced ​​to obtain a spliced ​​token sequence, and then based on the attention mechanism, the spliced ​​token sequence can be reasoned to obtain N reply token sequences, or the server 102 can also perform one or a combination of the above-mentioned word segmentation processing process, splicing processing process and reasoning processing process. When executed jointly by the server 102 and the terminal device 101, the server 102 can train the large language model and then deploy the pre-trained language model to the terminal device 101, and the terminal device 101 can implement the data inference process. Alternatively, part of the data inference process can be implemented by the server 102, and the other processes can be implemented by the terminal device 101. In actual application, specific configuration can be made according to the situation, and this application does not make specific limitations here.

[0067] The server 102 and the terminal device 101 may each include one or more processors, memories, and interactive I / O interfaces. Furthermore, the server 102 may also be configured with a database that can be used to store trained model parameters, trained text reasoning models, and the like. The memories of the server 102 and the terminal device 101 may also store program instructions required for execution in the recommendation reason generation method provided in the embodiments of the present application. These program instructions, when executed by the processor, can be used to implement the data reasoning process provided in the embodiments of the present application.

[0068] It should be noted that when the text data reasoning method provided in the embodiment of the present application is executed separately by the server 102 or the terminal device 101, the above-mentioned application scenario may also only include a single device of the server 102 or the terminal device 101, or the server 102 and the terminal device 101 may be considered to be the same device. Of course, in actual application, when the text data reasoning method provided in the embodiment of the present application is jointly executed by the server 102 and the terminal device 101, the server 102 and the terminal device 101 may also be the same device, that is, the server 102 and the terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0069] In the embodiment of the present application, the terminal device 101 and the server 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a Wireless Fidelity (WIFI) network. Of course, it can also be other possible networks, and the embodiment of the present application does not limit this. It should be noted that Figure 1 is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiment of the present application.

[0070] Below, in conjunction with the application scenarios described above, with reference to the accompanying drawings, the methods provided by the exemplary embodiments of this application are described. It should be noted that the above application scenarios are merely provided to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in this respect. It should also be noted that the following methods can be executed by the above-mentioned terminal device or server, or by both the terminal device and the server. The methods described here are specifically illustrated using the terminal device or server as an example.

[0071] Referring to FIG. 2 , which is a flowchart of an implementation method for reasoning about text data provided in an embodiment of the present application, taking a computer device representing a terminal device or a server as an example, the specific implementation process of the method is as follows:

[0072] Step 201: Obtain an initial token sequence corresponding to each of N text data; wherein N is an integer greater than 1, and a token in each initial token sequence represents a character in the corresponding text data.

[0073] In an embodiment of the present application, text data may be data in text form that requires a reply. One text data may be a conversation request triggered by a usage object, and N text data may be N conversation requests triggered by the same (or different) usage objects. In response to receiving the N conversation requests, the computer device uses the corresponding N text data as model inputs of a text inference model (e.g., a large language model), calls a large number of model parameters in the text inference model, and performs inference processing on the N text data.

[0074] Specifically, taking the dialogue application scenario as an example, the computer device will process N text data only after the object inputs N text data on the front-end interface of the computer device; or after the object inputs M (M is an integer greater than N) text data on the front-end interface of the computer device, the computer device extracts N text data for processing.

[0075] Refer to Figure 3A, which is a schematic diagram of a dialogue scenario, in which the user inputs "The next sentence of knowing the east wind is easy" in the front-end interface. In related technologies, after receiving this text data, the computer device can be regarded as receiving a dialogue request or a model input, and will call a large number of model parameters in the text reasoning model to infer "The next sentence of knowing the east wind is easy" and obtain the inference result of "A riot of colour always brings spring".

[0076] However, for the method shown in Figure 3A, each time the computer device receives a text data, it needs to call a large number of model parameters in the text inference model, which in turn consumes a large amount of GPU resources; to solve this problem, the embodiment of the present application will only process these text data after receiving two or more text data. The following is a dialogue scenario to which the technical solution provided by the embodiment of the present application is applicable.

[0077] As shown in Figure 3B, it is a schematic diagram of another dialogue scenario. Among them, the user inputs "What is the next line of 'easily recognize the east wind's face'" in the front-end interface. After the computer device receives this text data, it can be regarded as receiving a dialogue request or a model input. However, it will not immediately call the massive model parameters in the text inference model. Instead, after receiving N text data, it will call the massive model parameters in the text inference model to obtain the corresponding inference results. Here, taking the example of receiving 2 text data, that is, the user inputs again "Who is the author of this poem". Then, after the computer device receives two dialogue requests or two model inputs, the computer device will call the massive model parameters in the text inference model to obtain the inference result 2 "all the colors of spring are always there" corresponding to text data 1, and the inference result 2 "Zhu Xi" corresponding to text data 2. In this way, in the actual application process, facing massive text data, the method of processing multiple text data in this solution can improve the utilization rate of the GPU compared with the method of processing each single text data one by one.

[0078] The initial token sequence can be a sequence obtained by encoding each character of the text data and arranging the tokens corresponding to the characters in the order of the characters. In practical applications, the way to obtain the initial token sequences corresponding to N text data respectively can be to obtain the N text data to be replied, and then perform the following operations for each of the obtained N text data: for each text data, encode each character in the text data into the corresponding token to obtain the initial token sequence corresponding to one text data.

[0079] Specifically, taking one text data as an example, a text data can be segmented into multiple characters through a tokenizer to obtain the corresponding character sequence, and then each character is encoded. An encoded character can be regarded as a token, and thus the corresponding initial token sequence is obtained.

[0080] Among them, a token is the smallest semantic unit. In the embodiments of this application, mainly characters are used as examples for illustration. For example, the characters in "easily recognize the east wind's face" can be divided into "děng - xián - shí - dé - dōng - fēng - miàn - de - xià - yī - jù".

[0081] The embodiment of the present application tokenizes the text data, thereby capturing the vocabulary, grammar and semantic information in the text data, and thus better using it for subsequent reasoning tasks. As a more specific example, see Figure 4, which is a schematic diagram of a reasoning process based on a deep learning model framework, wherein X1, X2, X3, X4, and X5 represent the input of the model. Taking the large language model of the dialogue type (such as ChatGPT) as an example, X1, X2, X3, X4, and X5 can represent: each token in an initial token sequence obtained after encoding each character in a text data input by the object; h1, h2, h3, h4, and O0 represent: the output corresponding to each token in an initial token sequence after calculation by the large language model (that is, the reasoning token), wherein h1, h2, h3, and h4 will be omitted; then O0 represents the first reasoning token in the reasoning result generated by the model; O0 will be input into the model to continue calculating and generating the next reasoning token. Every time an inference token is generated, all model parameters in the large language model will be involved in the calculation. Here, we use a large language model with 10 billion parameters as an example. Assuming the model is saved in half-precision (FP16), the model size is 20GB. Each time an inference token is generated, the 20GB language model must be loaded from the graphics memory and used for calculation. Therefore, it is easy to understand that the graphics memory bandwidth becomes the main bottleneck in the text data inference process based on large language models.

[0082] In related technologies, in order to improve GPU utilization efficiency, multiple text data are generally spliced ​​together. In this way, by loading the model once, the inference results of multiple text data can be generated, thereby alleviating the memory bandwidth bottleneck problem in the model inference process.

[0083] However, in actual applications, the lengths of the initial token sequences corresponding to the N text data obtained are different, that is, the number of tokens contained in the N initial token sequences is different. In order to match the use of various GPU computing acceleration libraries (such as CuBLAS), related technologies need to use padding characters before calling the model to pad the N initial token sequences to the same length.

[0084] See Figure 5, which is a schematic diagram of filling in N initial token sequences of the related technology provided in an embodiment of the present application. Among them, a row represents: an initial token sequence; a non-blank square represents: a token; a blank square represents: a token corresponding to a filler character used to fill in the word segmentation sequence; the serial numbers 0 to 5 respectively represent: the position information of a single token in the corresponding initial token sequence. As shown in Figure 5, it can be seen that half of the tokens are used for filling. Since these filled tokens will also participate in the subsequent reasoning processing, it also leads to a lot of redundant calculations and bandwidth overhead.

[0085] In summary, after obtaining N initial token sequences, the embodiment of the present application also needs to execute a non-redundant splicing method provided in the following step 202 to improve the reasoning performance.

[0086] Step 202: concatenate the N initial token sequences into a concatenated token sequence, where the concatenated token sequence is a token sequence composed of the tokens in the N initial token sequences.

[0087] In the embodiment of the present application, since there is no need to pad the N initial token sequences, de-dependence on the length distribution of text data can be achieved. Compared with the character padding method provided by the related technology, it effectively saves the GPU resources required for subsequent reasoning. Moreover, the spliced ​​token sequence obtained in this way is compatible with the existing deep learning framework, thereby reducing the reasoning cost for multi-text data.

[0088] In one embodiment, N initial token sequences may be spliced ​​into a spliced ​​token sequence by splicing together a first token sequence based on the termination tokens of each of the N initial token sequences, and splicing together a second token sequence based on the tokens of each of the N initial token sequences except the termination token; then, the first token sequence is spliced ​​at the end of the second token sequence to obtain a spliced ​​token sequence.

[0089] Specifically, for N initial token sequences, each initial token sequence is divided into two parts: the termination token located at the termination position, and other tokens located at non-termination positions; then the first token sequence is spliced ​​from the N termination tokens, and the second token sequence is spliced ​​from several other tokens.

[0090] See Figure 6A, which is a schematic diagram of the division of N initial token sequences in an embodiment of the present application, wherein four initial token sequences are involved: initial token sequence 1 contains one token, initial token sequence 2 contains four tokens, initial token sequence 3 contains one token, and initial token sequence 4 contains six tokens. Then, the token with position information representation of 0 in initial token sequence 1 is determined to be termination token 1, the token with position information representation of 3 in initial token sequence 2 is determined to be termination token 2, the token with position information representation of 0 in initial token sequence 3 is determined to be termination token 3, and the token represented as 5 in initial token sequence 4 is extracted as termination token 4.

[0091] The termination token usually contains some important local information, and other tokens outside the termination token can reflect contextual information. Through this specific splicing processing, the embodiment of the present application can provide rich contextual information and local information for subsequent reasoning while maintaining a certain flexibility, thereby achieving better performance in subsequent reasoning tasks.

[0092] Optionally, the first token sequence and the second token sequence are respectively generated by splicing according to a preset splicing order, and the splicing order represents the processing order of the N text data.

[0093] Specifically, based on a preset splicing order, the determined N termination tokens are spliced ​​into a first token sequence. And based on a preset splicing order, the tokens other than the termination token in each of the N initial token sequences are spliced ​​into a second token sequence.

[0094] Refer to Figure 6B, which is a schematic diagram of obtaining the first token sequence in an embodiment of the present application. Taking the splicing order of initial token sequence 1 → initial token sequence 2 → initial token sequence 3 → initial token sequence 4 as an example, the termination token with position information represented as 0 in the initial token sequence 1, the termination token with position information represented as 3 in the initial token sequence 2, the termination token with position information represented as 0 in the initial token sequence 3, and the termination token with position information represented as 5 in the initial token sequence 4 are spliced ​​in sequence to obtain the first token sequence.

[0095] Correspondingly, based on the processing order of the N text data, the tokens of each of the N initial token sequences except the termination token are sequentially concatenated to obtain a second token sequence.

[0096] Refer to Figure 6C, which is a schematic diagram of obtaining the second token sequence in an embodiment of the present application. Taking the splicing order of initial token sequence 1 → initial token sequence 2 → initial token sequence 3 → initial token sequence 4 as an example, the three tokens whose position information is represented as 0, 1, and 2 in the initial token sequence 2 and the four tokens whose position information is represented as 0, 1, 2, 3, and 4 in the initial token sequence 4 are spliced ​​in sequence to obtain the second token sequence.

[0097] By splicing the N text data in the order in which they are processed, when there is a certain relationship or dependency between the N text data (such as a conversation, a story, etc.), these relationships can be better captured so as to generate reply data more accurately. For example, in a dialogue system, splicing the dialogue history in chronological order can help understand the context and intention of the dialogue. It should be noted that the processing order of the above-mentioned N text data can be determined in any of the following ways, and this application does not impose any specific restrictions on this. For example, the order in which the N text data are acquired is used as the processing order of the N text data; for example, the time order of the timestamps corresponding to the N text data is used as the processing order of the N text data; for another example, the order of the priority levels corresponding to the N text data is used as the processing order of the N text data.

[0098] After obtaining the first token sequence and the second token sequence, refer to Figure 6D, which is a schematic diagram of obtaining the spliced ​​token sequence in an embodiment of the present application. The first token sequence is spliced ​​to the end position (i.e., the tail) of the second token sequence to obtain the spliced ​​token sequence.

[0099] Optionally, as a possible implementation method, in the spliced ​​token sequence, each token may also be associated with the following information: position information of a token in the corresponding initial token sequence, association information between a token and the text data corresponding to the initial token sequence to which it belongs, etc.

[0100] For example, in the target sequence shown in FIG6D , the token corresponding to a block is obtained by encoding the corresponding character, and the block itself represents the encoding information of the corresponding character (i.e., the token), the block pattern is used to represent the association information between the corresponding token and the text data corresponding to the initial token sequence to which it belongs, and the numerical value marked below the block is used to represent the position information of the corresponding token in the initial token sequence to which it belongs.

[0101] In summary, the embodiment of the present application provides a non-redundant splicing method for N initial token sequences, so that the spliced ​​token sequence obtained can avoid the dependence of the related technology on the length distribution of the N initial token sequences; further, in order to ensure the correctness of the reasoning result corresponding to the spliced ​​token sequence, it is necessary to execute the following step 203 to perform reasoning processing on the spliced ​​token sequence.

[0102] Step 203: Perform inference processing on the concatenated token sequence to obtain N reply token sequences; wherein the N reply token sequences are used to generate reply data for each of the N text data, and the starting token in each reply token sequence is obtained by performing inference processing on each token belonging to the same initial token sequence in the concatenated token sequence.

[0103] After obtaining the concatenated token sequence, we can directly perform inference processing on the concatenated token sequence to obtain N reply token sequences, thereby generating reply data for each of the N text data. The reply token sequences are obtained by inferring each token in the concatenated token sequence that belongs to the same initial token sequence. Each reply token sequence is used to generate corresponding reply data for replying to the corresponding text data.

[0104] In an embodiment of the present application, an attention mechanism can be introduced into the inference process. In this case, the concatenated token sequence can be inferred based on the attention mechanism to obtain N reply token sequences. The attention mechanism isolates the tokens in the concatenated token sequence that belong to different initial token sequences, thereby ensuring the accuracy of the inference of the obtained N reply token sequences and, in turn, the accuracy of the reply data for each of the N text data.

[0105] This paper proposes an optimized inference method for multiple text data. It introduces an attention mechanism and performs inference processing on the concatenated token sequence to obtain N reply token sequences. These N reply token sequences are used to generate reply data for each of the N text data. In addition, the starting token in each reply token sequence is obtained by inferring the tokens belonging to the same initial token sequence in the concatenated token sequence. In other words, the attention mechanism introduces the inference process with the goal of obtaining reply data for each of the N text data. To obtain accurate N reply data, it is necessary to determine the starting token of each reply token sequence, based on which N reply token sequences are obtained. In this way, by introducing the attention mechanism into the data inference process, the tokens belonging to different initial token sequences in the concatenated token sequence are isolated, ensuring the inference accuracy of the obtained N reply token sequences, and thus the accuracy of the reply data for each of the N text data.

[0106] It should also be noted that the text data reasoning method provided in the embodiment of the present application can greatly improve the reasoning performance of computer equipment in calling massive model parameters for specific sequences in the reasoning scenario of applying a large language model; for example, for N text data with large length differences, this solution can improve the reasoning performance by more than double compared with the character completion method provided by the related technology, and ensure the correctness of the reasoning results based on the attention mechanism. It can not only adapt to the existing deep learning model framework, but also realize the independence of the model framework to adapt to the model framework proposed later, improve the utilization of GPU resources, and reduce the reasoning cost for text data. Specifically, based on the attention mechanism, the spliced ​​token sequence is inferred and processed, and a method for obtaining N reply token sequences can be that the spliced ​​token sequence is input into a text inference model, and according to the attention mechanism, each token in the spliced ​​token sequence is inferred by the text inference model, and a candidate token sequence is output, where each token in the candidate token sequence is: the inference result of a token in the spliced ​​token sequence; then, from the candidate token sequence, the inference results corresponding to the terminal tokens of the N initial token sequences are determined to obtain N selected tokens; then, based on the text inference model, the N selected tokens are iteratively inferred to obtain N reply token sequences output by the text inference model, where the N selected tokens are the starting tokens of the N reply token sequences.

[0107] In other words, the concatenated token sequence is input into the text inference model to obtain a candidate token sequence output by the text inference model according to the attention mechanism, and each token in the candidate token sequence is the output result of a token in the concatenated token sequence; then, N tokens are selected from the tokens in the candidate token sequence as N selected tokens, and these N selected tokens correspond to tokens in the concatenated token sequence and are the last token in each of the N initial token sequences. The termination token (last token) in each initial token sequence is used to represent the last character in a text data; then, the N selected tokens are respectively used as the first token in the N reply token sequences and input into the text inference model to obtain other tokens in the N reply token sequences, where the N reply token sequences are used to generate reply data for each of the N text data.

[0108] To facilitate understanding, the following first describes in detail the reasoning process of the spliced ​​token sequence in the embodiment of the present application in combination with a universal deep learning model framework.

[0109] Referring to Figure 7 , which is a schematic diagram of the process of inferring a spliced ​​token sequence based on a deep learning model framework according to an embodiment of the present application, a spliced ​​token sequence obtained by splicing two initial token sequences I00:n-1, I10:n-1 is taken as an example. For ease of understanding, the spliced ​​token sequence is shown as two corresponding initial token sequences in Figure 7 . Those skilled in the art should know that when a computer device performs inference processing on each token in the spliced ​​token sequence, it is actually performing inference processing on the tokens in the two initial token sequences separately.

[0110] As shown in Figure 7, it can be seen that the two initial token sequences I00:n-1, I10:n-1 (i.e., the concatenated token sequences) are input into the text reasoning model, and O00 and O10 are the reasoning tokens corresponding to the termination tokens of I00:n-1 and I10:n-1 after the model has reasoned about each token in I00:n-1 and I10:n-1 (i.e., each token in the concatenated token sequences). The first O of O00 and O10 represents the model output, the second 0 of O00 represents the corresponding first initial token sequence, the second 1 of O10 represents the corresponding second initial token sequence, and O00 represents the corresponding second initial token sequence. The third 0 in 00 and O10 respectively represents the inference token output after inference on the termination token in the corresponding token sequence. Then O00 and O10 also represent the first token of the reply token sequence of the text data corresponding to the two initial token sequences. Then O00 and O10 are input into the text inference model to generate the inference tokens corresponding to the two inputs respectively, and the cycle is repeated until the reply token sequences corresponding to the two text data are generated. One reply token sequence can be a concatenation sequence of O00, O01, ..., O0n; another reply token sequence can be a concatenation sequence of O10, O11, ..., O1n.

[0111] That is to say, in the reasoning process of the spliced ​​token sequence, the embodiment of the present application is mainly based on the attention mechanism, so that the text reasoning model ensures that each output reasoning token is only associated with its own corresponding input token and its own corresponding generated reasoning token during the data reasoning process, thereby realizing the reasoning isolation of different text data in the reasoning process of the spliced ​​token sequence.

[0112] The application form of the attention mechanism in the reasoning process can be an attention matrix, of course, it can also be other forms, which are not specifically limited here. This paper takes the preset attention matrix as an example and will not be repeated below.

[0113] The preset attention matrix represents the degree of attention paid to each token in the concatenated token sequence and its corresponding inference token during the inference process, based on the relationship between each token and its initial token sequence. The attention matrix can contain several rows of elements, each row representing the different degree of attention paid to each token in the concatenated token sequence during the inference process, or each row representing the different degree of attention paid to each element in the concatenated token sequence and its corresponding inference token during the inference process.

[0114] As an example, the attention matrix contains: Q+P×N rows of elements. Where Q is the total number of tokens in the concatenated token sequence, each row of Q elements represents: the different levels of attention paid to each token in the concatenated token sequence during the inference process, P is a positive integer, each row of P×N elements represents: the different levels of attention paid to each token in the concatenated token sequence and its corresponding inference token during the inference process, and each N rows of elements correspond to N text data.

[0115] To facilitate understanding, the following briefly explains the structure of the attention matrix proposed in the embodiment of the present application, taking the attention matrix corresponding to a single initial token sequence as an example.

[0116] Refer to Figure 8A, which is the attention matrix corresponding to a single initial token sequence, where the column number and row number in the attention matrix can represent the serial number of the token. Looking from top to bottom, only the first one in the first row is filled with slashes, which means that the first token in the single initial token sequence can only be processed with itself based on the attention mechanism for reasoning; only the first and second ones in the second row are filled with slashes, which means that the second token in the single initial token sequence can be processed with itself and the first token based on the attention mechanism for reasoning, and so on; in other words, the attention matrix shown in Figure 8A indicates that each token in a single initial token sequence can only be processed with itself (it should be noted that the token itself can also be an inference token) and the tokens before it based on the attention mechanism for reasoning.

[0117] After introducing the attention matrix corresponding to a single initial token sequence, the attention matrix corresponding to the spliced ​​token sequence provided in the embodiment of the present application is described in detail below.

[0118] For the Q+P×N rows of elements in the attention matrix above, the order of the Q row elements is determined based on the order of the tokens contained in each of the N initial token sequences, excluding the termination token. The Q row elements can be determined as follows: A starting token (non-termination token) in any initial token sequence is selected as the attention token for the inference process, and its position in the concatenated token sequence is determined to construct the first row element. Similar operations are then performed on each subsequent token (non-termination token) in the same initial token to generate the corresponding row elements. This process is repeated to generate Q row elements.

[0119] In one possible implementation, based on the attention mechanism, each token in the spliced ​​token sequence is inferred through a text inference model, and the method of outputting a candidate token sequence can be: obtain Q row elements in a preset attention matrix, and then, based on each row element in the Q row elements, perform inference processing on each token in the spliced ​​token sequence that belongs to the same initial token sequence, obtain the inference token corresponding to each token in the spliced ​​token sequence, and then obtain the candidate token sequence generated by splicing each inference token.

[0120] For the Q+P×N row elements in the above attention matrix, the arrangement order of the P×N row elements is: determined based on the processing order of the N text data, that is, consistent with the generation order of the N reply token sequences.

[0121] In one possible implementation, based on a text inference model, iterative inference is performed on N selected tokens to obtain N reply token sequences output by the text inference model. The method can be: use the N selected tokens as the starting tokens of the N reply token sequences respectively; obtain P×N row elements in a preset attention matrix, and then perform P operations on each N row elements in the P×N row elements in sequence to obtain N reply token sequences. The specific execution of the operation is as follows:

[0122] Based on N rows of elements, the N selected tokens currently obtained are respectively inferred to obtain N inference tokens. Here, each row of elements in the N rows represents: during the inference process, different levels of attention are paid to the N selected tokens currently obtained. Then, the N inference tokens are respectively spliced ​​at the end of the N reply token sequence, and the N inference tokens are used as the N selected tokens obtained next time.

[0123] The following is a relatively complete example to illustrate the reasoning process of the above text reasoning model based on the attention matrix, performing reasoning on the spliced ​​token sequence to obtain N reply token sequences.

[0124] As shown in Figure 8B, this is an attention matrix preset for the spliced ​​token sequence, wherein the spliced ​​token sequence is obtained by splicing together a first initial token sequence containing 5 tokens and a second initial token sequence containing 4 tokens. The splicing method can be found in the relevant description of step 202 and will not be elaborated here.

[0125] As shown in Figure 8B, input1 includes four rows of elements, which correspond to the first four tokens in the first initial token sequence that constitutes the spliced ​​token sequence, and input2 contains three rows of elements, which represent the first three tokens in the second initial token sequence that constitutes the spliced ​​token sequence; then input1+input2 includes 7 (i.e., Q) rows of elements, and based on these seven rows of elements, the candidate token sequence corresponding to the spliced ​​token sequence can be inferred; the 8th row represents the termination token in the first initial token sequence that constitutes the spliced ​​token sequence, that is, the termination token in the first initial token sequence is only processed with the first four tokens in the first initial token sequence based on the attention mechanism; the 9th row represents the termination token in the second initial token sequence that constitutes the spliced ​​token sequence, that is, the termination token in the second initial token sequence is only processed with the first three tokens in the second initial token sequence based on the attention mechanism; then the elements in the 8th and 9th rows constitute N rows of elements, and later, according to the output length of the text reasoning model, the attention matrix can be expanded to P×N rows of elements.

[0126] It should be noted that the self-expansion of the above-mentioned attention matrix is ​​related to the generation order of the inference results of the N text data. Therefore, the self-expansion of the attention matrix should generally follow the preset splicing order.

[0127] The following is an overall explanation of the multi-text data reasoning method provided in the embodiment of the present application in combination with actual application scenarios, taking the dialogue scenario using a large language model as an example.

[0128] A user inputs various questions or conversation data to be inferred in the display interface as text data to be processed. After receiving a piece of text data to be processed from the user, the computer device does not directly activate the large language model. Instead, it prepares to activate the large language model to infer N text data after receiving N (where N is an integer greater than 1).

[0129] Subsequently, for the N text data to be processed, the computer device encodes each character in each text data to obtain a corresponding initial token sequence, and then the N initial token sequences are divided into termination tokens and non-termination tokens. Based on a preset splicing order, a first token sequence formed by splicing the N termination tokens is spliced ​​at the end of a second token sequence formed by splicing several non-termination tokens to obtain a spliced ​​token sequence, wherein the spliced ​​token sequence contains all the tokens in the N initial token sequences.

[0130] Then, the concatenated token sequence is input into the large language model, and the large language model performs reasoning on each token in the concatenated token according to the attention mechanism, and outputs a candidate token sequence. Each token in the candidate token sequence is: the reasoning result of a token in the concatenated token sequence, and then from the candidate token sequence, the reasoning results corresponding to the terminal tokens of the N initial token sequences are determined to obtain N selected tokens; subsequently, based on the text reasoning model, iterative reasoning is performed on the N selected tokens to obtain N reply token sequences output by the text reasoning model, where the N selected tokens are the starting tokens of the N reply token sequences.

[0131] In summary, the embodiment of the present application provides an inference method for multi-text data, which can greatly improve the inference performance of large language models, especially for large language models of the Transformer type. In scenarios where the lengths of multi-text data vary greatly, this solution can improve the inference performance by more than 1 times compared to the existing method of completing text data, and thus it is hoped that the problem of the current high cost of inference for large language models will be alleviated to a certain extent. In addition, compared with the optimization solution of the deep learning model framework provided by the relevant technology, this solution not only has no dependence on the distribution of multi-text data, improves the inference performance, but is also compatible with existing deep learning model frameworks (such as Huggingface, Pytorch, Tensorflow, etc.), and improves the flexibility of inference for multi-text data.

[0132] Referring to FIG. 9 , based on the same inventive concept, an embodiment of the present application further provides a text data reasoning device, the device comprising:

[0133] An acquiring unit 901 acquires an initial token sequence corresponding to each of N text data; wherein N is an integer greater than 1, and a token in each initial token sequence represents a character in the corresponding text data;

[0134] A splicing unit 902 splices N initial token sequences into a spliced ​​token sequence, where the spliced ​​token sequence is a token sequence composed of the tokens in the N initial token sequences;

[0135] The inference unit 903 performs inference processing on the spliced ​​token sequence to obtain N reply token sequences; wherein the N reply token sequences are used to generate reply data for each of the N text data, and the starting token in each reply token sequence is obtained by performing inference processing on each token belonging to the same initial token sequence in the spliced ​​token sequence.

[0136] Optionally, the inference unit 903 is specifically configured to:

[0137] Based on the attention mechanism, the concatenated token sequence is subjected to inference processing to obtain N reply token sequences.

[0138] Optionally, the inference unit 903 is specifically configured to:

[0139] Inputting the concatenated token sequence into a text inference model, and performing inference on each token in the concatenated token sequence through the text inference model according to an attention mechanism, and outputting a candidate token sequence, wherein each token in the candidate token sequence is an inference result of a token in the concatenated token sequence;

[0140] Determine, from the candidate token sequences, the inference results corresponding to the respective termination tokens of the N initial token sequences to obtain N selected tokens;

[0141] Based on the text reasoning model, iterative reasoning is performed on the N selected tokens to obtain N reply token sequences output by the text reasoning model; wherein the N selected tokens are respectively the starting tokens of the N reply token sequences.

[0142] Optionally, the reasoning unit 903 is configured to:

[0143] Obtain Q rows of elements from a preset attention matrix, wherein Q is the total number of tokens in the concatenated token sequence, and each row of elements in the Q rows represents: during the inference process, a different degree of attention is paid to each token in the concatenated token sequence;

[0144] Based on each row element in the Q row elements, inference processing is performed on each token in the spliced ​​token sequence that belongs to the same initial token sequence to obtain an inference token corresponding to each token in the spliced ​​token sequence;

[0145] Obtain a candidate token sequence generated by concatenating the inference tokens.

[0146] Optionally, the reasoning unit 903 is configured to:

[0147] Using the N selected tokens as the starting tokens of N reply token sequences respectively;

[0148] Get the P×N row elements in the preset attention matrix; where P is a positive integer;

[0149] For each N rows of elements in the P×N rows of elements, perform the following operations in sequence:

[0150] Based on N rows of elements, inference processing is performed on the N selected tokens currently obtained to obtain N inference tokens; wherein each row of elements in the N rows of elements represents: during the inference process, different levels of attention are paid to the N selected tokens currently obtained;

[0151] splicing the N inference tokens at the end of the N reply token sequences respectively, and using the N inference tokens as the N selected tokens obtained next time;

[0152] Until P operations are performed, N reply token sequences are obtained.

[0153] Optionally, the acquiring unit 901 is specifically configured to:

[0154] Get N text data to be replied;

[0155] For each text data in the N text data, each character in the text data is encoded into a corresponding token to obtain an initial token sequence corresponding to the text data.

[0156] Optionally, the splicing unit 902 is specifically configured to:

[0157] Based on the respective termination tokens of the N initial token sequences, splicing together to generate a first token sequence;

[0158] and, based on the tokens of each of the N initial token sequences except the termination token, splicing to generate a second token sequence;

[0159] The first token sequence is spliced ​​onto the end of the second token sequence to obtain a spliced ​​token sequence.

[0160] Optionally, the first token sequence and the second token sequence are respectively generated by splicing according to a preset splicing order, and the splicing order represents the processing order of the N text data;

[0161] The splicing unit 902 is configured to splice and generate a first token sequence based on the respective termination tokens of the N initial token sequences, specifically to:

[0162] Based on the processing order of the N text data, sequentially concatenate the respective termination tokens of the N initial token sequences to obtain a first token sequence;

[0163] The splicing unit 902 is configured to splice the tokens other than the termination token in each of the N initial token sequences to generate a second token sequence, specifically:

[0164] Based on the processing order of the N text data, the tokens of the N initial token sequences except the termination token are sequentially concatenated to obtain a second token sequence.

[0165] Optionally, the processing order of the N text data is determined by any of the following methods, and the splicing unit 902 is further used to:

[0166] Using the acquisition order of the N text data as the processing order of the N text data;

[0167] The time sequence of the timestamps corresponding to the N text data is used as the processing order of the N text data;

[0168] The priority levels corresponding to the N text data are ranked in descending order as the processing order of the N text data.

[0169] Based on the above device, the initial token sequences corresponding to the N (N is an integer greater than 1) text data to be processed are spliced ​​to obtain a spliced ​​token sequence, and then the spliced ​​token sequence is inferred based on the attention mechanism to obtain N reply token results for generating corresponding reply data, so as to reduce the GPU resources consumed by inferring multiple text data and improve the inference performance of computer equipment for multiple text data.

[0170] The device can be used to execute the methods shown in the various embodiments of the present application. Therefore, for the functions that can be implemented by the various functional modules of the device, please refer to the description of the aforementioned embodiments and no further details will be given.

[0171] Please refer to Figure 10. Based on the same technical concept, an embodiment of the present application also provides a computer device 1000. The computer device 1000 can be the terminal device or server shown in Figure 1. The computer device 1000 can include a memory 1001 and a processor 1002.

[0172] Memory 1001 is used to store computer programs executed by processor 1002. Memory 1001 may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the like; the data storage area may store data generated based on the use of the computer device. Processor 1002 may be a central processing unit (CPU), a digital processing unit, or the like. The specific connection medium between memory 1001 and processor 1002 is not limited in the embodiments of the present application. In FIG10 , the embodiment of the present application illustrates that memory 1001 and processor 1002 are connected via bus 1003, which is represented by a bold line in FIG10 . The connection methods between other components are for illustrative purposes only and are not intended to be limiting. Bus 1003 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG10 uses only one bold line, but this does not imply that there is only one bus or only one type of bus.

[0173] Memory 1001 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1001 may be a combination of the aforementioned memories.

[0174] The processor 1002 is configured to execute the method executed by the device in each embodiment of the present application when calling the computer program stored in the memory 1001.

[0175] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a computer program product, which includes a computer program. When the computer program is run on a computer device, the computer program is used to enable the computer device to execute the steps of the method according to the various exemplary embodiments of the present application described above in this specification. For example, the computer device can execute the method executed by the device in each embodiment of the present application.

[0176] The computer program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0177] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0178] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for reasoning about text data, the method being executed by a computer device, the method comprising: Obtaining initial token sequences corresponding to N text data, respectively; wherein N is an integer greater than 1, and a token in each initial token sequence represents a character in the corresponding text data; splicing N initial token sequences into a spliced ​​token sequence, wherein the spliced ​​token sequence is a token sequence composed of each token in the N initial token sequences; The concatenated token sequence is subjected to inference processing to obtain N reply token sequences; wherein the N reply token sequences are used to generate reply data for each of the N text data, and the starting token in each reply token sequence is obtained by inference processing on each token belonging to the same initial token sequence in the concatenated token sequence.

2. The method according to claim 1, wherein the inference processing of the concatenated token sequence to obtain N reply token sequences comprises: Based on the attention mechanism, the concatenated token sequence is inferred and processed to obtain N reply token sequences.

3. The method according to claim 2, wherein the concatenated token sequence is subjected to inference processing based on an attention mechanism to obtain N reply token sequences, comprising: Input the concatenated token sequence into a text inference model, and according to the attention mechanism, infer each token in the concatenated token sequence through the text inference model, and output a candidate token sequence, wherein each token in the candidate token sequence is: an inference result of a token in the concatenated token sequence; Determine the inference results corresponding to the respective termination tokens of the N initial token sequences from the candidate token sequences to obtain N selected tokens; Based on the text reasoning model, iterative reasoning is performed on the N selected tokens to obtain N reply token sequences output by the text reasoning model; wherein the N selected tokens are respectively the starting tokens of the N reply token sequences.

4. The method according to claim 3, wherein the text inference model is used to infer each token in the concatenated token sequence according to the attention mechanism to output a candidate token sequence, comprising: Obtain Q row elements in a preset attention matrix; wherein Q is the total number of tokens in the concatenated token sequence, and each row element in the Q row elements represents: during the reasoning process, different levels of attention to each token in the concatenated token sequence; Based on each row element in the Q row elements, inference processing is performed on each token in the spliced ​​token sequence that belongs to the same initial token sequence to obtain an inference token corresponding to each token in the spliced ​​token sequence; Obtain a candidate token sequence generated by concatenating each inference token.

5. The method according to claim 3 or 4, comprising: Using the N selected tokens as the start tokens of N reply token sequences respectively; Get the P×N row elements in the preset attention matrix; where, P is a positive integer; For each N rows of elements in the P×N rows of elements, the following operations are performed in sequence: Based on N rows of elements, the N selected tokens currently obtained are respectively subjected to inference processing to obtain N inference tokens; wherein each row of elements in the N rows of elements represents: in the inference process, different attention levels are respectively paid to the N selected tokens currently obtained; splicing the N inference tokens to the tails of the N reply token sequences respectively, and using the N inference tokens as the N selected tokens obtained next time; Until P operations are performed, N reply token sequences are obtained.

6. The method according to any one of claims 1 to 5, wherein the step of obtaining the initial token sequences corresponding to the N text data comprises: Get N text data to be replied; For each text data in the N text data, each character in the text data is encoded into a corresponding token to obtain an initial token sequence corresponding to the text data.

7. The method according to any one of claims 1 to 6, wherein the step of splicing N initial token sequences into a spliced ​​token sequence comprises: Based on the respective termination tokens of the N initial token sequences, splicing and generating a first token sequence; And, based on the tokens of each of the N initial token sequences except the termination token, splicing to generate a second token sequence; The first token sequence is concatenated at the end of the second token sequence to obtain a concatenated token sequence.

8. The method according to claim 7, wherein the first token sequence and the second token sequence are respectively generated by splicing in a preset splicing order, and the splicing order represents the processing order of the N text data; Then the step of splicing and generating a first token sequence based on the respective termination tokens of the N initial token sequences includes: Based on the processing order of the N text data, sequentially concatenate the respective termination tokens of the N initial token sequences to obtain a first token sequence; Then the step of splicing and generating a second token sequence based on the tokens of each of the N initial token sequences except the termination token comprises: Based on the processing order of the N text data, the tokens of the N initial token sequences except the termination token are concatenated in turn to obtain a second token sequence.

9. The method according to claim 8, wherein the processing order of the N text data is determined by any of the following methods: Using the acquisition order of the N text data as the processing order of the N text data; Using the time sequence of the timestamps corresponding to the N text data as the processing sequence of the N text data; The priority levels corresponding to the N text data are arranged in order of high and low as the processing order of the N text data.

10. A text data reasoning device, the device being deployed on a computer device, the device comprising: An acquisition unit is configured to acquire initial token sequences corresponding to N text data, wherein N is an integer greater than 1, and a token in each initial token sequence represents a character in the corresponding text data; A splicing unit splices N initial token sequences into a spliced ​​token sequence, wherein the spliced ​​token sequence is a token sequence composed of the tokens in the N initial token sequences; An inference unit performs inference processing on the concatenated token sequence to obtain N reply token sequences; wherein the N reply token sequences are used to generate reply data for each of the N text data, and the starting number in each reply token sequence is The initial token is obtained by performing inference processing on each token belonging to the same initial token sequence in the concatenated token sequence.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 9 when executing the computer program.

12. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images