Model reasoning method and device for long sequence
By processing long sequences in parallel and combining state space network layer and sub-layer, the calculation and storage efficiency problems of large models when processing long sequences are solved, and efficient long-term memory is achieved.
Patent Information
- Application Number
- CN202411440909.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2024-12-20
AI Technical Summary
When large models process long sequence input, they need to save a large amount of historical state, resulting in reduced computing and storage efficiency.
Multiple inference modules are adopted, each module includes a state space network layer and a sub-layer. By segmenting long sequences into short sequences for parallel inference, the state space network layer generates output vectors and state vectors, and inputs them to the sub-order in sequence. The layer is processed, and the sub-layer includes a self-attention network layer and a feedforward network layer.
It significantly improves the computing and storage efficiency of the large-model inference system, reduces the need to save the historical state of each word unit, and ensures long-term memory.
Smart Images
Figure CN119168072A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of large model reasoning technology, and in particular to a model reasoning method and device for long sequences. Background Art
[0002] With the rapid development of artificial intelligence technology, large models have performed well in handling various complex tasks, especially in the fields of natural language processing and computer vision. However, these large models face a series of challenges when processing long sequence inputs. Long sequence inputs, such as long documents, long videos, or long conversation records, require the model to be able to capture and remember a large amount of contextual information, which poses a severe test to the traditional model architecture.
[0003] At present, the mainstream model reasoning architecture mainly relies on the self-attention mechanism to process the input sequence. The traditional model structure can process each word unit in the sequence in parallel, but as the sequence length increases, its computing and storage requirements increase exponentially. In addition, in order to maintain the contextual information of long sequences, the model needs to save a large amount of historical states, which reduces the computing and storage efficiency of the model reasoning architecture. How to improve the computing and storage efficiency of large model reasoning systems while ensuring long-term memory is a technical defect that needs to be solved at present. Summary of the invention
[0004] The present invention provides a model reasoning method and device for long sequences, which are used to solve the problem that the model reasoning architecture in the prior art needs to save a large number of historical states when processing long sequences, which reduces the computing and storage efficiency of the model reasoning architecture, so as to improve the computing and storage efficiency of the large model reasoning system while ensuring long-term memory.
[0005] The present invention provides a model reasoning method for long sequences, which is used for reasoning a model, wherein the reasoning model includes a plurality of reasoning modules, and the reasoning module includes a state space network layer and a sublayer; For each inference module, the method includes: Receive a long sequence of input, where the long sequence is an original input sequence or an output sequence of a previous reasoning module; Segmenting the long sequence into multiple short sequences, inputting the multiple short sequences into the state space network layer, and performing parallel reasoning on the multiple short sequences according to the previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each short sequence; The output vector sequence and the state vector sequence corresponding to each of the short sequences are sequentially input into the sublayer for processing to obtain the predicted output vector sequence output by the reasoning module.
[0006] According to the model inference method for long sequences provided by the present invention, according to the previous state vector sequence of the state space network layer, multiple short sequences are input in parallel to obtain an output vector sequence and a state vector sequence corresponding to each short sequence, specifically including: According to the previous state vector sequence of the state space network layer, parallel reasoning is performed on the input multiple short sequences to obtain a state vector sequence corresponding to each short sequence; According to each of the short sequences and its corresponding state vector sequence, an output vector sequence corresponding to each of the short sequences is obtained.
[0007] According to the model inference method for long sequences provided by the present invention, the sublayer includes a self-attention network layer and a feedforward network layer; The output vector sequence and the state vector sequence corresponding to each of the short sequences are sequentially input into the sublayer for processing to obtain the predicted output vector sequence output by the inference module, specifically including: Inputting the output vector sequence and the state vector sequence corresponding to each of the short sequences into the self-attention network layer in order for processing, and outputting the self-attention vector sequence; The self-attention vector sequence is input into the feed-forward network layer for processing, and the predicted output vector sequence is output.
[0008] According to the model inference method for long sequences provided by the present invention, after the output vector sequence and the state vector sequence corresponding to each of the short sequences are sequentially input into the sublayer for processing, and the predicted output vector sequence output by the inference module is obtained, the method further includes: When receiving a new input word, adding the new input word to the predefined additional sequence segment; Input the newly input word-unit into the state space network layer, perform reasoning based on the current state vector sequence corresponding to the newly added sequence segment and the newly input word-unit, and generate an updated state vector sequence corresponding to the newly added sequence segment; Reasoning is performed based on the updated state vector sequence corresponding to the word unit and the newly added sequence segment to obtain an output vector corresponding to the word unit.
[0009] According to the model inference method for long sequences provided by the present invention, after obtaining the output vector corresponding to the word unit, the method further includes: Inputting the output vector corresponding to the word unit and the updated state vector sequence corresponding to the newly added sequence segment into the sublayer for processing to obtain the predicted output vector corresponding to the word unit; When the number of word units of the newly added sequence segment reaches a threshold, the reasoning of the newly added sequence segment is stopped, the state vector sequence corresponding to the newly added sequence segment is saved, and the next sequence segment is added.
[0010] According to the model inference method for long sequences provided by the present invention, the sublayer includes a self-attention network layer and a feedforward network layer; the output vector corresponding to the word unit and the updated state vector sequence corresponding to the newly added sequence fragment are input into the sublayer for processing to obtain the predicted output vector corresponding to the word unit, specifically including: Input the output vector corresponding to the word unit and the updated state vector sequence corresponding to the newly added sequence fragment into the self-attention network layer for processing, and output the self-attention vector sequence corresponding to the word unit; The self-attention vector sequence corresponding to the word unit is input into the feedforward network layer for processing, and the predicted output vector sequence corresponding to the word unit is output.
[0011] The present invention also provides a model inference device for long sequences, which is used for inferring a model, wherein the inference model includes a plurality of inference modules, each of which includes a state space network layer and a sublayer; For each reasoning module, the apparatus comprises: A receiving module, used to receive a long sequence of input, where the long sequence is an original input sequence or an output sequence of a previous reasoning module; An inference module, used for segmenting the long sequence into multiple short sequences, inputting the multiple short sequences into the state space network layer, and performing parallel inference on the multiple short sequences input according to the previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each short sequence; The output module is used to input the output vector sequence and the state vector sequence corresponding to each of the short sequences into the sublayer in sequence for processing, so as to obtain the predicted output vector sequence output by the inference module.
[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model inference method for long sequences as described in any one of the above is implemented.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the model inference methods for long sequences described above.
[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described model inference methods for long sequences.
[0015] The model reasoning method and device for long sequences provided by the present invention, by adding a state space network layer to each reasoning module of the reasoning model, the received long sequence is segmented into multiple short sequences, the multiple short sequences are input into the state space network layer, and the multiple short sequences are input in parallel according to the previous state vector sequence of the state space network layer to obtain the output vector sequence and state vector sequence corresponding to each short sequence. The parallel processing mechanism can efficiently process long sequence data and significantly improve the computing efficiency; then the output vector sequence and state vector sequence corresponding to each short sequence are sequentially input into the sublayer for processing to obtain the predicted output vector sequence output by the reasoning module. The method of this embodiment generates the output vector sequence and state vector sequence corresponding to each short sequence through the state space network layer, and only needs to be based on the newly input multiple short sequences and the previous state vector sequence. Compared with the traditional long sequence processing method that needs to save the historical state information of each word unit, it improves the computing and storage efficiency of the large model reasoning system while ensuring long-term memory. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 It is a schematic diagram of the structure of the reasoning model provided by an embodiment of the present invention.
[0018] Figure 2 This is one of the flow charts of the model inference method for long sequences provided in an embodiment of the present invention.
[0019] Figure 3 This is the second flowchart of the model inference method for long sequences provided in an embodiment of the present invention.
[0020] Figure 4 It is a schematic diagram of the structure of a model inference device for long sequences provided in an embodiment of the present invention.
[0021] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] First, the terms involved in the embodiments of the present invention are schematically explained.
[0024] Long sequence: usually refers to data sequences with a large number of elements or combinations of elements in natural language processing (NLP), computer vision or other data-intensive fields. These sequences can be text, images, video frame sequences or other forms of data.
[0025] State space model (SSM): is a statistical model for time series analysis, which represents the state of the system as a dynamic process and describes the changes of the system over time through state variables. SSM models are widely used in signal processing, control theory, economics, biostatistics and other fields. In the SSM model, the processing of each new word depends only on the current word and the previous state vector, rather than the entire historical sequence. This recursive update mechanism allows the model to save only the current state vector at each time step, rather than the historical state of the entire sequence.
[0026] Feed-Forward Network (FFN): This is a network structure commonly used in deep learning, especially in the Transformer model. FFN usually consists of two linear transformations with an activation function in between.
[0027] Token: In natural language processing (NLP), a token is a basic processing unit in a text. Tokenization is the process of segmenting a text into a series of tokens, which can be words, characters, subwords, or other language elements. Tokenization is an important step in text preprocessing, which provides a basis for subsequent text analysis and model training.
[0028] In order to overcome the technical defect that the conventional model inference architecture needs to save the historical state information of each word unit, and the computing and storage requirements grow exponentially when processing long sequences, in an embodiment of the present invention, a state space network layer (SSM) is added to each inference module of the inference model to maintain the state vector sequence.
[0029] Since the state space network layer reasons in a recursive form, the current reasoning only depends on the current input and the state vector sequence of the previous moment. Therefore, compared with the traditional technology that needs to save the historical state information of each word unit, maintaining the state vector sequence through the state space network layer improves the computing and storage efficiency of the large model reasoning system while ensuring long-term memory.
[0030] The structure of the reasoning model involved in the embodiment of the present invention is schematically described below.
[0031] See also Figure 1 The reasoning model includes multiple reasoning modules, and the reasoning module includes a state space network layer and sub-layers; the sub-layers include a self-attention network layer and a feedforward network layer.
[0032] Taking the transformer model as an example, its reasoning module includes multiple encoding reasoning modules located in the encoder and multiple decoding reasoning modules located in the decoder. In this embodiment, for each reasoning module, a state space network SSM layer is added before the sublayer. After each reasoning module receives the input long sequence, the long sequence is first processed by the SSM layer, and then the output of the SSM layer is processed by the self-attention network layer and the feedforward network layer in turn to obtain the output prediction output vector sequence.
[0033] Combine the following Figure 2-Figure 3 The model inference method for long sequences according to an embodiment of the present invention is described.
[0034] Figure 2 This is one of the flow charts of the model inference method for long sequences provided by the present invention, such as Figure 2 As shown, the method includes the following steps 201-203: 201. Receive an input long sequence, where the long sequence is an original input sequence or an output sequence of a previous reasoning module.
[0035] Assume that the use case of the inference model is an intelligent customer service chatbot that needs to understand customer queries and provide appropriate responses. Customer queries can be very long and contain multiple sentences, so they need to be processed as long sequences.
[0036] The user input is "I want to know the latest policies about environmentally friendly electric vehicles, and how these policies affect the purchase cost of cars?" The results of word segmentation include word sequence: ["I", "want", "understand", "about", "environmentally friendly", "type", "electric", "car", "of", "latest", "policy", "," "and", "these", "policies", "how", "affect", "car", "buy", "cost", "?"].
[0037] Assuming that the length of each short sequence is 5 tokens, the sequence ["I", "think", "understand", "about", "environmental protection", "type", "electric", "car", "of", "latest", "policy", "and", "these", "policies"...] is split into the following short sequences: ["I", "think", "understand", "about"], ["environmental protection", "type", "electric", "car", "of"], ["latest", "policy", "and", "these", "policies"],...
[0038] 202. Segment the long sequence into multiple short sequences, input the multiple short sequences into a state space network layer, and perform parallel reasoning on the multiple short sequences according to a previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each short sequence.
[0039] Taking a long sequence of 1,000 tokens as an example, the long sequence is segmented into 10 short sequences, each of which contains 100 word-unit tokens. The 10 short sequences are input into the state space network SSM layer for parallel reasoning to obtain the output vector sequence and state vector sequence corresponding to each short sequence.
[0040] Specifically, step 202 includes: performing parallel reasoning on multiple short input sequences according to the previous state vector sequence of the state space network layer to obtain a state vector sequence corresponding to each short sequence; and obtaining an output vector sequence corresponding to each short sequence according to each short sequence and its corresponding state vector sequence.
[0041] Schematically, taking the example of a short sequence consisting of 5 tokens, a long sequence consists of 4 short sequences: Short sequence 1: [“I”, “think”, “understand”, “know”, “about”].
[0042] Short sequence 2: ["environmental protection", "type", "electric", "car", "of"].
[0043] Short sequence 3: ["latest", "policy", "and", "these", "policy"].
[0044] Short sequence 4: ["how", "affect", "car", "buy", "cost"].
[0045] The four short sequences are input into the state space network layer for parallel reasoning. For the state vector sequence, if the state vector sequence of the previous sequence is available, it can be used to initialize the state; otherwise, a zero vector or other initialization vector is used as the state vector sequence.
[0046] Specifically, the SSM layer is used to calculate the current input short sequence x t and the system's previous state vector sequence Z t-1 , the current output vector sequence X is predicted by system parameters A, B, C, D t+1 : The state vector sequence corresponding to each short sequence: Z t = A*Z t-1 + B *x t
[0047] The output vector sequence corresponding to each short sequence: X t+1 = C*Z t +D*x t
[0048] The state vector sequence corresponding to the obtained short sequence 1 is [s11, s12, s13, s14, s15], and the output vector sequence is [o11, o12, o13, o14, o15]; the state vector sequence corresponding to the short sequence 2 is [s21, s22, s23, s24, s25], and the output vector sequence is [o21, o22, o23, o24, o25]; the state vector sequence corresponding to the short sequence 3 is [s31, s32, s33, s34, s35], and the output vector sequence is [o31, o32, o33, o34, o35]; the state vector sequence corresponding to the short sequence 4 is [s41, s42, s43, s44, s45], and the output vector sequence is [o41, o42, o43, o44, o45].
[0049] It can be seen that the SSM layer only needs to save the state vector sequence of a single moment, and has low requirements for storage and memory access. For example, if the length of the long sequence is 1024, if the conventional Transformer structure is taken as an example, the historical state vector is saved in the form of KV cache, then the corresponding sequence dimension that needs to be saved is 1024*2=2048. The dimension of the state vector sequence maintained by the SSM layer of this embodiment is 16, so the state vector that needs to be saved in the embodiment of the present invention is 16 / (2*1024)=1 / 128 of the conventional Transformer structure, which greatly saves overhead.
[0050] However, since the SSM layer only needs to save the state vector sequence of a single moment, it cannot save longer historical information. Therefore, this embodiment combines the SSM layer with the original sublayer of the reasoning module to achieve reasoning of long sequences.
[0051] 203. Input the output vector sequence and the state vector sequence corresponding to each of the short sequences into the sublayer in order for processing, and obtain the predicted output vector sequence output by the inference module.
[0052] Among them, the output vector sequence and the state vector sequence are both composed of word units.
[0053] Specifically, the sublayers include the self-attention network layer Attention and the feed-forward network layer FFN; Step 203 includes: inputting the output vector sequence and the state vector sequence corresponding to each short sequence into the self-attention network layer in order for processing, and outputting the self-attention vector sequence; inputting the self-attention vector sequence into the feedforward network layer for processing, and outputting the predicted output vector sequence.
[0054] The output vector sequence of each short sequence output by the SSM layer is passed as input to the self-attention network layer. The self-attention network layer calculates the relationship between each word token and other word tokens in the sequence, generates an attention score, and weighted averages the self-attention vector sequence.
[0055] For each short sequence, the self-attention network layer outputs a set of self-attention vectors, which reflect the dependencies within the sequence. For example, the self-attention mechanism may strengthen the relationship between "environmentally friendly electric vehicles" and "policy" in the sequence because they are semantically related.
[0056] The sequence of self-attention vectors is then fed into a feed-forward network layer, which usually consists of two linear transformations with a nonlinear activation function such as ReLU in between. The feed-forward network layer performs further nonlinear transformations on the output of the self-attention layer to enhance the expressiveness of the model.
[0057] The feed-forward network layer outputs the final prediction output vector sequence, which is the final representation of the model for the input sequence. These vector sequences can be used for classification, generating answers or other downstream tasks.
[0058] Assume the following parameters for the self-attention and feed-forward layers: Parameters of the self-attention network layer: W Q , W K , W V ; Parameters of the feedforward network layer: W 1 , W 2 , activation function: ReLU.
[0059] For each short sequence i: Self-attention vector sequence: Attention(Q i , K i , V i )
[0060] Feedforward network output: FFN (Attention i )=max(0,W1 ˙Attention i +b 1 )W 2 +b 2
[0061] Among them, Q i , K i , V i They are respectively the output vector sequence of the short sequence i and the weight matrix W Q , W K , W V The query, key, and value matrices that are multiplied together.
[0062] Finally, for the entire long sequence, the predicted output vector sequences of multiple short sequences are concatenated in their original order to obtain a complete sequence representation, which can be used to generate the final output result.
[0063] It should be noted that the dimension of the predicted output vector sequence should match the output requirements of the model. For example, if it is a classification task, the output dimension should be the same as the number of categories.
[0064] In the above process, the SSM layer saves the state vector sequence of each short sequence, thereby storing more historical state information. The information of the entire sequence is mixed through the self-attention network layer, and then processed by the feedforward network layer to obtain the predicted output vector sequence, thereby realizing the reasoning of long sequences.
[0065] The model reasoning method for long sequences provided by the embodiment of the present invention adds a state space network layer to each reasoning module of the reasoning model, segments the received long sequence into multiple short sequences, inputs the multiple short sequences into the state space network layer, and performs parallel reasoning on the input multiple short sequences according to the previous state vector sequence of the state space network layer to obtain the output vector sequence and state vector sequence corresponding to each short sequence. The parallel processing mechanism can efficiently process long sequence data and significantly improve the computational efficiency; then the output vector sequence and state vector sequence corresponding to each short sequence are sequentially input into the sublayer for processing to obtain the predicted output vector sequence output by the reasoning module. The method of this embodiment generates the output vector sequence and state vector sequence corresponding to each short sequence through the state space network layer, and only needs to be based on the newly input multiple short sequences and the previous state vector sequence. Compared with the traditional long sequence processing method that needs to save the historical state information of each word unit, it improves the computational and storage efficiency of the large model reasoning system while ensuring long-term memory.
[0066] Furthermore, after receiving the long sequence, the inference model will further receive multiple new input words. For the multiple new input words, the state space network layer will not process them in parallel after segmentation like the long sequence, but will infer each word in turn to obtain the output vector corresponding to each word.
[0067] Specifically, see Figure 3 After step 203, the method further includes: 301. When a newly input word-gram is received, the newly input word-gram is added to a predefined newly added sequence segment.
[0068] Step 301 describes the step of receiving new tokens and adding them to new sequence segments during the long sequence model inference process. This usually occurs in interactive scenarios, such as chatbots or real-time translation, where the model needs to dynamically process new input data.
[0069] Suppose the user's first query is "I want to know the policy on environmentally friendly electric vehicles", and then the user continues to enter "Will this affect the purchase cost of the car?"
[0070] Received new word sequence: ["this", "will", "affect", "car", "buy", "cost", "?"].
[0071] Create a new sequence segment (the new sequence segment can include 10 word-gram tokens). For a new word-gram sequence including 7 word-gram tokens, add these 7 word-grams as new input word-grams to the predefined new sequence segment, and in subsequent steps, infer these 7 word-grams in turn.
[0072] The update of the state vector is key because it allows the model to take into account previous contextual information when processing new words.
[0073] For the first word-unit token, the initialization state vector is used as the corresponding current state vector sequence; for the second word-unit token, the updated state vector corresponding to the first word-unit is used as the corresponding current state vector sequence; for the third word-unit token, the updated state vector corresponding to the first word-unit and the updated state vector sequence corresponding to the two words-units are used as the corresponding current state vector sequence... and so on.
[0074] 302. Input the newly input word-gram into the state space network layer, perform reasoning based on the current state vector sequence corresponding to the newly added sequence segment and the newly input word-gram, and generate an updated state vector sequence corresponding to the newly added sequence segment.
[0075] 303. Infer based on the updated state vector sequence corresponding to the token and the newly added sequence segment to obtain the output vector corresponding to the token.
[0076] For steps 302 and 303, it is a process of iterative inference based on each newly input token. Specifically, based on the currently input token x t and the current state vector sequence Z t-1 , predict the output vector X corresponding to the token by system parameters A, B, C, D t+1 : The updated state vector sequence Z corresponding to the newly added sequence segment t = A * Z t-1 + B * x t ; The output vector X corresponding to this token t+1 = C * Z t + D * x t .
[0077] 304. Input the output vector corresponding to the token and the updated state vector sequence corresponding to the newly added sequence segment into the sub-layer for processing to obtain the predicted output vector corresponding to the token.
[0078] Specifically, step 304 includes: inputting the output vector corresponding to the token and the updated state vector sequence corresponding to the newly added sequence segment into the self-attention network layer for processing, and outputting the self-attention vector sequence corresponding to the token; inputting the self-attention vector sequence corresponding to the token into the feed-forward network layer for processing, and outputting the predicted output vector sequence corresponding to the token.
[0079] Suppose the user adds a new token "ma". The model has processed the previous token "cost" to obtain the output vector and the state vector sequence. The following are the processing steps: Receive the output vector and the state vector sequence of the new token "ma", assuming that these vector sequences have been generated through the SSM layer.
[0080] Self-attention network layer processing: Input the output vector and the state vector sequence of "ma" into the self-attention network layer to calculate its association with other tokens in the sequence.
[0081] Feed-forward network layer processing: The self-attention vector sequence is fed into the feed-forward network layer for further transformation.
[0082] Output prediction: The feed-forward network layer outputs the predicted output vector sequence of "ma", and these sequences reflect the model's final understanding of the user input.
[0083] 305 . When the number of word units in the newly added sequence segment reaches a threshold, stop reasoning about the newly added sequence segment, save the state vector sequence corresponding to the newly added sequence segment, and add the next sequence segment.
[0084] The model keeps track of the number of tokens in a newly added sequence segment. When the number reaches a preset threshold (e.g., a maximum processing length or memory limit is reached), the model stops processing the segment further.
[0085] Before stopping processing, the model saves the current sequence of state vectors. This is important because it allows the model to retain the current contextual information when processing the next sequence segment. Using the saved sequence of state vectors and the previously generated sequence of output vectors, the model generates the final predicted output vector for the current sequence segment.
[0086] The model then initializes a new sequence segment, ready to receive subsequent input tokens.
[0087] The embodiment of the present invention is based on the mainstream deep model structure and can be implemented on mainstream GPU servers. In particular, the segmented reasoning of the SSM layer is implemented by parallel computing, so it can be distributed to different computing cards and computing nodes to complete simultaneously, which can further improve computing efficiency and processing capacity.
[0088] In addition, the embodiment of the present invention significantly improves the efficiency of the model in processing long sequence data by dividing the long sequence into multiple short sequences for parallel processing.
[0089] The state space network layer (SSM) is used to maintain the state vector sequence, which reduces the need to store the historical state of each word unit, thereby reducing the consumption of storage and computing resources. Compared with the traditional long sequence processing method, this embodiment only needs to be based on multiple newly input short sequences and the previous state vector sequence. Compared with the traditional long sequence processing method that needs to save the historical state information of each word unit, it improves the computing and storage efficiency of the large model reasoning system while ensuring long-term memory.
[0090] The following describes a model reasoning device for a long sequence provided by an embodiment of the present invention. The model reasoning device for a long sequence described below and the model reasoning method for a long sequence described above can refer to each other.
[0091] The embodiment of the present invention provides a model reasoning device for long sequences, which is used for reasoning a model, wherein the reasoning model includes a plurality of reasoning modules, and each reasoning module includes a state space network layer and a sublayer; For each inference module, see Figure 4 , the device comprises: A receiving module 401 is used to receive an input long sequence, where the long sequence is an original input sequence or an output sequence of a previous reasoning module; The reasoning module 402 is used to segment the long sequence into multiple short sequences, input the multiple short sequences into the state space network layer, and perform parallel reasoning on the input multiple short sequences according to the previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each short sequence; The output module 403 is used to input the output vector sequence and the state vector sequence corresponding to each of the short sequences into the sublayer in order for processing, so as to obtain the predicted output vector sequence output by the inference module.
[0092] Optionally, the reasoning module 402 is specifically used to: perform parallel reasoning on multiple input short sequences according to the previous state vector sequence of the state space network layer to obtain a state vector sequence corresponding to each of the short sequences; and obtain an output vector sequence corresponding to each of the short sequences according to each of the short sequences and its corresponding state vector sequence.
[0093] Optionally, the sublayer includes a self-attention network layer and a feed-forward network layer; The output module 403 is specifically used to: input the output vector sequence and the state vector sequence corresponding to each of the short sequences into the self-attention network layer in sequence for processing, and output the self-attention vector sequence; input the self-attention vector sequence into the feedforward network layer for processing, and output the predicted output vector sequence.
[0094] Optionally, the device further comprises: A new word-unit input module is used to input the output vector sequence and the state vector sequence corresponding to each of the short sequences into the sublayer in order for processing, and after obtaining the predicted output vector sequence output by the inference module, add the newly input word-unit to the predefined newly added sequence segment when receiving the newly input word-unit; A state vector reasoning module, used to input the newly input word unit into the state space network layer, perform reasoning based on the current state vector sequence corresponding to the newly added sequence segment and the newly input word unit, and generate an updated state vector sequence corresponding to the newly added sequence segment; The output vector inference module is used to perform inference based on the updated state vector sequence corresponding to the word unit and the newly added sequence fragment to obtain the output vector corresponding to the word unit.
[0095] Optionally, the device further comprises a prediction output module, configured to: After obtaining the output vector corresponding to the word unit, the output vector corresponding to the word unit and the updated state vector sequence corresponding to the newly added sequence segment are input into the sublayer for processing to obtain the predicted output vector corresponding to the word unit; When the number of word units of the newly added sequence segment reaches a threshold, the reasoning of the newly added sequence segment is stopped, the state vector sequence corresponding to the newly added sequence segment is saved, and the next sequence segment is added.
[0096] Optionally, the prediction output module is specifically used to: input the output vector corresponding to the word unit and the updated state vector sequence corresponding to the newly added sequence fragment into the self-attention network layer for processing, and output the self-attention vector sequence corresponding to the word unit; input the self-attention vector sequence corresponding to the word unit into the feedforward network layer for processing, and output the prediction output vector sequence corresponding to the word unit.
[0097] The model inference device for long sequences provided in an embodiment of the present invention generates an output vector sequence and a state vector sequence corresponding to each short sequence through a state space network layer. It only needs to rely on multiple newly input short sequences and the previous state vector sequence. Compared with the traditional long sequence processing method that needs to save the historical state information of each word unit, it improves the computing and storage efficiency of the large model inference system while ensuring long-term memory.
[0098] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the model reasoning method for a long sequence. For each reasoning module, the method includes: receiving an input long sequence, the long sequence being the original input sequence or the output sequence of the previous reasoning module; segmenting the long sequence into multiple short sequences, inputting the multiple short sequences into the state space network layer, and performing parallel reasoning on the input multiple short sequences according to the previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each of the short sequences; and inputting the output vector sequence and the state vector sequence corresponding to each of the short sequences into the sublayer in sequence for processing to obtain a predicted output vector sequence output by the reasoning module.
[0099] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model inference method for long sequences provided by the above methods. For each inference module, the method includes: receiving an input long sequence, which is the original input sequence or the output sequence of the previous inference module; segmenting the long sequence into multiple short sequences, inputting the multiple short sequences into the state space network layer, and performing parallel inference on the input multiple short sequences according to the previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each of the short sequences; inputting the output vector sequence and the state vector sequence corresponding to each of the short sequences into the sublayer in sequence for processing to obtain a predicted output vector sequence output by the inference module.
[0101] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the model inference method for long sequences provided by the above-mentioned methods. For each inference module, the method includes: receiving an input long sequence, wherein the long sequence is the original input sequence or the output sequence of the previous inference module; segmenting the long sequence into multiple short sequences, inputting the multiple short sequences into the state space network layer, and performing parallel inference on the input multiple short sequences according to the previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each of the short sequences; and inputting the output vector sequence and the state vector sequence corresponding to each of the short sequences into the sublayer in sequence for processing to obtain a predicted output vector sequence output by the inference module.
[0102] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A model inference method for long sequences, It is characterized in that For an inference model, the inference model includes a plurality of inference modules, and the inference modules include a state space network layer and sublayers; For each inference module, the method includes: Receive a long sequence of input, where the long sequence is an original input sequence or an output sequence of a previous reasoning module; Segmenting the long sequence into multiple short sequences, inputting the multiple short sequences into the state space network layer, and performing parallel reasoning on the multiple short sequences according to the previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each short sequence; The output vector sequence and the state vector sequence corresponding to each of the short sequences are sequentially input into the sublayer for processing to obtain the predicted output vector sequence output by the reasoning module.
2. The model inference method for long sequences according to claim 1, It is characterized in that According to the last state vector sequence of the state space network layer, parallel reasoning is performed on the input multiple short sequences, specifically including: According to the previous state vector sequence of the state space network layer, parallel reasoning is performed on the input multiple short sequences to obtain a state vector sequence corresponding to each short sequence; According to each of the short sequences and its corresponding state vector sequence, an output vector sequence corresponding to each of the short sequences is obtained.
3. The model inference method for long sequences according to claim 1, It is characterized in that The sublayers include a self-attention network layer and a feedforward network layer; The output vector sequence and the state vector sequence corresponding to each of the short sequences are sequentially input into the sublayer for processing to obtain the predicted output vector sequence output by the inference module, specifically including: Inputting the output vector sequence and the state vector sequence corresponding to each of the short sequences into the self-attention network layer in order for processing, and outputting the self-attention vector sequence; The self-attention vector sequence is input into the feed-forward network layer for processing, and the predicted output vector sequence is output.
4. The model inference method for long sequences according to claim 1, It is characterized in that After the output vector sequence and the state vector sequence corresponding to each of the short sequences are sequentially input into the sublayer for processing to obtain the predicted output vector sequence output by the inference module, the method further includes: When receiving a newly input word, adding the newly input word to the predefined additional sequence segment; Input the newly input word-unit into the state space network layer, perform reasoning based on the current state vector sequence corresponding to the newly added sequence segment and the newly input word-unit, and generate an updated state vector sequence corresponding to the newly added sequence segment; Reasoning is performed based on the updated state vector sequence corresponding to the word unit and the newly added sequence segment to obtain an output vector corresponding to the word unit.
5. The model inference method for long sequences according to claim 4, It is characterized in that After obtaining the output vector corresponding to the word unit, the method further includes: Inputting the output vector corresponding to the word unit and the updated state vector sequence corresponding to the newly added sequence segment into the sublayer for processing to obtain the predicted output vector corresponding to the word unit; When the number of word units of the newly added sequence segment reaches a threshold, the reasoning of the newly added sequence segment is stopped, the state vector sequence corresponding to the newly added sequence segment is saved, and the next sequence segment is added.
6. The model inference method for long sequences according to claim 5, It is characterized in that The sublayers include a self-attention network layer and a feedforward network layer; The output vector corresponding to the word unit and the updated state vector sequence corresponding to the newly added sequence segment are input into the sublayer for processing to obtain the predicted output vector corresponding to the word unit, specifically including: Input the output vector corresponding to the word unit and the updated state vector sequence corresponding to the newly added sequence fragment into the self-attention network layer for processing, and output the self-attention vector sequence corresponding to the word unit; The self-attention vector sequence corresponding to the word unit is input into the feedforward network layer for processing, and the predicted output vector sequence corresponding to the word unit is output.
7. A model inference device for long sequences, It is characterized in that For an inference model, the inference model includes a plurality of inference modules, each of the inference modules includes a state space network layer and a sublayer; For each reasoning module, the apparatus comprises: A receiving module, used to receive a long sequence of input, where the long sequence is an original input sequence or an output sequence of a previous reasoning module; An inference module, used for segmenting the long sequence into multiple short sequences, inputting the multiple short sequences into the state space network layer, and performing parallel inference on the multiple short sequences input according to the previous state vector sequence of the state space network layer to obtain an output vector sequence and a state vector sequence corresponding to each short sequence; The output module is used to input the output vector sequence and the state vector sequence corresponding to each of the short sequences into the sublayer in sequence for processing, so as to obtain the predicted output vector sequence output by the inference module.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the model inference method for long sequences is implemented as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the model inference method for long sequences as described in any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the model inference method for long sequences as described in any one of claims 1 to 6 is implemented.