Text information extraction method and system based on hybrid model
By fusing the linear state space model and the optimized Transformer model, the problem of excessive inference delay in long text information extraction is solved, and efficient text information extraction is achieved, combining global and local information modeling capabilities.
Patent Information
- Application Number
- CN202411027795.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-29
Smart Images

Figure CN119047562A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of information extraction, and in particular to a text information extraction method and system based on a hybrid model. Background Art
[0002] With the advent of the big data era, text data, as an important part of it, has become the focus of attention in all walks of life. Text information extraction technology is based on the extraction of valuable information from huge and complex text data, that is, automatically extracting specific information from huge unstructured or semi-structured data. At present, the more commonly used models are based on the Transformer architecture, but the complexity of its training depends on the length of training and the dimension of the model. The length and dimension can be used to derive the complexity of generating a single word when the Transformer model is inferring. The complexity of generating each word grows linearly with the total length of the text between the current word, that is, the longer the text, the higher the complexity of generation and the higher the delay of inference. Therefore, due to the high inference delay caused by the Transformer model itself, it is easy to have the problem of low efficiency of text information extraction.
[0003] In view of this, a text information extraction method and system based on a hybrid model is needed. Summary of the invention
[0004] The embodiments of the present application provide a text information extraction method and system based on a hybrid model, which are used to solve the problem of low efficiency in text information extraction.
[0005] The first aspect of the embodiment of the present application provides a text information extraction method based on a hybrid model, comprising:
[0006] Convert the acquired information to be extracted into a text sequence;
[0007] Constructing a hybrid model that combines a linear state-space model and an optimized Transformer model;
[0008] Inputting the text sequence into the hybrid model so that the encoding network in the hybrid model performs feature encoding on the text sequence to obtain a feature representation corresponding to the text sequence;
[0009] The feature representation is decoded using the decoding network in the hybrid model, and an extraction result corresponding to the information to be extracted is output.
[0010] Furthermore, the hybrid model is constructed, and the hybrid model combines the linear state space model and the optimized Transformer model, including:
[0011] The linear state space model includes continuous form, cyclic form and convolutional form. The continuous form is used to initialize the parameter matrix, the cyclic form is used for inference, and the convolutional form is used for high-speed parallel training.
[0012] The optimized Transformer model includes gated linear units, RMS normalization layers, grouped query self-attention mechanism, and sliding window attention.
[0013] Furthermore, the expression of the continuous form of the linear state space model is as follows:
[0014] x(t)=Ax(t)+Bu(t);
[0015] y(t)=Cx(t)+Du(t);
[0016] Where: u(t) is the d-dimensional signal input at the current time step t, x(t) is the current hidden layer state, y(t) is the output at time step t, and A, B, C, and D are learnable parameter matrices.
[0017] Furthermore, the expression of the linear state space model in the cyclic form is as follows:
[0018]
[0019] y n =Cx n ∈R 1×d ;
[0020]
[0021] Where: N is the dimension of the state space, Δ∈R d is the discrete step size.
[0022] Furthermore, the expression of the linear state space model in the convolution form is as follows:
[0023]
[0024]
[0025] Where: is the global convolution kernel, and L is the length of the text sequence.
[0026] Furthermore, the expression of the gated linear unit is as follows:
[0027] Relu 2 (X) = max(0, X) 2 ;
[0028]
[0029] Y = OW2;
[0030] Where: W1, V∈R d×d′ is the parameter matrix of the linear transformation of dimensionality increase, d′ is the dimension after dimensionality reduction, Relu 2 (·) is a nonlinear activation function, O is the output representation, W2∈R d′×d is the parameter matrix of the dimensionality reduction linear transformation.
[0031] Furthermore, the expression of the normalization layer is as follows:
[0032]
[0033] Where: Mean(x 2 ) is the root mean square.
[0034] A second aspect of the embodiment of the present application provides a text information extraction system based on a hybrid model, comprising:
[0035] A text sequence conversion unit, used to convert the acquired information to be extracted into a text sequence;
[0036] A hybrid model building unit, used to build a hybrid model, wherein the hybrid model combines a linear state space model and an optimized Transformer model;
[0037] A feature encoding unit, used for inputting the text sequence into the hybrid model, so that the encoding network in the hybrid model performs feature encoding on the text sequence to obtain a feature representation corresponding to the text sequence;
[0038] An extraction result output unit is used to use the decoding network in the hybrid model to decode the feature representation and output the extraction result corresponding to the information to be extracted.
[0039] A third aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the steps of the hybrid model-based text information extraction method as described in any one of the above are implemented.
[0040] A fourth aspect of an embodiment of the present application provides a readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the steps of the text information extraction method based on the hybrid model as described in any one of the above are implemented.
[0041] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0042] This application first converts the acquired information to be extracted into a text sequence and constructs a hybrid model, wherein the hybrid model integrates the linear state space model and the optimized Transformer model, and then inputs the text sequence into the hybrid model, so that the encoding network in the hybrid model performs feature encoding on the text sequence, obtains the feature representation corresponding to the text sequence, and finally uses the decoding network in the hybrid model to decode the feature representation and output the extraction result corresponding to the information to be extracted. This application optimizes the Transformer model in many aspects and introduces a linear state space model. The linear state space model captures and models the long text information in the entire sequence, ensuring that the text information can be effectively transmitted and expressed in the long text sequence. The hybrid model not only retains the low inference latency characteristics of the overall model, but also can accurately capture and model the local information within the attention window. The hybrid model complements each other's advantages and effectively improves the efficiency of text information extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A flow chart of an embodiment of a method for extracting text information based on a hybrid model in the present invention;
[0044] Figure 2 It is a schematic diagram of the inference delay of the large model in the present invention;
[0045] Figure 3 A schematic diagram of group query attention and multi-head attention in the present invention;
[0046] Figure 4 It is the sliding window attention matrix diagram in the present invention;
[0047] Figure 5 It is a structural schematic diagram of the hybrid model in the present invention;
[0048] Figure 6 It is a structural block diagram of a text information extraction system based on a hybrid model in the present invention;
[0049] Figure 7 The figure is a structural block diagram of a computer device in the present invention. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0051] The text information extraction method based on the hybrid model in this embodiment is used to effectively improve the efficiency of text information extraction by fusing the linear state space model with the optimized Transformer model. The implementation method in this embodiment can be implemented in the system, can be implemented in the server, and can also be implemented in the terminal, without specific limitation.
[0052] Embodiment 1
[0053] See also Figure 1 , an embodiment of a text information extraction method based on a hybrid model in the present invention comprises the following steps:
[0054] S11. Convert the acquired information to be extracted into a text sequence;
[0055] In practical applications, the current large models are based on the Transformer architecture, and the complexity of their training is T = (2n 2 d+4nd 2 )+12nd 2 , where n is the length of the training and d is the dimension of the model. The complexity of generating a single word during Transformer model reasoning is (2n c d+4d 2 )+12d 2 , where n c Represents the total length of the text before the current word, which indicates the complexity of reasoning and n c It shows linear correlation, and as the generated sequence becomes longer (n c The complexity required to generate each word grows linearly, such as Figure 2 As shown, the longer the text, the higher the complexity of generation and the higher the delay of reasoning.
[0056] Therefore, in order to reduce the inference delay of large models, the present invention first optimizes the structure of its basic architecture Transformer to overcome the inherent factors that cause the inference delay to be too high. Then introduce a linear model to combine the two. The linear model usually has a lower inference delay and a stronger inductive bias, which can further enhance the global dependency information modeling capability of the Transformer model and reduce the inference delay. First, the system converts the acquired text information to be extracted into a text sequence, so that subsequent encoding feature representation and decoding processing can be performed.
[0057] S12. Build a hybrid model that combines the linear state space model and the optimized Transformer model;
[0058] In this embodiment, the linear state space model includes a continuous form, a cyclic form and a convolutional form. The continuous form is used to initialize the parameter matrix, the cyclic form is used for reasoning, and the convolutional form is used for high-speed parallel training.
[0059] The optimized Transformer model, here named Sliding Windows Transformer++, SWT++ model, includes gated linear units, RMS normalization layers, grouped query self-attention mechanism, and sliding window attention.
[0060] The hybrid model combines the linear state space model (SSM) and the SWT++ model, here it is State space model Based Sl iding Windows Transformer++, SSWT++.
[0061] 1. Specifically, the state space model (SSM) is a linear model. The SSM setting has the following definition:
[0062] First, the d-dimensional signal u(t) input at the current time step t is mapped to N×d dimensions, where N represents the dimension of the state space. Then it is fused with the current hidden layer state x(t) and then remapped to d dimensions as the output y(t) at time step t. The expression of the continuous form of the linear state space model is as follows:
[0063] x(t)=Ax(t)+Bu(t);
[0064] y(t)=Cx(t)+Du(t);
[0065] Where: A, B, C, D are learnable parameter matrices.
[0066] Based on the above continuous form of SSM, SSM has the ability to model long text sequences. Here, the Hippo-D matrix is used to initialize the A and B matrices. After the above process, the specially initialized parameter matrices A and B can theoretically enable x(t) to remember all the historical information of the input signal u(t), thereby preventing the occurrence of information forgetting. The expression for initializing the Hippo-D matrix is as follows:
[0067]
[0068] Among them: A matrix is composed of The diagonal matrix composed of the eigenvalues of the matrix, that is, B,C∈R D×Nis the parameter matrix of the model. The calculation process of SSM is single-value-in single-value-out (SISO), that is, each dimension of the current input u(t) uses a set of SSM parameters A, B, C independently. In addition, to simplify the processing, the parameter matrix D acting as the residual connection is omitted.
[0069] In order to model the text sequence, the continuous SSM needs to be discretized to adapt to the discrete text sequence. Here, the zero-order hold method (ZOH) is used to discretize the SSM. The expression of the SSM converted into a discrete form, that is, the cyclic linear state space model, is as follows:
[0070]
[0071] y n =Cx n ∈R 1 × d ;
[0072]
[0073] Where: N is the dimension of the state space, Δ∈R d is the discrete step size.
[0074] The above-mentioned cyclic form of SSM has the advantage of low inference latency, but it is difficult to train in parallel during training because the dimension (N×d) of the hidden layer state is too large.
[0075] In order to enable parallel training, the model also has a third form that can meet high-speed parallel training, namely the convolution form. After expanding the loop body of the cyclic form of SSM, the global calculation form is obtained as follows:
[0076]
[0077] The above form can be vectorized into a global convolution form, and has d (the dimension of the hidden state) convolution kernels with the same length as the text sequence length, expressed as follows:
[0078]
[0079] Where: is the global convolution kernel, and L is the length of the text sequence.
[0080] The convolution form of the above SSM, as long as the convolution kernel is given The global convolution form of SSM can be parallelized by fast Fourier transform (FFT) with a time complexity of O(LlogL).
[0081] because The matrix is a diagonal matrix, calculating the convolution kernel The time complexity of is very low and can be ignored. The calculation method is shown in the following formula:
[0082] in,
[0083] Where: N represents the parameter C and The vector dimension of the vector, C n , Represents parameters C and The nth element of a vector, Representation parameters The nth eigenvalue of the matrix is raised to the power of l, ⊙ represents the Hadamard product, V L represents the Vandermonde matrix, as shown in the following formula:
[0084]
[0085] Using the Vandermonde determinant, the convolution kernel can be calculated with a time complexity of O(L+N) As mentioned above, each dimension of the hidden state requires a unique set of parameters (element-wise, EW) and independent full convolution calculations. Therefore, the number of convolution kernels is d, and d full convolution calculations are required. Therefore, the final calculation time complexity of the convolutional SSM is O(LlogLd+Ld+Nd). It can be seen that the overall time complexity of SSM is better than the quadratic complexity of transformer calculations, and it also has good parallelism.
[0086] The three forms of SSM mentioned above, namely continuous, convolutional and cyclic, can give full play to their advantages in text sequence modeling. The continuous form is used to initialize the parameter matrix to prevent the model from forgetting information; the convolutional form is used for high-speed parallel training to improve the efficiency of training; the cyclic form is used for reasoning to reduce the latency during reasoning.
[0087] 2. Specifically, the Transformer model uses a gated linear unit (GLU) to replace the original FFN layer. The gating mechanism separates the nonlinear structure from the linear structure in the FFN layer, effectively alleviating the possible gradient vanishing problem while retaining the nonlinear capability of the FFN layer. In order to further enhance the nonlinear characteristics of the Transformer model, the ReLU with stronger nonlinear characteristics is used here. 2 The function is used as a nonlinear activation function to replace the original activation function. The expression is as follows:
[0088] Relu 2 (X) = max(0, X) 2 ;
[0089]
[0090] Y = OW2;
[0091] Where: W1, V∈R d×d′ is the parameter matrix of the linear transformation of dimensionality increase, d′ is the dimension after dimensionality reduction, Relu 2 (·) is a nonlinear activation function, O is the output representation, W2∈R d ′× d is the parameter matrix of the dimensionality reduction linear transformation.
[0092] The normalization layer in the Transformer model uses root mean square normalization (RMS norm) to replace the original layer normalization (LN). Here, using RMS norm as the normalization function has lower computational complexity and increases the computational speed by 40%, without affecting performance. The specific calculation process expression is as follows:
[0093]
[0094] Where: Mean(x 2 ) is the root mean square, not the mean used in layer norm.
[0095] In order to reduce the computational complexity of the self-attention mechanism, the Transformer model first uses the Group Query Self-Attention (GQA) mechanism to replace the original multi-head self-attention mechanism. Here, the queries are grouped, and the queries in the group can share Key, Value pairs. GQA is close to Multi-head Self-Attention in effect, but effectively saves computational complexity. The comparison of the two attention mechanisms is as follows: Figure 3 As shown in the figure. Secondly, the sliding window attention (SWA) is used to replace the original global attention, which is mainly used to reduce the huge memory consumption and reasoning delay caused by the infinite growth of the key value cache (KVCache) of the Transformer model. The sliding window attention mechanism is as follows Figure 4 shown.
[0096] In the reasoning process of the Transformer model, in order to avoid repeated calculation of the attention matrix, it is necessary to store the key (K) and value (V) matrices of the entire sequence in memory, namely KVCache, which will bring huge memory overhead when facing long sequences. The introduction of the sliding window attention mechanism can limit the scope of attention so that the model only needs to process a smaller window when calculating the attention score matrix, thereby greatly reducing the memory requirements of KVCache. In addition, due to the globality of the calculation of the attention score matrix of the Transformer model, the word unit generated at each time step needs to calculate the attention score with all the words before the word unit, which will cause the reasoning delay of the Transformer model to increase linearly with the increase of KVCache. The sliding window attention mechanism can window the calculation process of the attention matrix, so that the calculation amount of each time step no longer depends on the length of the entire sequence, but only on the size of the attention window, which enables the Transformer model to maintain a low level of reasoning latency when processing long sequences. However, sliding window attention also has disadvantages, that is, it will cause the Transformer model to only focus on the information within the window, while the long text information outside the window is discarded, making the Transformer model lose the ability to model global dependency information. The above-mentioned linear model with the ability to model global dependency information completely solves this problem and gives full play to its advantages in text sequence modeling.
[0097] The structure of the hybrid model formed by integrating the state space model and the SWT++ model is shown in the figure below. Figure 5 As shown in the figure, the main function of the linear model is to capture and model the long text information in the entire sequence, ensuring that the text information can be effectively transmitted and expressed in the long text sequence, and introducing an inductive bias that is beneficial to the long text processing capability. The SWT++ model is an optimized Transformer model. The SWT++ model not only retains the low inference latency characteristics of the overall model, but also can accurately capture and model the local information within the attention window, so that the SWT++ model can complement the linear model.
[0098] S13. Inputting the text sequence into the hybrid model, so that the encoding network in the hybrid model performs feature encoding on the text sequence to obtain feature representation corresponding to the text sequence;
[0099] S14. Use the decoding network in the hybrid model to decode the feature representation and output the extraction result corresponding to the information to be extracted.
[0100] Based on the above hybrid model, the text sequence is input into the hybrid model so that the encoding network performs feature encoding on the text sequence to obtain the feature representation corresponding to the text sequence. The decoding network in the hybrid model is used to decode the feature representation and output the extraction result corresponding to the information to be extracted, where the extraction result is also represented by a text sequence.
[0101] The model framework includes not only the pre-training of the proposed hybrid model and fine-tuning on the proposed information extraction synthetic dataset, but also fine-tuning of multiple open source large models of the same scale for comparison and verification to verify the effectiveness of the proposed model. The following is an explanation in conjunction with the experiment. The experiment uses the LongRE long text information extraction task dataset synthesized by the large model. The statistical data of the training set and the test set are shown in Table 1 and Table 2:
[0102] Table 1
[0103]
[0104] Table 2
[0105]
[0106] The above experimental part aims to evaluate the effectiveness of the data augmentation method and generative information extraction model proposed in this invention. The average perplexity (Perplexity, PPL) of the test set, the average accuracy of each word on the information extraction task (Avg Token Acc), the average accuracy of the information extraction task (Avg Task Acc) and the average word generation time (Time cost) are used as evaluation indicators. The selected baselines include the mainstream general information extraction models: Tinyl lama-1.1B, Gemma-2B, Qwen-1.8B, Mamba-1.4B and RWKV6-1.6B. The comparison results of the model SSWT++ and the above baseline models on the test set Test_set are shown in Table 3:
[0107] Table 3
[0108]
[0109] By observing the performance results of different models in the table on different indicators of the test set Test_set2, the following analysis results are obtained:
[0110] 1. Under the PPL measurement index on Test_set2, the performance of the SSWT++ model is slightly worse than that of Gemma-2B and Qwen-1.8B, but better than Tinyl lama-1.1B, Mamba-1.4B, and RWKV6-1.6B. This proves that the three proposed models have good text modeling effects on information extraction datasets.
[0111] 2. SSWT++ achieved the best performance in the average token accuracy (Avg TokenAcc.) and average accuracy (Avg Task Acc.) of the information extraction task on Test_set2. The two indicators of Test_set2 were 0.8% and 0.64% higher than the best baseline model Qwen-1.8B, respectively, and its effectiveness in long text information extraction tasks was proven.
[0112] 3. Average word generation time on Test_set2 (Tim e cos t) Under the measurement index, the word unit generation speed of the SSWT++ model is only slower than that of Mamba-1.4B, and the generation speed is 40.54 times faster than Tinyllama-1.1B, 42.85 times faster than Gemma-2B, and 41.89 times faster than Qwen-1.8B. This proves the efficient generation ability of the proposed model.
[0113] The above embodiment analyzes the influencing factors of insufficient performance and high inference delay of the Transformer model in the scenario of long text information extraction, proposes an optimization model of the Transformer model, namely the SWT++ model, and then proposes a hybrid model formed by the fusion of the linear state space model and SWT++, namely the SSWT++ model. The global dependency information modeling ability and inductive bias of the linear state space model make up for the weak long text processing ability of the SWT++ model. At the same time, the SWT++ model also makes up for the local dependency information modeling ability of the linear state space model. The SSWT++ model combines the respective advantages of the linear state space model and the SWT++ model, and effectively improves the efficiency of text information extraction through pre-training and fine-tuning on the information extraction synthetic data set.
[0114] Embodiment 2
[0115] See also Figure 6 , an embodiment of a text information extraction system based on a hybrid model in the present invention comprises the following steps:
[0116] A text sequence conversion unit 101 is used to convert the acquired information to be extracted into a text sequence;
[0117] A hybrid model construction unit 102, used to construct a hybrid model, the hybrid model combines the linear state space model and the optimized Transformer model;
[0118] A feature encoding unit 103 is used to input the text sequence into the hybrid model so that the encoding network in the hybrid model performs feature encoding on the text sequence to obtain a feature representation corresponding to the text sequence;
[0119] The extraction result output unit 104 is used to use the decoding network in the hybrid model to decode the feature representation and output the extraction result corresponding to the information to be extracted.
[0120] For the specific definition of the system, please refer to the definition of the method above, which will not be repeated here. Each module in the above system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0121] Embodiment 3
[0122] See also Figure 7 The present invention provides a computer device, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the above method are implemented.
[0123] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0124] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored. In addition, each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0125] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.
[0126] It can be understood that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A text information extraction method based on a hybrid model, It is characterized in that include: Convert the acquired information to be extracted into a text sequence; Constructing a hybrid model that combines a linear state-space model and an optimized Transformer model; Inputting the text sequence into the hybrid model so that the encoding network in the hybrid model performs feature encoding on the text sequence to obtain a feature representation corresponding to the text sequence; The feature representation is decoded using the decoding network in the hybrid model, and an extraction result corresponding to the information to be extracted is output.
2. The text information extraction method based on the hybrid model according to claim 1, It is characterized in that The hybrid model is constructed, and the hybrid model combines the linear state space model and the optimized Transformer model, including: The linear state space model includes continuous form, cyclic form and convolutional form. The continuous form is used to initialize the parameter matrix, the cyclic form is used for inference, and the convolutional form is used for high-speed parallel training. The optimized Transformer model includes gated linear units, RMS normalization layers, grouped query self-attention mechanism, and sliding window attention.
3. The text information extraction method based on the hybrid model according to claim 2, It is characterized in that The linear state space model in continuous form is expressed as follows: x(t)=Ax(t)+Bu(t); y(t)=Cx(t)+Du(t); Where: u(t) is the d-dimensional signal input at the current time step t, x(t) is the current hidden layer state, y(t) is the output at time step t, and A, B, C, D are learnable parameter matrices.
4. The text information extraction method based on the hybrid model according to claim 2, It is characterized in that The expression of the linear state space model in the cyclic form is as follows: y n =Cx n ∈R 1×d ; Where: N is the dimension of the state space, Δ∈R d is the discrete step size.
5. The text information extraction method based on the hybrid model according to claim 2, It is characterized in that The expression of the linear state space model in convolutional form is as follows: Where: is the global convolution kernel, and L is the length of the text sequence.
6. The text information extraction method based on the hybrid model according to claim 2, It is characterized in that The expression of the gated linear unit is as follows: resume 2 (X)=max(0,X) 2 ; Y=OW 2 ; Where: W 1 ,V∈R d×d′ is the parameter matrix of the linear transformation of dimensionality increase, d′ is the dimension after dimensionality reduction, Relu 2 (·) is a nonlinear activation function, O is the output representation, W 2 ∈R d′×d is the parameter matrix of the dimensionality reduction linear transformation.
7. The text information extraction method based on the hybrid model according to claim 2, It is characterized in that The expression of the normalization layer is as follows: Where: Mean(x 2 ) is the root mean square.
8. A text information extraction system based on hybrid model, It is characterized in that include: A text sequence conversion unit, used to convert the acquired information to be extracted into a text sequence; A hybrid model building unit, used to build a hybrid model, wherein the hybrid model combines a linear state space model and an optimized Transformer model; A feature encoding unit, used for inputting the text sequence into the hybrid model, so that the encoding network in the hybrid model performs feature encoding on the text sequence to obtain a feature representation corresponding to the text sequence; An extraction result output unit is used to use the decoding network in the hybrid model to decode the feature representation and output the extraction result corresponding to the information to be extracted.
9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer-readable instructions, the steps of the text information extraction method based on the hybrid model as described in any one of claims 1 to 7 are implemented.
10. A readable storage medium storing computer-readable instructions. It is characterized in that When the computer-readable instructions are executed by a processor, the steps of the text information extraction method based on the hybrid model as described in any one of claims 1 to 7 are implemented.
