Intelligent text feature extraction and analysis hardware
By using multi-layer model and MSDFN model in intelligent text feature extraction and analysis hardware, combined with N-gram and Transformer technology, the problem of difficult to balance processing speed and data accuracy in the existing technology is solved, and more efficient and accurate text feature extraction is achieved.
Patent Information
- Application Number
- CN202411891631.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-04-18
AI Technical Summary
When using the existing intelligent text feature extraction and analysis hardware, it is difficult to take into account both processing speed and data sparsity by using the N-gram model alone, resulting in difficulty in taking into account both extraction speed and accuracy.
Multi-layer model is used for text feature extraction, including the underlying layer using N-gram to capture local features, the middle layer extends the context range, and the high layer captures global and long-distance features based on the MSDFN model, and performs feature fusion and smoothing processing.
Through the combination of hierarchical model and MSDFN model, we can understand the text more comprehensively, improve the comprehensiveness and accuracy of information extraction, and at the same time have high flexibility and adaptability, and can be optimized on different tasks and data sets to achieve the best results.
Smart Images
Figure CN119848761A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent text feature extraction and analysis, and in particular to intelligent text feature extraction and analysis hardware. Background Art
[0002] Intelligent text feature extraction and analysis hardware is a hardware device specifically used to process text data. It can extract useful features from text data through various algorithms and techniques, and analyze and process them. This hardware device is usually equipped with a high-performance processor and memory, as well as software and algorithms specifically used for text processing. Intelligent text feature extraction and analysis hardware can be applied to multiple fields, such as natural language processing, information retrieval, sentiment analysis, text classification, etc. It can help users process large amounts of text data more quickly and accurately, thereby improving work efficiency and the accuracy of data analysis. In general, intelligent text feature extraction and analysis hardware is an efficient tool that can help users process text data and can play an important role in various application scenarios.
[0003] However, the existing intelligent text feature extraction and analysis hardware uses the N-gram model to process problems. However, it is difficult to balance processing speed and data sparsity by using the N-gram model alone. The pursuit of processing speed often makes the extraction less accurate, while the pursuit of accurate data extraction often slows down the overall processing speed.
[0004] To solve this problem, we propose intelligent text feature extraction and analysis hardware. Summary of the invention
[0005] 1. Technical issues to be solved
[0006] In view of the deficiencies in the prior art, the present invention provides intelligent text feature extraction and analysis hardware, which has the advantages of capturing the specific semantics of events, capturing more complete semantics in a shorter time, and improving the efficiency of semantic feature extraction, and can effectively solve the problems in the background technology.
[0007] (II) Technical solution
[0008] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: intelligent text feature extraction and analysis hardware, including a collection module for collecting text data;
[0009] A data processing module is used to extract grammatical and semantic features from text data;
[0010] A feature analysis module, used for analyzing feature information according to the grammatical and semantic features extracted by the data processing module;
[0011] The data processing module performs grammatical and semantic feature extraction including:
[0012] S1. Building the bottom layer: Using N-gram to capture local grammatical and semantic features;
[0013] S2. Build the middle layer: Combine the N-gram model to expand the context range;
[0014] S3. Build high-level layers: Capture global and long-range features based on the MSDFN model;
[0015] S4. Feature fusion: Fusion of features extracted at different levels;
[0016] S5. Smooth the data after feature fusion.
[0017] Preferably, the data processing module also includes preprocessing the text before extracting grammatical and semantic features, and the text preprocessing includes word segmentation, building a vocabulary, text indexing, and serialization, dividing the text string into smaller units, creating a vocabulary, mapping all unique words or characters to a unique index, using the vocabulary to replace each word or character in the text with its corresponding index, and converting the indexed text into a numerical sequence.
[0018] Preferably, the MSDFN model includes a time series decomposition module, a seasonal component prediction module and a trend component prediction module;
[0019] The MSDFN model sets three latest long sequences based on Transformer to embed the time series decomposition module to predict the seasonal component and the placeholder of the prediction sequence, wherein the three latest long sequences are scalar, timestamp information and location information respectively;
[0020] The time series decomposition module eliminates periodic fluctuations through sliding average to highlight long-term trends, and the seasonal component prediction module consists of a time series encoding layer and a multi-scale seasonal feature capture layer;
[0021] The input sequence is decomposed using the time series decomposition module to obtain trend components, and the pre-decomposed trend components are modeled and predicted separately.
[0022] Preferably, the time series decomposition module, for the original input sequence X I , the processing process is as follows:
[0023] X t =Softmax(L(X I ))*(F(X I ));
[0024] X S =X I -X t ;
[0025] Where X s , X t ∈R I ×d is the seasonal component and trend component obtained by MOEDecomp. The Softmax function is used to convert the linear output into a probability distribution. I ) represents the extraction of seasonal trends, F(X I ) represents the decomposition of seasonal component and trend component.
[0026] Preferably, when performing feature fusion in step S4, features extracted by the N-gram model, the method of extending the context range, and different levels of the MSDFN model are captured and spliced and fused on the output of the position encoding layer in the Transformer model.
[0027] Preferably, the data after feature fusion is processed, and the high-order probability information is allocated to the low-order model using a linear combination of the high-order model and the low-order model.
[0028] Preferably, the smoothing process in step S5 includes using an interpolation method to perform smoothing, determining missing values in the data or abnormal fluctuations that need to be smoothed, and using spline interpolation to perform calculation and filling.
[0029] Preferably, the feature analysis module is used to analyze the feature information according to the grammatical and semantic features extracted by the data processing module using the LDA model for data analysis, and the specific analysis steps are as follows:
[0030] S01. Select LDA parameters: Determine the hyperparameters of Dirichlet distribution, document-topic distribution α and vocabulary-subject distribution β, based on the number of discovered topics K. The β parameter controls the distribution of vocabulary among topics. β is set to a constant or a function of the vocabulary size.
[0031] S02. Conduct topic explanation: Check the first 2-5 words under each topic and explain the meaning of each topic;
[0032] S03. Result interpretation and visualization: Use pyLDAvis to display the results of the topic model to help better understand the topic distribution.
[0033] Preferably, the α parameter in S02 controls the sharpness of the document topic distribution, and the α value is adjusted and set as a function of the document length.
[0034] Preferably, the adjustment of the α value is performed by cross-validation, the LDA model is trained under different α values, and the perplexity is used as a performance indicator.
[0035] (III) Beneficial effects
[0036] Compared with the prior art, the present invention provides intelligent text feature extraction and analysis hardware, which has the following beneficial effects:
[0037] 1. The intelligent text feature extraction and analysis hardware, through the set hierarchical model, performs local feature extraction at the bottom layer, controls the N value in the range of 1 to 3 to reduce data sparsity, and focuses on capturing local grammatical and semantic features. The middle layer performs context expansion, using a larger N value than the bottom layer or combining multiple N-gram models to expand the context range and provide richer context information. The high layer performs global feature extraction and uses the MSDFN model to capture long-distance dependencies and global features. Information is extracted in layers, so that the model can understand the text more comprehensively, thereby improving the comprehensiveness and accuracy of information extraction. At the same time, the hierarchical model has high flexibility and adaptability. By adjusting the parameters and model structures at different levels, it can be optimized for different tasks and data sets to achieve the best information extraction effect.
[0038] 2. The intelligent text feature extraction and analysis hardware, through the setting of the MSDFN model, can consider the features of multiple scales at the same time, which helps to capture information of different granularities. The MSDFN model can capture broader and deeper features, which makes the model have stronger generalization ability when facing unseen data. At the same time, the MSDFN model can be better integrated with the N-gram model. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a system structure diagram of the intelligent text feature extraction and analysis hardware of the present invention;
[0040] Figure 2 A flowchart of a hierarchical model for constructing the intelligent text feature extraction and analysis hardware of the present invention;
[0041] Figure 3 This is a data analysis flow chart of the intelligent text feature extraction and analysis hardware of the present invention. DETAILED DESCRIPTION
[0042] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further described below in conjunction with the accompanying drawings and implementation methods.
[0043] Please refer to Figure 1 , Figure 2 as well as Figure 3 ,in, Figure 1 This is a system structure diagram of the intelligent text feature extraction and analysis hardware of the present invention; Figure 2 A flowchart of a hierarchical model for constructing the intelligent text feature extraction and analysis hardware of the present invention; Figure 3This is a data analysis flow chart of the intelligent text feature extraction and analysis hardware of the present invention.
[0044] Embodiment 1
[0045] In the specific implementation process, Figure 1 As shown, the intelligent text feature extraction and analysis hardware provided by the present invention includes a collection module for collecting text data, extracting text content from a data source, and cleaning and storing the extracted text;
[0046] A data processing module is used to extract grammatical and semantic features from text data;
[0047] A feature analysis module, used for analyzing feature information according to the grammatical and semantic features extracted by the data processing module;
[0048] like Figure 2 As shown, the intelligent text feature extraction and analysis hardware provided by the present invention, the specific data processing module performs grammatical and semantic feature extraction including the following steps:
[0049] S1. Building the bottom layer: Using N-gram to capture local grammatical and semantic features;
[0050] S101. Word segmentation: split the text into character sequences;
[0051] S102.N-gram generation: Generate 1-gram from the segmented text to capture local grammatical and semantic information;
[0052] S103. Feature extraction: extract features from 1-gram;
[0053] S2. Build the middle layer: Combine the N-gram model to expand the context range;
[0054] S201. Multi-valued N-gram: Use the N-gram model with an N value of 3 to expand the context range;
[0055] S201. Context modeling: Understanding broader semantic and grammatical structures by analyzing word combinations;
[0056] S203. Feature integration: Combine the features extracted from the middle layer with the bottom layer features to model richer contextual information;
[0057] S3. Build high-level layers: Capture global and long-range features based on the MSDFN model;
[0058] S301.MSDFN architecture;
[0059] S302. Long-distance dependency identification: Use deep learning technology (Transformer) to identify and model long-distance dependencies;
[0060] S303. Global feature extraction: extracting global features that reflect the entire text structure and theme;
[0061] S4. Feature fusion: Features from different levels are compared and fused on the same dimension through feature concatenation;
[0062] S5. Smooth the data after feature fusion.
[0063] The data processing module also includes preprocessing of the text before extracting grammatical and semantic features. Text preprocessing includes word segmentation, building a vocabulary, text indexing, and serialization. It divides the text string into smaller units, creates a vocabulary, maps all unique words or characters to a unique index, uses the vocabulary to replace each word or character in the text with its corresponding index, and converts the indexed text into a numerical sequence.
[0064] The MSDFN model includes a time series decomposition module, a seasonal component prediction module, and a trend component prediction module;
[0065] The MSDFN model is based on Transformer and sets three latest long sequences to embed the time series decomposition module to predict the seasonal component and the placeholder of the prediction sequence. The three latest long sequences are scalar, timestamp information and location information respectively.
[0066] The time series decomposition module eliminates periodic fluctuations through sliding average to highlight long-term trends, and the seasonal component prediction module consists of a time series encoding layer and a multi-scale seasonal feature capture layer;
[0067] X emb Yes X s and X0 to obtain X emb Input time series encoding layer, X emb as follows:
[0068] X emb =Concat(X s ,X0);
[0069] Where X emb ∈R (I+O)×d , X S ∈R I×d ,X0∈R O×d is a placeholder for the prediction sequence, set to 0;
[0070] X (l-1,msfn) =sum(FC(X emb ),DE,PE),if and only if l=1,
[0071] X(l,msfn )=MSFN(X (l-1,SFC) ),L∈{1,2,…,M}
[0072] Y s =Slice(Projection(X (l,SFC) ),if and only if l=M
[0073] Among them, X (l-1,msfn) ,X( l,msfn )∈R (I+O)×D ,Y s ∈R O×d ;X (l-1,msfn) (l=1) for X emb The three-part embedding of X (l,SFC) )(l=M) represents the hidden state after the multi-layer MSFN structure; Y s Indicates that X( l,msfn The linear projection of )(l=M) and the final seasonal component prediction after slicing operation;
[0074] The time series decomposition module is used to decompose the input sequence to obtain trend components. The pre-decomposed trend components are modeled and predicted separately. For the trend component X obtained by TSDecomp pre-decomposition t ∈R I×d , where I is the length of the time series and d is the dimension of the variable, the process is as follows:
[0075] Y (MLP,t) =FC(Relu(FC(X t )));
[0076] The fully connected layer takes the input X t Mapped to the hidden layer, the ReLU activation function is applied to increase the nonlinearity of the model and accelerate convergence. The ReLU function is defined as f(x) = max(0,x). Another fully connected layer extracts features and patterns. The last fully connected layer converts the output of the hidden layer into the final prediction Y(MLP,t) of the model.
[0077] The average value of the original sequence is used as the forecast of the trend component, denoted as Y (MEAN,t) ∈R O×d , the process is as follows:
[0078] Y (MEAN,t) = MEAN(X I ).
[0079] The time series decomposition module, for the original input sequence X I , the processing process is as follows:
[0080] Xt =Softmax(L(X I ))*(F(X I ));
[0081] X S =X I -X t ;
[0082] Where X s , X t ∈R I ×d is the seasonal component and trend component obtained by MOEDecomp. The Softmax function is used to convert the linear output into a probability distribution. I ) represents the extraction of seasonal trends, F(X I ) represents the decomposition of seasonal component and trend component.
[0083] When the step S4 performs feature fusion, the features extracted by the N-gram model, the method of expanding the context range, and the different layers of the MSDFN model are captured and spliced and fused on the output of the position encoding layer in the Transformer model.
[0084] For the data after feature fusion, the high-order model and the low-order model are linearly combined, and the high-order and low-order language models are used to assign the high-order probability information to the low-order model.
[0085] The smoothing process in step S5 includes using an interpolation method to perform smoothing, determining missing values in the data or abnormal fluctuations that need to be smoothed, and using spline interpolation to perform calculation and filling.
[0086] like Figure 3 As shown, the intelligent text feature extraction and analysis hardware analysis specifically uses the LDA model for data analysis. The specific analysis steps are as follows:
[0087] S01. Select LDA parameters: Determine the hyperparameters of Dirichlet distribution, document-topic distribution α and vocabulary-subject distribution β, based on the number of discovered topics K. The β parameter controls the distribution of vocabulary among topics. β is set to a constant or a function of the vocabulary size.
[0088] S02. Conduct topic explanation: Check the first 2-5 words under each topic and explain the meaning of each topic;
[0089] S03. Result interpretation and visualization: Use pyLDAvis to display the results of the topic model to help better understand the topic distribution;
[0090] The α parameter in S02 controls the sharpness of the document topic distribution, and the α value is adjusted and set as a function of the document length;
[0091] Adjustment of α value,adjust α value through cross validation, train LDA model under different α values, and use perplexity as performance indicator.
[0092] Embodiment 2
[0093] Intelligent text feature extraction and analysis hardware, including a collection module, for collecting text data;
[0094] A data processing module is used to extract grammatical and semantic features from text data;
[0095] A feature analysis module, used for analyzing feature information according to the grammatical and semantic features extracted by the data processing module;
[0096] The specific data processing module performs grammatical and semantic feature extraction, which includes the following steps:
[0097] S1. Building the bottom layer: Using N-gram to capture local grammatical and semantic features;
[0098] S101. Word segmentation: split the text into character sequences;
[0099] S102.N-gram generation: Generate 2-grams from the segmented text to capture local grammatical and semantic information;
[0100] S103. Feature extraction: extract features from 2-gram;
[0101] S2. Build the middle layer: Combine the N-gram model to expand the context range;
[0102] S201. Multi-valued N-gram: Use the N-gram model with an N value of 4 to expand the context range;
[0103] S201. Context modeling: Understanding broader semantic and grammatical structures by analyzing word combinations;
[0104] S203. Feature integration: Combine the features extracted from the middle layer with the bottom layer features to model richer contextual information;
[0105] S3. Build high-level layers: Capture global and long-range features based on the MSDFN model;
[0106] S301.MSDFN architecture;
[0107] S302. Long-distance dependency identification: Use deep learning technology (Transformer) to identify and model long-distance dependencies;
[0108] S303. Global feature extraction: extracting global features that reflect the entire text structure and theme;
[0109] S4. Feature fusion: Features from different levels are compared and fused on the same dimension through feature concatenation;
[0110] S5. Smooth the data after feature fusion.
[0111] Comparative Example 1
[0112] Intelligent text feature extraction and analysis hardware, including a collection module, for collecting text data;
[0113] A data processing module is used to extract grammatical and semantic features from text data;
[0114] A feature analysis module, used for analyzing feature information according to the grammatical and semantic features extracted by the data processing module;
[0115] The specific data processing module performs grammatical and semantic feature extraction, which includes the following steps:
[0116] S1. Extract features from data using 1-gram;
[0117] S2. Smooth the data after feature fusion.
[0118] Comparative Example 2
[0119] Intelligent text feature extraction and analysis hardware, including a collection module, for collecting text data;
[0120] A data processing module is used to extract grammatical and semantic features from text data;
[0121] A feature analysis module, used for analyzing feature information according to the grammatical and semantic features extracted by the data processing module;
[0122] The specific data processing module performs grammatical and semantic feature extraction, which includes the following steps:
[0123] S1. Extract features from data using 4-gram;
[0124] S2. Smooth the data after feature fusion.
[0125] The following experiments were conducted to study the speed of feature extraction in the data processing of the above-mentioned embodiments 1-2 and comparative example 1. Based on the intelligent text feature extraction and analysis hardware, a text dataset with 10,000 different words and a text dataset with 20,000 different words were extracted. The results are shown in the following table:
[0126]
[0127]
[0128] From the data in the table, it can be seen from the comparison between Example 1 and Example 2 that Example 1 and Example 2 take into account both the extraction speed and the text accuracy when extracting data;
[0129] Compared with the comparative example 1, the perplexity of the embodiment 1 and the embodiment 2 is lower than that of the comparative example 1, and the extracted text is more accurate. When the extraction time is extended by 0.26 to 0.39, the perplexity is reduced by 0.53 to 0.64, which consumes less time and greatly reduces the perplexity of text extraction.
[0130] Compared with Comparative Example 2, the perplexity of Example 1 and Example 2 is lower than that of Comparative Example 1, and the time consumed for extracting text is shorter. When the perplexity increases by 0.12 to 0.25, the extraction time is shortened by 0.62 to 0.67, saving a lot of time, while the perplexity of text extraction is only slightly improved.
[0131] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any figure mark in the claims should not be regarded as limiting the claims involved.
[0132] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. Intelligent text feature extraction and analysis hardware, characterized by: It includes a collection module for collecting text data; A data processing module is used to extract grammatical and semantic features from text data; A feature analysis module, used for analyzing feature information according to the grammatical and semantic features extracted by the data processing module; The data processing module performs grammatical and semantic feature extraction including: S1. Building the bottom layer: Using N-gram to capture local grammatical and semantic features; S2. Build the middle layer: Combine the N-gram model to expand the context range; S3. Build high-level layers: Capture global and long-range features based on the MSDFN model; S4. Feature fusion: Fusion of features extracted at different levels; S5. Smooth the data after feature fusion.
2. The intelligent text feature extraction and analysis hardware according to claim 1, characterized in that: The data processing module also includes preprocessing the text before extracting grammatical and semantic features. The text preprocessing includes word segmentation, building a vocabulary, text indexing, and serialization. The text string is divided into smaller units, a vocabulary is created, all unique words or characters are mapped to a unique index, each word or character in the text is replaced with its corresponding index using the vocabulary, and the indexed text is converted into a numerical sequence.
3. The intelligent text feature extraction and analysis hardware according to claim 2, characterized in that: The MSDFN model includes a time series decomposition module, a seasonal component prediction module and a trend component prediction module; The MSDFN model sets three latest long sequences based on Transformer to embed the time series decomposition module to predict the seasonal component and the placeholder of the prediction sequence, wherein the three latest long sequences are scalar, timestamp information and location information respectively; The time series decomposition module eliminates periodic fluctuations through sliding average to highlight long-term trends, and the seasonal component prediction module consists of a time series encoding layer and a multi-scale seasonal feature capture layer; The input sequence is decomposed using the time series decomposition module to obtain trend components, and the pre-decomposed trend components are modeled and predicted separately.
4. The intelligent text feature extraction and analysis hardware according to claim 3, characterized in that: The time series decomposition module, for the original input sequence X I , the processing process is as follows: X t =Softmax(L(X I ))*(F(X I )); X S =X I -X t ; Where X s , X t ∈R I ×d is the seasonal component and trend component obtained by MOEDecomp. The Softmax function is used to convert the linear output into a probability distribution. I ) represents the extraction of seasonal trends, F(X I ) represents the decomposition of seasonal component and trend component.
5. The intelligent text feature extraction and analysis hardware according to claim 4, characterized in that: When the step S4 performs feature fusion, the features extracted by the N-gram model, the method of expanding the context range, and the different layers of the MSDFN model are captured and spliced and fused on the output of the position encoding layer in the Transformer model.
6. The intelligent text feature extraction and analysis hardware according to claim 5, characterized in that: The data after feature fusion is processed, and the high-order probability information is assigned to the low-order model using the linear combination of the high-order model and the low-order model.
7. The intelligent text feature extraction and analysis hardware according to claim 6, characterized in that: The smoothing process in step S5 includes using an interpolation method to perform smoothing, determining missing values in the data or abnormal fluctuations that need to be smoothed, and using spline interpolation to perform calculation and filling.
8. The intelligent text feature extraction and analysis hardware according to claim 7, characterized in that: The feature analysis module is used to analyze the feature information based on the grammatical and semantic features extracted by the data processing module using the LDA model for data analysis. The specific analysis steps are as follows: S01. Select LDA parameters: Determine the hyperparameters of Dirichlet distribution, document-topic distribution α and vocabulary-subject distribution β, based on the number of discovered topics K. The β parameter controls the distribution of vocabulary among topics. β is set to a constant or a function of the vocabulary size. S02. Conduct topic explanation: Check the first 2-5 words under each topic and explain the meaning of each topic; S03. Result interpretation and visualization: Use pyLDAvis to display the results of the topic model to help better understand the topic distribution.
9. The intelligent text feature extraction and analysis hardware according to claim 8, characterized in that: The α parameter in S02 controls the sharpness of the document topic distribution, and the α value is adjusted and set as a function of the document length.
10. The intelligent text feature extraction and analysis hardware according to claim 9, characterized in that: The adjustment of the α value is performed through cross-validation, the LDA model is trained under different α values, and the perplexity is used as a performance indicator.