Micro-learning service information extraction method based on deep sequence labeling model

Abstract

The application provides a micro-learning service information extraction method based on a deep sequence labeling model, in the micro-learning service information extraction process, a deep sequence labeling model is used: first, an embedding layer is used to map high-dimensional sparse original data into low-dimensional continuous but dense data representation, and semantic information is extracted; after the embedding layer, two continuous CNN layers are used to further mine and summarize the local features of adjacent inputs in the model; an RNN layer is further arranged after the embedding layer, which is used for extracting and modeling time sequence features; a fusion block is arranged after the CNN and the RNN, which is used for combining different types of potential features; a CRF layer is arranged before a final output layer, which is used for the local constraint of an output sequence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, micro-learning, and in particular to a micro-learning service information extraction method based on a deep sequence labeling model. BACKGROUND

[0002] In the sequence labeling model of micro-learning, information flow is input into the model in chronological order, and the output is a sequence of relevant labels of data elements, each label can be the characteristic or category of each data element. Several representative methods about sequence labeling include:

[0003] A. Hidden Markov Model and Conditional Random Field Hidden Markov Model, Conditional Random Field and its variants are widely used in many studies to model various time series and temporal problems. These models perform well in modeling local features and constraints. However, this modeling strategy is based on linear information and heavily relies on manual features, which makes them lack the ability to model complex labeling problems in more general cases (such as online informal learning).

[0004] B. Recurrent Neural Network

[0005] Unlike other types of neural networks, recurrent neural networks show outstanding ability in simulating the time dynamic behavior of the target problem. It can remember historical information and combine these historical information with the current received information, and then make predictions. Long Short-Term Memory Network (LSTM) as a representative class of RNN, shows satisfactory performance in modeling short-term memory and long-term memory. In addition, Bidirectional Long Short-Term Memory (Bi-LSTM) can further utilize "future" input information to model sequence patterns. Therefore, it can not only mine forward sequence information, but also mine backward sequence information. However, this type of model performs poorly in modeling local constraints, especially when adjacent outputs in the sequence have a strong influence on each other.

[0006] C. Combination of RNN and CRF

[0007] Based on the advantages and disadvantages of various sequence modeling strategies, LSTM-CRF and its variants are the latest solutions to handle sequence modeling problems. This network structure has the ability to effectively utilize past input features through LSTM layers and effectively utilize sequence-level local information through CRF layers. Due to the advantages of Bi-LSTM, the bidirectional LSTM-CRF model is used in many studies for sequence labeling or tagging tasks. In many cases, it performs better than the LSTM-CRF-based model. However, this type of method cannot well cope with the complex data characteristics in the education field, and lacks universality. SUMMARY

[0008] In summary, in order to solve the problems that the Hidden Markov Model and the Conditional Random Fields Hidden Markov Model are seriously dependent on manual feature extraction and the local constraint modeling of the mainstream recurrent neural network is not good, the application proposes a new model based on LSTM-CRF for micro-learning services, and gives a micro-learning service information extraction method based on a deep sequence labeling model.

[0009] Specifically, the following technical solutions are adopted:

[0010] A micro-learning service information extraction method based on a deep sequence labeling model, in the micro-learning service information extraction process, a deep sequence labeling model is used:

[0011] First, an embedding layer is used to map high-dimensional sparse original data to low-dimensional continuous but dense data representation, and semantic information is extracted;

[0012] After the embedding layer, two consecutive CNN layers are used to further mine and summarize the local features of adjacent inputs in the model;

[0013] An RNN layer is also provided after the embedding layer for extracting and modeling time series features;

[0014] A fusion block is provided after the CNN and RNN for combining different types of potential features;

[0015] A CRF layer is provided before the final output layer for local constraints on the output sequence.

[0016] Further, a fixed-size sliding window is used in each CNN layer to extract potential features of adjacent embedded inputs; let E i is the i th embedded vector generated by the embedding layer, in each iteration, the information of adjacent input embedded vectors is summarized through the sliding window.

[0017] Further, the RNN layer adopts a Bi-LSTM structure, and the embedded information is sent to the Bi-LSTM layer in two ways of order and reverse order; then, for each time step, the Bi-LSTM layer will output the prediction based on the information of the "past" and "future" at the current time.

[0018] Further, the fusion block first merges the potential features extracted by the CNN layer and the RNN layer together through a concatenation operation; then uses multiple nonlinear transformation layers to further combine the potential features into fine-grained features.

[0019] Further, the fusion block uses a Bi-LSTM layer and a fully connected neural network to realize nonlinear conversion.

[0020] Further, the calculation process of the fusion block is shown in equations (1) to (5):

[0021] x t =concat[x CNN ,x RNN ] (1)

[0022]

[0023]

[0024] Among them, x t is the concatenated vector information, which includes the modeling results of the CNN layer and the Bi-LSTM layer, denoted as x CNN and x RNN ; and The spliced ​​information is fused and modeled from different time series directions, W and b are the corresponding weights and bias terms, y t is the preliminary result after time series fusion; in order to characterize the information of the fusion result in the specified dimension, y is further fused through multiple fully connected layers after time series fusion. t Perform nonlinear changes and finally output y.

[0025] Furthermore, the implementation process includes the following steps:

[0026] S1: Data input: including data streams from different fields;

[0027] S2: Data preprocessing, including:

[0028] 1) Data cleaning: remove missing values ​​and abnormal data;

[0029] 2) Feature extraction;

[0030] S3: Model training, including:

[0031] 1) Use input data to train the model;

[0032] 2) Adjust model parameters;

[0033] 3) Test and evaluate model performance and iterate training;

[0034] S4: Model deployment: Deploy the trained model to specific application scenarios to assist in information extraction.

[0035] And, a micro-learning service information extraction device based on a deep sequence annotation model is based on a computer system and uses the micro-learning service information extraction method based on a deep sequence annotation model as described above during operation.

[0036] A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: during operation, the micro-learning service information extraction method based on the deep sequence annotation model as described above is used.

[0037] Compared to existing technologies, the model provided by the present invention and its preferred embodiment is highly robust: the proposed deep sequence labeling model significantly outperforms current mainstream methods (pure CRF model and pure Bi-LSTM model). The model's classification capability is approximately 2% higher than that of the mainstream Bi-LSTM-CRF model. For different types of information, the proposed model demonstrates better performance than any other model. This result demonstrates that the model of the present invention is highly robust and can accurately locate and classify different types of information with different distributions.

[0038] The proposed model boasts higher training efficiency: Compared to the mainstream sequence labeling model, Bi-LSTM-CRF, the addition of CNN layers and fusion blocks improves accuracy without significantly decreasing model training efficiency. Furthermore, the difference in training efficiency between the proposed model with and without the fusion block is minimal. Therefore, the proposed model does not require additional training steps to reach optimal performance, resulting in higher training efficiency.

[0039] The proposed deep sequence annotation model helps effectively and accurately mine and extract valuable information from massive and redundant data, making it an important preprocessing step for personalized online services. The proposed model is versatile and combines the advantages of different types of classical neural networks. It achieves impressive performance compared to traditional information extraction methods, such as conditional random fields and bidirectional recurrent neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0041] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0042] Figure 2 This is a structural diagram of a model according to an embodiment of the present invention;

[0043] Figure 3 This is a sliding window strategy diagram according to an embodiment of the present invention;

[0044] Figure 4 This is a flowchart of the Bi-LSTM process according to an embodiment of the present invention;

[0045] Figure 5 This is a structural diagram of a fusion block according to an embodiment of the present invention;

[0046] Figure 6 This is a CRF network structure diagram of an embodiment of the present invention; DETAILED DESCRIPTION

[0047] To make the features and advantages of this patent more clearly understood, the following embodiments are specifically described in detail with reference to the accompanying drawings:

[0048] The main process of the method of the present invention is as follows Figure 1 As shown, the model structure is as Figure 2 The model shown contains four layers and one block.

[0049] A. Embedding layer — semantic modeling & dimensionality reduction

[0050] Embedding technology is an effective dimensionality reduction and feature representation method that transfers points in a high-dimensional space to a low-dimensional space while approximately maintaining the pairwise distances between the points. This technology has shown satisfactory performance in reducing data and model complexity and has been used in various machine learning-related tasks such as information retrieval, multimedia data processing, and data mining. For sequence signal processing, especially in natural language processing (NLP), the embedding layer also helps to model the semantic information of the original input data. In the model proposed in the present invention, the embedding layer is used to map the high-dimensional sparse original data into a low-dimensional continuous but dense data representation and extract the semantic information.

[0051] B.CNN layer - latent feature extraction

[0052] Due to the excellent performance of CNN in extracting latent features, it is not only widely used in the research field of computer vision, but also widely used in many other fields such as NLP. Many previous models mainly use CNN to extract and model character-level semantic information, such as prefixes and suffixes. However, not all types of information contain character-level semantic information. Therefore, in order to build a more general model for information extraction tasks, after the embedding layer, the present invention uses two consecutive CNN layers to further mine and summarize the local features of adjacent inputs (object level) in the model. The utility of this CNN layer is very different from existing studies. The present invention assumes that such a CNN layer can capture different types of information, thereby improving the performance of Bi-LSTM-CRF. In computer vision, the sliding window strategy is widely used to limit the amount of information involved in each summary process. As Figure 3 As shown in the model proposed in this invention, a fixed-size sliding window is used in each CNN layer to extract the potential features of adjacent embedded inputs, E i is the i-th embedding vector generated by the embedding layer, which summarizes the information of adjacent input embedding vectors. The rectangle is a sliding window moving from left to right, and in each iteration, the information within the window will be summarized.

[0053] C. Bi-LSTM layer — sequence labeling

[0054] In the model proposed in this invention, an RNN layer is added after the embedding layer as a core component. The purpose of this layer is to extract and model time series features. For a typical information extraction task, it can be assumed that both past and future inputs can provide valuable information in identifying and locating target information. Therefore, in the model of this invention, the RNN layer uses a Bi-LSTM structure. The workflow of Bi-LSTM is as follows Figure 4 As shown in Figure 2, the embedded information is fed into the Bi-LSTM layer in both sequential and reverse order. Then, for each time step, the Bi-LSTM layer outputs a prediction based on the "past" and "future" information at the current moment.

[0055] D. Fusion block - used to merge different types of latent features

[0056] Appropriate fusion of different types of features can make information representation more reliable and accurate. The present invention adds a fusion block after the CNN and Bi-LSTM components, which is designed to better combine these different types of potential features. In the fusion block of this model, the potential features extracted by the CNN layer and the Bi-LSTM layer are first merged together through a series operation. Then multiple nonlinear transformation layers are used to further combine the potential features into fine-grained features. Such nonlinear transformation layers can be personalized according to the complexity of the problem domain and the input data. The present invention uses another Bi-LSTM layer and a fully connected neural network to simulate this nonlinear transformation. The structural details of this fusion block are as follows: Figure 5 The specific calculation process is shown in formulas (1) to (5):

[0057] x t =concat[x CNN ,x RNN ] (1)

[0058]

[0059]

[0060] Among them, x t is the concatenated vector information, which includes the modeling results of the CNN layer and the Bi-LSTM layer, denoted as x CNN and x RNN . and The spliced ​​information is fused and modeled from different time series directions, W and b are the corresponding weights and bias terms, y t is the preliminary result after time series fusion. In order to characterize the information of the fusion result in the specified dimension, y is further fused through multiple fully connected layers after time series fusion. t Perform nonlinear changes and finally output y.

[0061] E. Adding Local Constrained CRF Layers to Sequence Models

[0062] As mentioned above, the pure RNN model has its own shortcomings in modeling local constraints. Therefore, this solution adds a CRF layer before the final output layer of the entire model. In the CRF model, for a given sequence x, the probability of output y can be simply expressed as formula (6). It can be easily observed from the formula that y i and y i-1 They influence each other, which indicates that the probability value can take into account the correlation between adjacent outputs. The CRF network structure is as follows Figure 6 Here, the prediction of the second output Y2 is not only determined by the second input X2, but also affected by the first output Y1. Function t(Y i-1 ,Y i ) and s(Y i-1 ,X i ) to model state transition and forgetting respectively. The CRF layer is designed to add more local constraints, especially the local constraints of the output sequence, which cannot be captured by the previous embedding layer, Bi-LSTM layer and CNN layer.

[0063]

[0064] From the above, it can be seen that the embodiment of the present invention adopts a brand-new solution in the feature extraction part. By fusing different models, it deeply captures the various patterns and complex relationships in the data, thereby achieving more comprehensive data understanding and information capture. Specifically, the Bi-LSTM layer is used to bidirectionally extract the time-varying patterns of the input sequence, while the convolutional neural network layer extracts potential features in the local area from the input data. Subsequently, the outputs of the CNN layer and the Bi-LSTM layer are jointly fed into the fusion block. In the fusion block, a combination of multiple linear and nonlinear transformation layers is designed. Through this combination of linear and nonlinear models, the extracted information can be more effectively fused, which can greatly avoid the information loss caused by the information fusion process compared to existing methods. This innovative solution not only improves the accuracy of feature extraction, but also enhances the robustness and generalization ability of the model, providing a more ideal solution for complex pattern data processing.

[0065] Through extensive experiments, this example compares the information recognition and extraction capabilities of this method with a number of existing solutions (such as CRF, LSTM, Bi-LSTM, Bi-LSTM-CRF, and several variants), evaluating multiple metrics including recall, precision, F1 value (F1), and the total number of positive and negative examples (TP+TN). The results show that the proposed model exhibits the best performance, with a recall rate of 0.8223, a precision rate of 0.8188, and an F1 value of 0.8206, significantly outperforming traditional CRF, LSTM, Bi-LSTM, and their combined model Bi-LSTM-CRF. The model performs well across all metrics, especially in terms of accuracy and balance, demonstrating its strong comprehensive capabilities for information extraction tasks.

[0066] Based on the above model design, its usage process includes:

[0067] 1. Data input: data streams from different fields

[0068] 2. Data Preprocessing

[0069] 1) Data cleaning: removing missing values ​​and abnormal data

[0070] 2) Feature extraction: Extract useful features from the raw data, such as document type, field, and suitable population.

[0071] 3. Model Training

[0072] 1) Use input data to train this model

[0073] 2) Adjust model parameters such as learning rate, batch size, training rounds, etc.

[0074] 3) Test and evaluate model performance and iterate training

[0075] 4. Model deployment: Deploy the trained model to specific application scenarios to assist in information extraction.

[0076] The application scenarios that can be realized are as follows:

[0077] a. Automatically generate learning content:

[0078] The present invention can more accurately and automatically extract key information and knowledge points from a large number of documents, web pages and other text resources. This information can be sorted and organized into concise and easy-to-understand micro-learning units for learners to quickly learn.

[0079] b. Update learning materials in real time:

[0080] The present invention can extract important information from the latest scientific research papers, news reports, and industry reports in real time, ensuring that micro-learning content is always up-to-date and relevant. This is particularly important in rapidly changing fields such as technology and business.

[0081] c. Knowledge Management and Retrieval:

[0082] The information extraction technology of the present invention can help build a knowledge base, making it convenient for learners to quickly retrieve relevant information and knowledge points when needed, thereby improving learning efficiency.

[0083] In summary, the application of information extraction technology in micro-learning can not only greatly improve the production efficiency of learning content, but also provide personalized and real-time updated learning experience, thereby meeting the needs of modern learners for efficient, flexible and interactive learning.

[0084] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0089] This patent is not limited to the above-mentioned best implementation mode. Anyone can derive various other forms of micro-learning service information extraction methods based on deep sequence annotation models under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of this invention should fall within the scope of this patent.

Claims

1. A micro-learning service information extraction method based on a deep sequence annotation model, characterized by: In the process of extracting micro-learning service information, a deep sequence annotation model is used: First, an embedding layer is used to map the high-dimensional sparse raw data into a low-dimensional continuous but dense data representation and extract semantic information. After the embedding layer, two consecutive CNN layers are used to further mine and summarize the local features of adjacent inputs in the model; An RNN layer is also set after the embedding layer to extract and model time series features; A fusion block is set after CNN and RNN to combine different types of latent features; A CRF layer is set before the final output layer Used for local constraints on the output sequence; The RNN layer adopts a Bi-LSTM structure, and the embedded information is sent to the Bi-LSTM layer in both sequential and reverse order; Then, for each time step, the Bi-LSTM layer will output a prediction based on the "past" and "future" information of the current moment; The fusion block first merges the latent features extracted by the CNN layer and the RNN layer through a series operation; then uses multiple nonlinear transformation layers to further combine the latent features into fine-grained features; The fusion block uses a Bi-LSTM layer and a fully connected neural network to achieve nonlinear transformation; The calculation process of the fusion block is shown in equations (1) to (5): = concat [ ] (1) （2） （3） （4） （5） in, is the concatenated vector information, which includes the modeling results of the CNN layer and the Bi-LSTM layer, recorded as and ; and The spliced ​​information is fused and modeled from different time series directions, W and are the corresponding weights and bias terms, is the preliminary result after time series fusion; in order to characterize the information of the fusion result in the specified dimension, multiple fully connected layers are used to further Perform nonlinear changes and finally output .

2. The micro-learning service information extraction method based on the deep sequence annotation model according to claim 1 is characterized by: A fixed-size sliding window is used in each CNN layer to extract latent features of adjacent embedded inputs; set up is the i-th embedding vector generated by the embedding layer. In each iteration, the information of adjacent input embedding vectors is summarized through a sliding window.

3. The micro-learning service information extraction method based on the deep sequence annotation model according to claim 1 or 2 is characterized in that: The implementation process includes the following steps: S1: Data input: including data streams from different fields; S2: Data preprocessing, including: 1) Data cleaning: remove missing values ​​and abnormal data; 2) Feature extraction; S3: Model training, including: 1) Use input data to train the model; 2) Adjust model parameters; 3) Test and evaluate model performance and iterate training; S4: Model deployment: Deploy the trained model to specific application scenarios to assist in information extraction.

4. A micro-learning service information extraction device based on a deep sequence annotation model, characterized by: Based on a computer system, the micro-learning service information extraction method based on the deep sequence annotation model as described in any one of claims 1 to 3 is used during operation.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: During operation, the micro-learning service information extraction method based on the deep sequence annotation model as described in any one of claims 1 to 3 is used.

Images