A method for extracting intelligence tags using heterogeneous BERT and semi-supervised SVM models

Abstract

This invention discloses a method for extracting intelligence tags using heterogeneous BERT and semi-supervised SVM models, comprising: acquiring the original unstructured intelligence text dataset and performing text preprocessing and data augmentation operations on it; extracting topic words from the intelligence text based on BTM short text and LDA long text topic models to obtain topic word vectors of the intelligence text; representing the intelligence document with sentence vectors based on the Doc2Vec model; inputting the topic word vectors and sentence vectors into the BERT model for pre-training to obtain feature vectors with topic information; training and parameter tuning the semi-supervised SVM model using a small amount of labeled data and a large amount of unlabeled data; and inputting the feature vectors with topic information into the semi-supervised SVM binary classification algorithm model for training to obtain the data tag extraction results. This invention can effectively extract data tags from intelligence text, make full use of unlabeled data information, and improve the accuracy and generalization ability of the data tag extraction system.

Description

Technical Field

[0001] This invention relates to the field of tag extraction technology, specifically to a method for extracting intelligence tags using heterogeneous BERT and semi-supervised SVM models. Background Technology

[0002] In the field of intelligence analysis, tag extraction technology is a crucial task. It aims to automatically extract key information and tags from large amounts of text data to help analysts quickly and accurately understand and process intelligence content. Traditional tag extraction methods typically rely on manual rules or statistical methods. These methods are often limited by the accuracy of the rules and the generalization ability of the statistical models, making them difficult to handle large-scale and complex text data.

[0003] In recent years, with the development of deep learning and natural language processing technologies, neural network-based label extraction techniques have gradually become mainstream, including methods based on pre-trained language models, such as the BERT model. The BERT model can learn semantic representations of large amounts of text data and has strong contextual understanding capabilities, thus achieving significant results in label extraction tasks. However, due to the complexity and diversity of intelligence data, a single model often struggles to meet the needs of various intelligence analysis tasks. Therefore, this invention proposes an intelligence label extraction method based on heterogeneous BERT and semi-supervised SVM models, aiming to combine the rich semantic representations of the BERT model with the effective utilization of unlabeled data by the semi-supervised SVM model, thereby improving the accuracy and efficiency of intelligence label extraction. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligence label extraction method based on heterogeneous BERT and semi-supervised SVM models, which can make full use of unlabeled data in unstructured intelligence text data, enabling the system to automatically and efficiently complete label extraction, thereby improving the quality and efficiency of intelligence data system construction.

[0005] To achieve the above objectives, the specific solution of the present invention is as follows:

[0006] Step S101: Obtain the original unstructured intelligence text dataset, perform data cleaning, text segmentation, and stop word removal preprocessing operations to obtain a preprocessed intelligence text dataset, and then perform data augmentation through synonym replacement, random insertion, random swapping, random deletion, and reverse translation to obtain an augmented intelligence text dataset.

[0007] Step S102: Encode the intelligence text using the bag-of-words model to obtain the topic word encoding vector, input it into the BTM short text topic extraction model and the LDA long text topic extraction model, and use the Gibbs sampling method to obtain the topic distribution probability and the topic word distribution probability of each intelligence; the topic word distribution probabilities of all topic words form a topic word distribution matrix, set an appropriate threshold for the topic word distribution matrix, and filter out words that exceed the threshold as initial topic words;

[0008] Step S103: Input the intelligence text into the Doc2Vec model, use PV-DM technology to take context words and sentence vectors as input, use neural network training to predict target words, capture the semantic information of the intelligence text, and obtain the sentence vector representation of the text;

[0009] Step S104: Convert the encoding vectors of the initially selected topic words obtained from the topic extraction model in step S102. Compared with the sentence vector obtained by the Doc2Vec model in step S103 By concatenating the vectors, a merged vector is obtained. Where d k Let m be the feature dimension of the vector, m be the length of the initial selected topic word encoding vector, and n be the sentence vector encoding length. The merged vectors are input into the BERT model for masked language modeling pre-training to obtain topic word feature vectors containing intelligence topic information.

[0010] Step S105: Convert the topic word feature vector obtained in step 104 into a single vector. The input is fed into a semi-supervised SVM model, and semi-supervised training is performed based on a human interaction method.

[0011] Step S106: Using the topic word feature vector containing intelligence topic information obtained in step S104 as input, input it into the semi-supervised SVM model trained in step S105, and use the binary classification algorithm to perform probability prediction to obtain the label extraction result.

[0012] Furthermore, in step S101, the data cleaning includes removing special characters, converting case, handling missing values, handling outliers, and handling duplicate values.

[0013] Furthermore, in step S101, the synonym replacement involves replacing some words in the text with words that have the same or similar meanings to generate enhanced samples; the random insertion involves randomly adding new words to the text to generate enhanced samples; the random deletion involves randomly deleting some words from the text to generate enhanced samples; the random swapping involves randomly swapping the positions of two words in the text to generate enhanced samples; and the reverse translation involves translating the sample into another language using one translation software and then translating it back into the original language using another translation software to obtain enhanced samples with the same semantics but different sentences.

[0014] Furthermore, in step S102, the process of encoding the intelligence text using a bag-of-words model to obtain the topic word encoding vector specifically includes: using the word embedding method of the bag-of-words model, taking the preprocessed intelligence text and the enhanced intelligence text as input, inputting them into the bag-of-words model for encoding, predicting the target word through the words in the context, thereby learning the distributed representation of words, obtaining the topic word encoding vector of the intelligence text, and providing input for subsequent model training.

[0015] Furthermore, in step S102, the BTM short text topic extraction model establishes a "topic-vocabulary" matrix and uses Dirichlet hyperparameters α and β to describe the different vocabulary and topic distributions in the corpus; the LDA long text topic extraction model includes vocabulary, topics and documents, establishes a "topic-vocabulary" matrix and a "document-topic" matrix, and optimizes the "topic-vocabulary" matrix and "document-topic" matrix through Bayesian estimation and prior distribution.

[0016] Further, in step S102, obtaining the topic distribution probability and the word distribution probability of each piece of intelligence using the Gibbs sampling method specifically includes: using the Gibbs sampling method to estimate the parameters of the BTM short text topic extraction model and the LDA long text topic extraction model, and obtaining the topic distribution probability and the word distribution probability of each topic for each piece of intelligence; setting a threshold for the obtained word distribution matrix ZW, and selecting words in the word distribution matrix ZW that exceed the threshold as initial word selections. The word distribution matrix ZW for each piece of intelligence is represented as follows:

[0017]

[0018] Where z represents the probability of each topic. Let w represent the topic distribution of the m-th text, where the sum of the probabilities of each topic is 1; w represents the probability distribution of each word. This is used to represent the word distribution corresponding to the k-th topic, and the sum of the probability distributions of all words is 1.

[0019] Furthermore, in step S103, the preprocessed intelligence text and enhanced intelligence text obtained in step S101 are input into the Doc2Vec model. The document and words are embedded in a fixed vector space using PV-DM technology. The context words and sentence vectors are used as input. The target words are predicted by a neural network composed of fully connected layers to generate the Doc2Vec sentence vector of the intelligence document. The parameters of the neural network are adjusted by the backpropagation algorithm to ensure that the neural network can accurately predict the target words and generate accurate document vectors.

[0020] Furthermore, in step S104, the specific steps for inputting the merged vector into the BERT model for masked language modeling pre-training include:

[0021] The merged vector of each topic is used as a token in the BERT model, d k This indicates the number of tokens in the text, where "m+n" represents the feature dimensions included in the tokens.

[0022] Randomly select 15% of the tokens and replace them with the special marker "MASK" in the mask;

[0023] Predict words masked by "MASK" using bidirectional contextual information.

[0024] Furthermore, step S105 specifically includes the following steps:

[0025] The intelligence dataset is divided into unlabeled data and unlabeled data. Labels are added to all unlabeled data using manual annotation to obtain labeled data.

[0026] The topic word feature vectors obtained from the labeled data in step S104 are input into the semi-supervised SVM model for model training, and the model parameters are adjusted.

[0027] The feature vectors of the topic words corresponding to the unlabeled data are also input into the model for prediction to generate pseudo-labels for the unlabeled data;

[0028] A training dataset is constructed using both labeled and pseudo-labeled data. This dataset is then input into a semi-supervised SVM model for fine-tuning, optimizing the model parameters, and resulting in a fully trained semi-supervised SVM model.

[0029] Furthermore, in step S106, for the intelligence text from which tags are to be extracted, the topic word feature vector containing intelligence topic information is obtained through steps S102, S103, and S104. Construct set D:

[0030]

[0031] in, Let y be the vector of the i-th topic word, m be the number of topic words, and y be the vector of the i-th topic word. i ∈{0,1} represents the word vector label, y={y1,y2…,y m The initial value is 1. The semi-supervised SVM model trained in step S105 is used to perform probability prediction using a binary classification algorithm to obtain the label extraction result.

[0032] Compared with existing data tag extraction methods, the present invention has the following advantages:

[0033] 1. This invention employs a topic extraction model combining BTM (Browser Targeting) for short text and LDA (Long Text Analytical Analysis) for long text. This approach comprehensively considers the characteristics of both short and long texts, improving the accuracy and comprehensiveness of intelligence tag extraction. BTM is suitable for processing short text intelligence data, capturing the correlation information between word pairs, while LDA is suitable for processing long text intelligence data, better capturing word distribution and semantic information. By comprehensively considering the extraction results of both models, the diversity of tags can be enhanced, and the interpretability of the tag extraction results can be improved, helping users to more comprehensively understand the content and topic structure of the intelligence text data.

[0034] 2. This invention uses the Doc2Vec model to generate sentence vectors for intelligence texts. Combining sentence vectors with topic word vectors as input to the pre-trained model can integrate multi-level semantic information. By combining word-level and sentence-level features, it can improve the model's ability to understand intelligence texts and its contextual awareness, while also enhancing the model's generalization ability and its ability to handle rare words. This provides a more comprehensive and accurate semantic representation for intelligence analysis and mining.

[0035] 3. This invention uses the BERT model to pre-train topic word vectors and sentence vectors. It consists of a multi-layer Transformer encoder, with an internal structure including a self-attention mechanism and a feedforward neural network, enabling a comprehensive understanding of the semantic and contextual information of the text. Through bidirectional context modeling, BERT can effectively capture long-distance dependencies, and the parameters pre-trained on large-scale corpora have good generalization ability, improving the effectiveness and reliability of intelligence analysis.

[0036] 4. This invention takes into account unlabeled data in intelligence texts and uses a semi-supervised SVM model to perform secondary label screening. This can make full use of unlabeled data resources, improve model performance, reduce the risk of overfitting, reduce manual labeling costs, and enhance classification performance and generalization ability, thereby providing more effective support for intelligence label extraction systems. Attached Figure Description

[0037] Figure 1 A flowchart illustrating an information tag extraction method using heterogeneous BERT and semi-supervised SVM models provided in this embodiment of the invention;

[0038] Figure 2 A detailed flowchart of an information tag extraction method for heterogeneous BERT and semi-supervised SVM models provided in this embodiment of the invention;

[0039] Figure 3 A diagram illustrating a data preprocessing method provided in an embodiment of the present invention;

[0040] Figure 4 A diagram illustrating a data augmentation method provided in an embodiment of the present invention;

[0041] Figure 5 A CBOW encoding diagram provided for an embodiment of the present invention;

[0042] Figure 6 This is a diagram of the BTM topic extraction model provided in an embodiment of the present invention;

[0043] Figure 7 This is a diagram of the LDA topic extraction model provided in an embodiment of the present invention;

[0044] Figure 8 This is a sentence vector generation graph based on the Doc2Vec model provided in an embodiment of the present invention;

[0045] Figure 9 A diagram illustrating a semi-supervised SVM model training method provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the features and advantages of the present invention more apparent and understandable, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Figure 1 This is a flowchart of an information tag extraction method using heterogeneous BERT and semi-supervised SVM models according to an embodiment of the present invention. Figure 2 For a detailed flowchart, the method of the present invention includes the following steps:

[0048] Step S101: Obtain the raw unstructured intelligence text dataset, and perform text preprocessing and data augmentation;

[0049] Step S102: Extracting intelligence topic terms based on the BTM short text and LDA long text topic extraction models;

[0050] Step S103: Sentence vector representation of intelligence documents based on the Doc2Vec model;

[0051] Step S104: Perform BERT-based model pre-training on topic word vectors and sentence vectors;

[0052] Step S105: Training a semi-supervised SVM model based on human-computer interaction;

[0053] Step S106: Label extraction based on semi-supervised SVM binary classification algorithm.

[0054] In a specific embodiment of the present invention, step S101 specifically includes:

[0055] Obtain the raw unstructured intelligence text dataset and perform text preprocessing, such as... Figure 3 As shown, it includes the following sub-steps:

[0056] Step 1: Data cleaning, which involves removing special characters, converting case, handling missing values, handling outliers, and handling duplicate values.

[0057] Step 2: Text segmentation, which breaks down the text into words or phrases. There are three types of methods: dictionary-based matching segmentation, statistical segmentation, and deep learning-based segmentation.

[0058] Step 3: Remove stop words, removing words that have no practical meaning or are very frequent in the text analysis.

[0059] Furthermore, data augmentation operations are performed on the preprocessed intelligence samples, such as... Figure 4 As shown, it includes the following sub-steps:

[0060] Step 1: Synonym replacement, generating enhanced samples by replacing some words in the text with their synonyms;

[0061] Step 2: Random insertion. New words are randomly inserted into the text to generate enhanced samples.

[0062] Step 3: Random deletion. Randomly delete some words from the text to generate augmented samples;

[0063] Step 4: Random Swap. Randomly swap the positions of two words in the text to generate augmented samples;

[0064] Step 5: Reverse translation technology. Using reverse translation technology, the sample is translated into another language by one translation software, and then translated back into the original language by another translation software, thereby obtaining different sentences with the same meaning as enhanced samples.

[0065] In a specific embodiment of the present invention, step S102 specifically includes:

[0066] The preprocessed intelligence samples and their augmented samples are encoded using the bag-of-words (CBOW) model with word embeddings to obtain their word vector representations, such as... Figure 5 As shown, it includes the following sub-steps:

[0067] Step 1: Word segmentation. The text is segmented into words according to certain rules or algorithms, dividing it into a sequence of words.

[0068] Step 2: Construct a vocabulary. Collect all the words that appear in the text and build a vocabulary, where each word corresponds to a unique index;

[0069] Step 3: Calculate word frequency by counting the frequency of each word in the text or by using other statistical measures to obtain a word frequency vector;

[0070] Step 4: Vectorization. Based on the vocabulary and word frequency vectors, the text is represented as a vector, where each dimension of the vector corresponds to a word in the vocabulary, and the value of that dimension represents the word frequency or other statistical measure in the text.

[0071] The generated word vectors are input into a topic extraction model for training to extract intelligence topics. This includes two topic models: BTM (Browser-Template) short text topic extraction model and LDA (Long Text-Based Allocation) long text topic extraction model. For example... Figure 6 As shown, the BTM model constructs a topic-word pair matrix and uses Dirichlet hyperparameters α and β to describe the distribution of different words and topics in the corpus. i and w j These represent different words in the corpus, z and θ represent the topic distribution and corpus-topic distribution in the corpus, respectively, and φ k With N B These represent the number of words belonging to topic K and the number of word pairs in the corpus belonging to topic Z, respectively; for example... Figure 7 As shown, the LDA model is a Bayesian probabilistic model with a structure consisting of three layers: words, topics, and documents. Each document is considered to be a mixture of multiple topics, and each topic is composed of the distribution of words. The LDA model optimizes the "topic-vocabulary" and "document-topic" matrices through Bayesian estimation and prior distribution. α and β are hyperparameters.

[0072] The Gibbs sampling method is used to estimate the parameters of the topic model, obtain the topic of each word in the intelligence dataset, until the Gibbs sampling converges, and obtain the topic distribution probability of each intelligence and the word distribution probability of each topic; the process of the Gibbs sampling method is as follows:

[0073] Step 1: Randomly assign an initial topic to each word:

[0074]

[0075] Step Two: When encountering a new topic, the topic calculation formula is as follows:

[0076]

[0077] in, This represents the number of times word t in document i belongs to topic k. This represents the number of all words in document i that belong to topic k. This represents the number of times topic k in document i belongs to word m;

[0078] Step 3: Repeat step 2 until the algorithm converges, and obtain the parameters of the topic to which each word belongs according to formulas (5) and (6):

[0079]

[0080] Furthermore, a suitable threshold of 0.05 is set for the obtained keyword distribution matrix, and words in ZW that exceed the threshold are selected as initial keyword terms; the keyword distribution matrix for each piece of information is shown in equation (7):

[0081]

[0082] Where z represents the probability of each topic. Let w represent the topic distribution of the m-th text, where the sum of the probabilities of each topic is 1; w represents the probability distribution of each word. This is used to represent the word distribution corresponding to the k-th topic, and the sum of the probability distributions of all words is 1.

[0083] In a specific embodiment of the present invention, step S103 specifically includes:

[0084] like Figure 8 As shown, the preprocessed intelligence samples and their augmented samples are input into the Doc2Vec model. The PV-DM technology is used to embed documents and words into a fixed vector space. The context words and sentence vectors are used as input. The target words are predicted by the neural network to generate the Doc2Vec sentence vector of the intelligence document. The model parameters are adjusted by the backpropagation algorithm to ensure that the model can accurately predict the target words and generate accurate document vectors.

[0085] In a specific embodiment of the present invention, step S104 specifically includes:

[0086] The intelligence topic word vectors obtained in step S102 and the intelligence document Doc2Vec sentence vectors obtained in step S103 are concatenated and used as the input vector. This input is then fed into the BERT model for masked language modeling pre-training to learn the rich semantic information and contextual relationships in the intelligence text, resulting in a feature vector containing topic information. Each intelligence topic matrix has d rows. k The number of columns is n+m (text length is n, number of keywords is m). The encoding vector for the initially selected keywords. Compared with the sentence vector obtained by the Doc2Vec model By concatenating the vectors, a merged vector is obtained. After pre-training using masked language modeling, we obtain topic word feature vectors containing intelligence topic information.

[0087] In one specific embodiment of the present invention, the merged vector of each topic is used as a token of the BERT model, d k This indicates the number of tokens in the text, where "m+n" represents the feature dimension of the tokens; 15% of the tokens are randomly selected and replaced with the special mask "MASK"; words masked by "MASK" are predicted using bidirectional contextual information.

[0088] In a specific embodiment of the present invention, in step S105, a semi-supervised SVM model is trained using labeled and unlabeled data. A semi-supervised SVM model is an SVM model trained under semi-supervised conditions. For example... Figure 9 As shown, its sub-steps include:

[0089] Step 1: Divide the intelligence dataset into labeled and unlabeled data using manual annotation.

[0090] Step 2: Input the labeled data into the semi-supervised SVM model for model training and adjust the model parameters;

[0091] Step 3: Input the unlabeled data into the model for prediction and generate pseudo-labels for the unlabeled data;

[0092] Step 4: Construct a training dataset using labeled data and pseudo-labels, input it into the model for fine-tuning, optimize model parameters, and obtain a trained semi-supervised SVM model.

[0093] In a specific embodiment of the present invention, step S106 specifically includes:

[0094] The topic word feature vector containing intelligence topic information obtained in step S104 Construct set D:

[0095]

[0096] in, Let y be the vector of the i-th topic word, m be the number of topic words, and y be the vector of the i-th topic word. i ∈{0,1} represents the word vector label, y={y1,y2…,y m The initial value is 1. The semi-supervised SVM model trained in step S105 is used to perform probability prediction using a binary classification algorithm to obtain the label extraction result.

[0097] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for extracting information tags from heterogeneous BERT and semi-supervised SVM models, characterized in that, Includes the following steps: Step S101: Obtain the original unstructured intelligence text dataset, perform data cleaning, text segmentation, and stop word removal preprocessing operations to obtain a preprocessed intelligence text dataset, and then perform data augmentation through synonym replacement, random insertion, random swapping, random deletion, and reverse translation to obtain an augmented intelligence text dataset. Step S102: Encode the intelligence text using the bag-of-words model to obtain the topic word encoding vector, input it into the BTM short text topic extraction model and the LDA long text topic extraction model, and use the Gibbs sampling method to obtain the topic distribution probability and the topic word distribution probability of each intelligence; the topic word distribution probabilities of all topic words form a topic word distribution matrix, set an appropriate threshold for the topic word distribution matrix, and filter out words that exceed the threshold as initial topic words; Step S103: Input the intelligence text into the Doc2Vec model, use PV-DM technology to take context words and sentence vectors as input, use neural network training to predict target words, capture the semantic information of the intelligence text, and obtain the sentence vector representation of the text; Step S104: Convert the encoding vectors of the initially selected topic words obtained from the topic extraction model in step S102. Compared with the sentence vector obtained by the Doc2Vec model in step S103 By concatenating the vectors, a merged vector is obtained. Where d k Let m be the feature dimension of the vector, m be the length of the initial selected topic word encoding vector, and n be the sentence vector encoding length. The merged vectors are input into the BERT model for masked language modeling pre-training to obtain topic word feature vectors containing intelligence topic information. Step S105: Convert the topic word feature vector obtained in step 104 into a single vector. The input is fed into a semi-supervised SVM model, and semi-supervised training is performed based on a human interaction method. Step S106: Using the topic word feature vector containing intelligence topic information obtained in step S104 as input, input it into the semi-supervised SVM model trained in step S105, and use the binary classification algorithm to perform probability prediction to obtain the label extraction result.

2. The information tag extraction method for heterogeneous BERT and semi-supervised SVM models according to claim 1, characterized in that: In step S101, the data cleaning includes removing special characters, converting case, handling missing values, handling outliers, and handling duplicate values.

3. The method for extracting information tags from heterogeneous BERT and semi-supervised SVM models according to claim 2, characterized in that: In step S101, the synonym replacement involves replacing some words in the text with words that have the same or similar meanings to generate enhanced samples; the random insertion involves randomly adding new words to the text to generate enhanced samples. The random deletion involves randomly deleting some words from the text to generate enhanced samples. The random swapping involves randomly exchanging the positions of two words in the text to generate enhanced samples. The reverse translation involves translating the sample into another language using one translation software, and then translating it back into the original language using another translation software, in order to obtain an enhanced sample with the same semantics but a different sentence.

4. The method for extracting information tags from heterogeneous BERT and semi-supervised SVM models according to claim 3, characterized in that: In step S102, the process of encoding the intelligence text using a bag-of-words model to obtain the topic word encoding vector specifically includes: using the word embedding method of the bag-of-words model, taking the preprocessed intelligence text and the enhanced intelligence text as input, inputting them into the bag-of-words model for encoding, predicting the target word through the words in the context, thereby learning the distributed representation of words, obtaining the topic word encoding vector of the intelligence text, and providing input for subsequent model training.

5. The method for extracting information tags from heterogeneous BERT and semi-supervised SVM models according to claim 4, characterized in that: In step S102, the BTM short text topic extraction model establishes a "topic-vocabulary" matrix and uses Dirichlet hyperparameters α and β to describe the different vocabulary and topic distributions in the corpus; the LDA long text topic extraction model includes vocabulary, topics and documents, establishes a "topic-vocabulary" matrix and a "document-topic" matrix, and optimizes the "topic-vocabulary" matrix and "document-topic" matrix through Bayesian estimation and prior distribution.

6. The method for extracting information tags from heterogeneous BERT and semi-supervised SVM models according to claim 5, characterized in that: In step S102, obtaining the topic distribution probability and the word distribution probability of each piece of intelligence using the Gibbs sampling method specifically includes: using the Gibbs sampling method to estimate the parameters of the BTM short text topic extraction model and the LDA long text topic extraction model, and obtaining the topic distribution probability and the word distribution probability of each topic for each piece of intelligence; setting a threshold for the obtained word distribution matrix ZW, and selecting words in the word distribution matrix ZW that exceed the threshold as initial word selections. The word distribution matrix ZW for each piece of intelligence is represented as follows: Where z represents the probability of each topic. Let w represent the topic distribution of the m-th text, where the sum of the probabilities of each topic is 1; w represents the probability distribution of each word. This is used to represent the word distribution corresponding to the k-th topic, and the sum of the probability distributions of all words is 1.

7. The method for extracting information tags from heterogeneous BERT and semi-supervised SVM models according to claim 6, characterized in that: In step S103, the preprocessed intelligence text and enhanced intelligence text obtained in step S101 are input into the Doc2Vec model. The document and words are embedded in a fixed vector space using PV-DM technology. The context words and sentence vectors are used as input. The target words are predicted by a neural network composed of fully connected layers to generate the Doc2Vec sentence vector of the intelligence document. The parameters of the neural network are adjusted by the backpropagation algorithm to ensure that the neural network can accurately predict the target words and generate accurate document vectors.

8. The method for extracting information tags from heterogeneous BERT and semi-supervised SVM models according to claim 7, characterized in that, In step S104, the specific steps for inputting the merged vector into the BERT model for masked language modeling pre-training include: The merged vector of each topic is used as a token in the BERT model, d k This indicates the number of tokens in the text, where "m+n" represents the feature dimensions contained in the tokens. Randomly select 15% of the tokens and replace them with the special marker "MASK" in the mask; Predict words masked by "MASK" using bidirectional contextual information.

9. The method for extracting information tags from heterogeneous BERT and semi-supervised SVM models according to claim 8, characterized in that, Step S105 specifically includes the following steps: The intelligence dataset is divided into unlabeled data and unlabeled data. Labels are added to all unlabeled data using manual annotation to obtain labeled data. The topic word feature vectors obtained from the labeled data in step S104 are input into the semi-supervised SVM model for model training, and the model parameters are adjusted. The feature vectors of the topic words corresponding to the unlabeled data are also input into the model for prediction to generate pseudo-labels for the unlabeled data; A training dataset is constructed using both labeled and pseudo-labeled data. This dataset is then input into a semi-supervised SVM model for fine-tuning, optimizing the model parameters, and resulting in a fully trained semi-supervised SVM model.

10. The method for extracting information tags from heterogeneous BERT and semi-supervised SVM models according to claim 9, characterized in that: In step S106, for the intelligence text from which tags are to be extracted, the topic word feature vector containing intelligence topic information is obtained through steps S102, S103, and S104. Construct set D: in, Let y be the vector of the i-th topic word, m be the number of topic words, and y be the vector of the i-th topic word. i ∈{0,1} represents the word vector label, y={y1,y2…,y m The initial value is 1. The semi-supervised SVM model trained in step S105 is used to perform probability prediction using a binary classification algorithm to obtain the label extraction result.

Images