Information label extraction method of heterogeneous BERT and semi-supervised SVM model

Abstract

The invention discloses an intelligence label extraction method for heterogeneous BERT and semi-supervised SVM models, and the method comprises the steps: obtaining an original unstructured intelligence text data set, and carrying out the text preprocessing and data enhancement operation of the original unstructured intelligence text data set; based on a BTM short text and LDA long text topic model, subject term extraction is carried out on the intelligence text, and subject term vectors of the intelligence text are obtained; performing sentence vector representation on the intelligence document based on a Doc2Vec model; inputting the subject term vector and the sentence vector into a BERT model for pre-training to obtain a feature vector with subject information; performing training parameter adjustment on the semi-supervised SVM model by using a small amount of labeled data and a large amount of unlabeled data; and inputting the feature vector with the theme information into a semi-supervised SVM binary classification algorithm model for training to obtain an extraction result of the data label. According to the method, the data label of the information text can be effectively extracted, the label-free data information is fully utilized, and the accuracy and generalization ability of a data label extraction system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of label extraction, and in particular to an intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model. Background Art

[0002] In the field of intelligence analysis, label extraction technology is a key task, which aims to automatically extract key information and labels from large amounts of text data to help analysts quickly and accurately understand and process intelligence content. Traditional label extraction methods usually rely on artificial rules or statistical methods, which are often limited by the accuracy of the rules and the generalization ability of the statistical model, and are difficult to handle large-scale and complex text data.

[0003] In recent years, with the development of deep learning and natural language processing technology, label extraction technology based on neural networks has gradually become mainstream, including methods based on pre-trained language models, such as the BERT model. The BERT model can learn the semantic representation of a large amount of text data and has a strong context understanding ability, so it has achieved remarkable results in label extraction tasks. However, due to the complexity and diversity of intelligence data, a single model is often difficult to meet the needs of various intelligence analysis tasks. Therefore, the present invention proposes an intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model, which aims to combine the rich semantic representation of the BERT model and the effective use of unlabeled data by the semi-supervised SVM model, thereby improving the accuracy and efficiency of intelligence label extraction. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model, which can make full use of the unlabeled data in unstructured intelligence text data, enable the system to automatically complete label extraction with high quality, and improve the quality and efficiency of intelligence data system construction.

[0005] To achieve the above object, the specific scheme of the present invention is as follows:

[0006] Step S101: obtaining an original unstructured intelligence text dataset, performing data cleaning, text segmentation, and data preprocessing operations of removing stop words on it to obtain a preprocessed intelligence text dataset, and then performing data enhancement by means of synonym replacement, random insertion, random exchange, random deletion, and reverse translation to obtain an enhanced intelligence text dataset;

[0007] Step S102: Encode the intelligence text using the bag-of-words model to obtain a subject word encoding vector, which is then input into the BTM short text subject extraction model and the LDA long text subject extraction model. The Gibbs sampling method is used to obtain the subject distribution probability of each piece of intelligence and the distribution probability of each subject word. All subject word distribution probabilities form a subject word distribution matrix, and a suitable threshold is set for the subject word distribution matrix to filter out words that exceed the threshold as primary subject words.

[0008] Step S103: Input the intelligence text into the Doc2Vec model, use the PV-DM technology to take the context words and sentence vectors as input, use the neural network training to predict the target words, capture the semantic information of the intelligence text, and obtain the sentence vector representation of the text;

[0009] Step S104: The encoding vector of the primary selected topic words obtained by the topic extraction model in step S102 is The sentence vector obtained by the Doc2Vec model in step S103 Concatenate to get the merged vector where d k is the feature dimension of the vector, m is the length of the primary selected topic word encoding vector, and n is the sentence vector encoding length. The combined vector is input into the BERT model for masked language modeling pre-training to obtain the topic word feature vector containing intelligence topic information.

[0010] Step S105: The feature vector of the subject word obtained in step 104 is Input into the semi-supervised SVM model and perform semi-supervised training based on manual interaction method;

[0011] Step S106: The subject word feature vector containing intelligence subject information obtained in step S104 is used as input to the semi-supervised SVM model trained in step S105, and a binary classification algorithm is used to perform probability prediction to obtain a label extraction result.

[0012] Furthermore, in step S101, the data cleaning includes removing special characters, converting uppercase and lowercase characters, processing missing values, processing abnormal values, and processing duplicate values.

[0013] Furthermore, in step S101, the synonym replacement is to replace some words in the text with words with the same or similar meanings to generate enhanced samples; the random insertion is to randomly add new words in the text to generate enhanced samples; the random deletion is to randomly delete some words in the text to generate enhanced samples; the random exchange is to randomly exchange the positions of two words in the text to generate enhanced samples; the back translation is to translate the sample into other languages ​​through a translation software, and then translate it back to the original language through another translation software to obtain enhanced samples with the same semantics but different sentences.

[0014] Furthermore, in step S102, the bag-of-words model encoding of the intelligence text to obtain the subject word encoding vector specifically includes: adopting the word embedding method of the bag-of-words model, taking the preprocessed intelligence text and the enhanced intelligence text as input, inputting them into the bag-of-words model for encoding, predicting the target words through the words in the context, thereby learning the distributed representation of words, obtaining the subject word encoding vector of the intelligence text, and providing input for subsequent model training.

[0015] Furthermore, in step S102, the BTM short text topic extraction model establishes a "topic-vocabulary" matrix, and uses the Dirichlet hyperparameters α and β to describe the distribution of different vocabularies and topics in the corpus; the LDA long text topic extraction model includes vocabulary, topics and documents, establishes a "topic-vocabulary" matrix and a "document-topic" matrix, and optimizes the "topic-vocabulary" matrix and the "document-topic" matrix through Bayesian estimation and prior distribution.

[0016] Further, in step S102, the use of the Gibbs sampling method to obtain the topic distribution probability of each piece of intelligence and the distribution probability of each subject word specifically includes: using the Gibbs sampling method to estimate the parameters of the BTM short text topic extraction model and the LDA long text topic extraction model, and obtaining the topic distribution probability of each piece of intelligence and the word distribution probability of each subject; setting a threshold for the obtained subject word distribution matrix ZW, selecting words exceeding the threshold in the subject word distribution matrix ZW as preliminary subject words, and the subject word distribution matrix ZW of each piece of intelligence is expressed as:

[0017]

[0018] Among them, z represents the probability of each topic, is the topic distribution of the mth text, the sum of the probabilities of each topic is 1; w represents the distribution probability of each word, It is used to represent the word distribution corresponding to the kth topic, and the sum of the distribution probabilities of all words is 1.

[0019] Furthermore, in step S103, the preprocessed intelligence text and enhanced intelligence text obtained in step S101 are input into the Doc2Vec model, and the documents and words are embedded in a fixed vector space through the PV-DM technology. The context words and sentence vectors are used as input, and the target words are predicted through a neural network composed of fully connected layers to generate Doc2Vec sentence vectors for the intelligence documents, wherein the neural network parameters are adjusted through the back-propagation algorithm to ensure that the neural network can accurately predict the target words and generate accurate document vectors.

[0020] Furthermore, in step S104, the specific steps of inputting the merged vector into the BERT model for pre-training of masked language modeling include:

[0021] The combined vector of each topic is used as a Token of the BERT model, d k Indicates the number of tokens in the text, and “m+n” is the feature dimension contained in the token;

[0022] Randomly select 15% of the tokens and replace them with a special masked token “MASK”;

[0023] Predict the masked word by “MASK” through bidirectional context information.

[0024] Furthermore, step S105 specifically includes the following steps:

[0025] The intelligence data set is divided into unlabeled data and unlabeled data, and labels are added to all unlabeled data by manual labeling to obtain labeled data;

[0026] Input the subject word feature vector obtained by step S104 of the labeled data into the semi-supervised SVM model for model training and adjusting the model parameters;

[0027] The feature vector of the topic word corresponding to the unlabeled data is also input into the model for prediction to generate pseudo labels for the unlabeled data;

[0028] The labeled data and pseudo-labeled data are used together to construct a training data set, which is input into the semi-supervised SVM model for model fine-tuning and optimization of model parameters to obtain a trained semi-supervised SVM model.

[0029] Further, in step S106, for the intelligence text whose label is to be extracted, the subject word feature vector containing the intelligence subject information is obtained through steps S102, S103 and S104. Construct set D:

[0030]

[0031] in, is the i-th topic word vector, m is the number of topic words, y i ∈{0,1} is the word vector label, y={y 1 ,y 2 …,y m} is initialized to 1, and the semi-supervised SVM model trained in step S105 is used to perform probability prediction using a binary classification algorithm to obtain the label extraction result.

[0032] Compared with the existing data label extraction method, the present invention has the following advantages:

[0033] 1. The present invention uses the topic extraction model of BTM short text and LDA long text combination, which can comprehensively consider the characteristics of short text and long text, and improve the accuracy and comprehensiveness of intelligence label extraction. BTM is applicable to processing short text intelligence data, and can capture the associated information between word pairs, while LDA is applicable to processing long text intelligence data, and can better capture vocabulary distribution and semantic information. By comprehensively considering the extraction results of two models, the diversity of labels can be enhanced, and the interpretability of label extraction results can be improved, which helps users to understand the content and theme structure of intelligence text data more comprehensively.

[0034] 2. The present invention uses the Doc2Vec model to generate sentence vectors for intelligence text, and combines the sentence vectors with the keyword vectors as the input of the pre-training model. It can integrate multi-level semantic information, combine word-level and sentence-level features, and improve the model's ability to understand intelligence text and perceive context. At the same time, it enhances the model's generalization ability and ability to process rare words, thereby providing a more comprehensive and accurate semantic representation for intelligence analysis and mining.

[0035] 3. The present invention uses the BERT model to pre-train the subject word vector and sentence vector. It consists of a multi-layer Transformer encoder, and its internal structure includes a self-attention mechanism and a feedforward neural network, which can fully understand the semantics and contextual information of the text. Through bidirectional context modeling, BERT can effectively capture long-distance dependencies, and the parameters pre-trained on large-scale corpus have good generalization ability, which can improve the effect and reliability of intelligence analysis.

[0036] 4. The present invention takes into account the unlabeled data in the intelligence text and adopts the semi-supervised SVM model to perform secondary screening of labels. It can make full use of unlabeled data resources, improve model performance, reduce the risk of overfitting, and at the same time reduce the cost of manual labeling, enhance classification performance and generalization ability, thereby providing more effective support for the intelligence label extraction system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1A flow chart of an intelligence label extraction method for a heterogeneous BERT and semi-supervised SVM model provided in an embodiment of the present invention;

[0038] Figure 2 A detailed flowchart of an intelligence label extraction method for a heterogeneous BERT and semi-supervised SVM model provided in an embodiment of the present invention;

[0039] Figure 3 A diagram of a data preprocessing method provided by an embodiment of the present invention;

[0040] Figure 4 A diagram of a data enhancement method provided by an embodiment of the present invention;

[0041] Figure 5 A CBOW coding diagram provided by an embodiment of the present invention;

[0042] Figure 6 A BTM topic extraction model diagram provided by an embodiment of the present invention;

[0043] Figure 7 An LDA topic extraction model diagram provided by an embodiment of the present invention;

[0044] Figure 8 A sentence vector generation graph based on the Doc2Vec model provided in an embodiment of the present invention;

[0045] Fig. 9 A diagram of a semi-supervised SVM model training method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the features and advantages of the present invention more obvious and understandable, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Figure 1 This is a flow chart of an intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to an embodiment of the present invention. Figure 2 As a detailed flow chart, the method of the present invention comprises the following steps:

[0048] Step S101: obtaining an original unstructured intelligence text dataset, and performing text preprocessing and data enhancement;

[0049] Step S102: extracting intelligence keywords based on the BTM short text and LDA long text topic extraction model;

[0050] Step S103: intelligence document sentence vector representation based on Doc2Vec model;

[0051] Step S104: Pre-training the topic word vector and sentence vector based on the BERT model;

[0052] Step S105: semi-supervised SVM model training based on human-computer interaction;

[0053] Step S106: label extraction based on the semi-supervised SVM binary classification algorithm.

[0054] In a specific embodiment of the present invention, step S101 specifically includes:

[0055] Obtain the original unstructured intelligence text dataset and perform text preprocessing on it, such as Figure 3 As shown, it includes the following sub-steps:

[0056] Step 1: Data cleaning, which includes removing special characters, converting uppercase and lowercase characters, processing missing values, processing abnormal values, processing duplicate values, etc.

[0057] Step 2: Text segmentation, splitting the text into words or phrases. There are three methods: dictionary-based matching segmentation, statistics-based segmentation, and deep learning-based segmentation.

[0058] Step 3: Remove stop words, which are words that have no practical meaning or high frequency in text analysis.

[0059] Furthermore, data enhancement operations are performed on the preprocessed intelligence samples, such as Figure 4 As shown, it includes the following sub-steps:

[0060] Step 1: Synonym replacement, generating enhanced samples by replacing some words in the text with their synonyms;

[0061] Step 2: Random insertion: randomly insert some new words into the text to generate enhanced samples;

[0062] Step 3: Random deletion: randomly delete some words in the text to generate enhanced samples;

[0063] Step 4: Random swap: randomly swap the positions of two words in the text to generate enhanced samples;

[0064] Step 5: Back translation technology: using reverse translation technology, the sample is translated into other languages ​​through one translation software, and then translated back to the original language through another translation software, so as to obtain different sentences with the same semantics as enhanced samples.

[0065] In a specific embodiment of the present invention, step S102 specifically includes:

[0066] The preprocessed intelligence samples and their enhanced samples are encoded using the word embedding method of the bag-of-words model (CBOW) to obtain their word vector representation, such as Figure 5 As shown, it includes the following sub-steps:

[0067] Step 1: Word segmentation: segment the text according to certain rules or algorithms and divide it into a sequence of words;

[0068] Step 2: Build a vocabulary, collect all the words that appear in the text, and build a vocabulary, in which each word corresponds to a unique index;

[0069] Step 3: Calculate the word frequency, count the frequency of each word in the text or other statistics, and get a word frequency vector;

[0070] Step 4: Vectorization: Based on the vocabulary and word frequency vector, the text is represented as a vector, where each dimension of the vector corresponds to a word in the vocabulary, and the value of the dimension represents the word frequency or other statistics of the word in the text.

[0071] The generated word vectors are input into the topic extraction model for training to extract intelligence topics, including two topic models: BTM short text topic extraction model and LDA long text topic extraction model. Figure 6 As shown in the figure, the BTM model builds a topic-word pair matrix and uses Dirichlet hyperparameters α and β to describe the distribution of different words and topics in the corpus. i and w j represents different words in the corpus, z and θ represent the topic distribution and corpus-topic distribution in the corpus respectively, φ k With N B Respectively represent the number of words belonging to topic K and the number of word pairs belonging to topic Z in the corpus; Figure 7 As shown in the figure, the LDA model is a Bayesian probability model, whose structure includes three layers: words, topics, and documents. Each document is considered to be a mixture of multiple topics, and each topic is composed of the distribution of words. The LDA model optimizes the "topic-vocabulary" and "document-topic" matrices through Bayesian estimation and prior distribution. α and β are hyperparameters.

[0072] Use the Gibbs sampling method to estimate the parameters of the topic model, obtain the topic of each word in the intelligence data set, until the Gibbs sampling converges, and obtain the topic distribution probability of each intelligence and the word distribution probability of each topic; the process of the Gibbs sampling method is as follows:

[0073] Step 1: Randomly assign an initial topic to each word:

[0074]

[0075] Step 2: When encountering a new topic, the formula for calculating the topic is as follows:

[0076]

[0077] in, It represents the number of times word t in document i belongs to topic k. represents the number of all words belonging to topic k in document i, represents the number of times topic k belongs to word m in document i;

[0078] Step 3: Repeat step 2 until the algorithm converges, and obtain the parameters of the topic to which each word belongs according to formulas (5) and (6):

[0079]

[0080] Furthermore, a suitable threshold of 0.05 is set for the obtained subject word distribution matrix, and words exceeding the threshold in ZW are selected as the primary subject words; the subject word distribution matrix of each piece of intelligence is shown in formula (7):

[0081]

[0082] Among them, z represents the probability of each topic, is the topic distribution of the mth text, the sum of the probabilities of each topic is 1; w represents the distribution probability of each word, It is used to represent the word distribution corresponding to the kth topic, and the sum of the distribution probabilities of all words is 1.

[0083] In a specific embodiment of the present invention, step S103 specifically includes:

[0084] like Figure 8 As shown in the figure, the preprocessed intelligence samples and their enhanced samples are input into the Doc2Vec model, and the documents and words are embedded in a fixed vector space through the PV-DM technology. The context words and sentence vectors are used as input, and the target words are predicted through the neural network to generate the Doc2Vec sentence vectors of the intelligence documents. Among them, the model parameters are adjusted through the back propagation algorithm to ensure that the model can accurately predict the target words and generate accurate document vectors.

[0085] In a specific embodiment of the present invention, step S104 specifically includes:

[0086] The intelligence topic word vector obtained in step S102 and the intelligence document Doc2Vec sentence vector obtained in step S103 are concatenated as the vector input and input into the BERT model for masked language modeling pre-training to learn the rich semantic information and contextual relationships in the intelligence text and obtain a feature vector containing topic information. The number of rows in each intelligence topic matrix is ​​d k , the number of columns is n+m (the length of the text is n, and the number of keywords is m). The encoding vector of the primary selected keyword And the sentence vector obtained by Doc2Vec model Concatenate to get the merged vector After pre-training with masked language modeling, we obtain the feature vector of the subject word containing intelligence subject information.

[0087] In a specific embodiment of the present invention, the merged vector of each topic is used as a Token of the BERT model. k Represents the number of tokens in the text, "m+n" is the feature dimension contained in the token; 15% of the tokens are randomly selected and replaced with the special masked tag "MASK"; the words masked by "MASK" are predicted through bidirectional context information.

[0088] In a specific embodiment of the present invention, in step S105, the semi-supervised SVM model is trained using labeled and unlabeled data. The so-called semi-supervised SVM model is a semi-supervised training of the SVM model. Fig. 9 As shown, its sub-steps include:

[0089] Step 1: Use manual labeling to divide the intelligence data set into labeled data and unlabeled data;

[0090] Step 2: Input the labeled data into the semi-supervised SVM model for model training and adjust the model parameters;

[0091] Step 3: Input the unlabeled data into the model for prediction and generate pseudo labels for the unlabeled data;

[0092] Step 4: Construct a training data set with labeled data and pseudo-labels, input the model to fine-tune the model, optimize the model parameters, and obtain a trained semi-supervised SVM model.

[0093] In a specific embodiment of the present invention, step S106 specifically includes:

[0094] The subject word feature vector containing intelligence subject information obtained in step S104 Construct set D:

[0095]

[0096] in, is the i-th topic word vector, m is the number of topic words, y i ∈{0,1} is the word vector label, y={y 1 ,y 2 …,y m} is initialized to 1, and the semi-supervised SVM model trained in step S105 is used to perform probability prediction using a binary classification algorithm to obtain the label extraction result.

[0097] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for extracting intelligence labels from heterogeneous BERT and semi-supervised SVM models, characterized in that: The following steps are involved: Step S101: obtaining an original unstructured intelligence text dataset, performing data cleaning, text segmentation, and data preprocessing operations of removing stop words on it to obtain a preprocessed intelligence text dataset, and then performing data enhancement by means of synonym replacement, random insertion, random exchange, random deletion, and reverse translation to obtain an enhanced intelligence text dataset; Step S102: Encode the intelligence text using the bag-of-words model to obtain a subject word encoding vector, which is then input into the BTM short text subject extraction model and the LDA long text subject extraction model. The Gibbs sampling method is used to obtain the subject distribution probability of each piece of intelligence and the distribution probability of each subject word. All subject word distribution probabilities form a subject word distribution matrix, and a suitable threshold is set for the subject word distribution matrix to filter out words that exceed the threshold as primary subject words. Step S103: Input the intelligence text into the Doc2Vec model, use the PV-DM technology to take the context words and sentence vectors as input, use the neural network training to predict the target words, capture the semantic information of the intelligence text, and obtain the sentence vector representation of the text; Step S104: The encoding vector of the primary selected topic words obtained by the topic extraction model in step S102 is The sentence vector obtained by the Doc2Vec model in step S103 Concatenate to get the merged vector where d k is the feature dimension of the vector, m is the length of the primary selected topic word encoding vector, and n is the sentence vector encoding length. The combined vector is input into the BERT model for masked language modeling pre-training to obtain the topic word feature vector containing intelligence topic information. Step S105: The feature vector of the subject word obtained in step 104 is Input into the semi-supervised SVM model and perform semi-supervised training based on manual interaction method; Step S106: The subject word feature vector containing intelligence subject information obtained in step S104 is used as input to the semi-supervised SVM model trained in step S105, and a binary classification algorithm is used to perform probability prediction to obtain a label extraction result.

2. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 1, characterized in that: In step S101, the data cleaning includes removing special characters, converting uppercase and lowercase characters, processing missing values, processing abnormal values, and processing duplicate values.

3. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 2, characterized in that: In step S101, the synonym replacement is to replace some words in the text with words with the same or similar meanings to generate enhanced samples; the random insertion is to randomly add new words in the text to generate enhanced samples; The random deletion is to randomly delete some words in the text to generate enhanced samples; The random exchange is to randomly exchange the positions of two words in the text to generate enhanced samples; The reverse translation is to translate the sample into other languages ​​through a translation software, and then translate it back to the original language through another translation software to obtain an enhanced sample with the same semantics but different sentences.

4. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 3, characterized in that: In step S102, the bag-of-words model encoding of the intelligence text to obtain the subject word encoding vector specifically includes: using the word embedding method of the bag-of-words model, taking the preprocessed intelligence text and the enhanced intelligence text as input, inputting them into the bag-of-words model for encoding, predicting the target word through the words in the context, thereby learning the distributed representation of words, obtaining the subject word encoding vector of the intelligence text, and providing input for subsequent model training.

5. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 4, characterized in that: In step S102, the BTM short text topic extraction model establishes a "topic-vocabulary" matrix, and uses the Dirichlet hyperparameters α and β to describe the distribution of different words and topics in the corpus; the LDA long text topic extraction model includes vocabulary, topics and documents, establishes a "topic-vocabulary" matrix and a "document-topic" matrix, and optimizes the "topic-vocabulary" matrix and the "document-topic" matrix through Bayesian estimation and prior distribution.

6. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 5, characterized in that: In step S102, the method of using the Gibbs sampling method to obtain the topic distribution probability of each piece of intelligence and the distribution probability of each subject word specifically includes: using the Gibbs sampling method to estimate the parameters of the BTM short text topic extraction model and the LDA long text topic extraction model, and obtaining the topic distribution probability of each piece of intelligence and the word distribution probability of each subject; setting a threshold for the obtained subject word distribution matrix ZW, selecting words exceeding the threshold in the subject word distribution matrix ZW as primary subject words, and the subject word distribution matrix ZW of each piece of intelligence is expressed as: Among them, z represents the probability of each topic, is the topic distribution of the mth text, the sum of the probabilities of each topic is 1; w represents the distribution probability of each word, It is used to represent the word distribution corresponding to the kth topic, and the sum of the distribution probabilities of all words is 1.

7. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 6, characterized in that: In step S103, the preprocessed intelligence text and enhanced intelligence text obtained in step S101 are input into the Doc2Vec model, and the documents and words are embedded in a fixed vector space through the PV-DM technology. The context words and sentence vectors are used as input, and the target words are predicted through a neural network composed of fully connected layers to generate Doc2Vec sentence vectors for the intelligence documents. The neural network parameters are adjusted through the back-propagation algorithm to ensure that the neural network can accurately predict the target words and generate accurate document vectors.

8. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 7, characterized in that: In step S104, the specific steps of inputting the merged vector into the BERT model for pre-training of masked language modeling include: The combined vector of each topic is used as a Token of the BERT model, d k Indicates the number of tokens in the text, "m+n" is the feature dimension contained in the token; Randomly select 15% of the tokens and replace them with a special masked token "MASK"; Predict the word masked by "MASK" through bidirectional context information.

9. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 8, characterized in that: Step S105 specifically includes the following steps: The intelligence data set is divided into unlabeled data and unlabeled data, and labels are added to all unlabeled data by manual labeling to obtain labeled data; Input the subject word feature vector obtained by step S104 of the labeled data into the semi-supervised SVM model for model training and adjusting the model parameters; The feature vector of the topic word corresponding to the unlabeled data is also input into the model for prediction to generate pseudo labels for the unlabeled data; The labeled data and pseudo-labeled data are used together to construct a training data set, which is input into the semi-supervised SVM model for model fine-tuning and optimization of model parameters to obtain a trained semi-supervised SVM model.

10. The intelligence label extraction method of a heterogeneous BERT and semi-supervised SVM model according to claim 9, characterized in that: In step S106, for the intelligence text whose label is to be extracted, the subject word feature vector containing the intelligence subject information is obtained through steps S102, S103 and S104. Construct set D: in, is the i-th topic word vector, m is the number of topic words, y i ∈{0,1} is the word vector label, y={y1,y2…,y m } is initialized to 1, and the semi-supervised SVM model trained in step S105 is used to perform probability prediction using a binary classification algorithm to obtain the label extraction result.

Images