Structured Semantic Analysis Method for Defense Science and Technology Texts Applicable to Sparse Sample Scenarios

Abstract

This application proposes a structured semantic analysis method for defense science and technology texts suitable for few-shot scenarios, comprising: acquiring defense science and technology text data; automatically matching templates to the defense science and technology text data, and manually annotating and filtering the matched templates to obtain training data; using the training data to perform unsupervised semantic training on BERT pairs using contrastive learning to obtain a contrastive learning model; using the training data to perform semi-supervised joint training on the contrastive learning model and the named entity model to obtain a trained named entity model; acquiring the unlabeled defense science and technology text to be predicted, inputting the defense science and technology text data into the named entity recognition model, and outputting BIOS tags as the recognition result. This invention, employing the above scheme, can fully utilize the information provided by unlabeled data in few-shot scenarios, improving the accuracy of structured semantic analysis of defense science and technology texts.

Description

Technical Field

[0001] This application relates to the field of semi-supervised learning technology, and in particular to a method and apparatus for structured semantic analysis of defense science and technology texts applicable to scenarios with few samples. Background Technology

[0002] Structured semantic analysis of defense science and technology texts is fundamental to semantic understanding and search in defense science and technology early warning systems. Accurate identification of research tasks and solutions within these texts enables precise intelligent retrieval. Currently, research on structured semantic analysis of defense science and technology texts is relatively limited, and large-scale labeled data is lacking in the industry. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, the first objective of this application is to propose a structured semantic analysis method for defense science and technology texts applicable to low-sample scenarios. This method solves the technical problem that the lack of labeled data in existing methods leads to low accuracy of the trained recognition model, and enables full utilization of information provided by unlabeled data in low-sample scenarios, thereby improving the accuracy of structured semantic analysis of defense science and technology texts.

[0005] The second objective of this application is to propose a structured semantic analysis device for defense science and technology texts suitable for scenarios with few samples.

[0006] To achieve the above objectives, the first aspect of this application proposes a structured semantic analysis method for defense science and technology texts suitable for scenarios with few samples. The method includes: acquiring defense science and technology text data, wherein the data includes labeled defense science and technology texts and unlabeled defense science and technology texts; performing automatic template matching on the defense science and technology text data, and manually labeling and filtering the matched templates to obtain training data; using the training data to perform unsupervised semantic training on BERT pairs using a contrastive learning approach to obtain a contrastive learning model, wherein the contrastive learning model includes a text generation module, BERT, and a similarity judgment module; using the training data to perform semi-supervised joint training on the contrastive learning model and the named entity model to obtain a trained named entity model; acquiring the unlabeled defense science and technology text to be predicted, inputting the defense science and technology text data into the named entity recognition model, and outputting BIOS tags as the recognition result.

[0007] The present application provides a structured semantic analysis method for defense science and technology texts applicable to low-sample scenarios. Seed data is obtained through automatic template matching, and training data is obtained through manual correction. Finally, a semi-supervised learning method is used to jointly train a contrastive learning and named entity recognition model, thereby achieving structured semantic analysis of defense science and technology texts in low-sample scenarios.

[0008] Optionally, in one embodiment of this application, after obtaining the defense science and technology text data, the method further includes using a word segmentation tool to preprocess the defense science and technology text data, wherein the data preprocessing includes:

[0009] Use regular expressions to remove HTML tags, garbled characters, and meaningless symbols from defense science and technology text data;

[0010] The defense science and technology text data is divided into several sentences according to punctuation marks, and each sentence is further segmented into several words using word segmentation technology.

[0011] Optionally, in one embodiment of this application, automatic template matching of text data includes:

[0012] All sentences are traversed using a fixed-size sliding window, and the word frequency and co-occurrence frequency of each word are counted.

[0013] While traversing the graph, a directed acyclic graph is built based on the words in the sentence window;

[0014] The point mutual information is calculated based on word frequency and co-occurrence frequency. Based on the point mutual information and a set threshold, the directed acyclic graph is divided into several subgraphs. For each subgraph after division, a clustering algorithm based on dynamic programming is used to find several optimal paths as sentence templates.

[0015] Point-to-point mutual information is represented as follows:

[0016]

[0017] Here, x and y are any two words within the sliding window, p(x) and p(y) represent the probabilities of words x and y appearing respectively, and p(x,y) represents the probability of words x and y appearing together.

[0018] Optionally, in one embodiment of this application, training data is obtained by manually annotating and filtering the matched templates, including:

[0019] Sentence templates are manually annotated according to entity type;

[0020] Based on the labeled sentence templates, the defense science and technology text data is traversed, and the entity types are labeled by combining sentence templates and rules to obtain training data.

[0021] Optionally, in one embodiment of this application, a contrastive learning approach is used to perform unsupervised semantic training on BERT pairs using training data to obtain a contrastive learning model, including:

[0022] The training data is processed by the text generation module to obtain training text, which includes original text, positive samples, and negative samples. Positive samples are obtained by data augmentation of the original text, and negative samples are obtained by random sampling.

[0023] The training text is encoded using BERT with shared parameters. The error of the BERT encoding is calculated through a similarity judgment module, and the parameters of BERT are optimized using a gradient descent algorithm.

[0024] Optionally, in one embodiment of this application, the error of BERT encoding is represented as:

[0025]

[0026] Where q represents the original text, k i k represents the positive and negative samples after data augmentation. + This only represents positive samples, and τ represents the temperature coefficient.

[0027] Optionally, in one embodiment of this application, semi-supervised joint training of the contrastive learning model and the named entity model is performed using training data to obtain a trained named entity model, including:

[0028] The training data is encoded and predicted using a named entity model, and the posterior probability of the BIOS label is output. The supervised learning loss value is obtained by calculating the difference between the predicted path and the real path.

[0029] The training data is encoded using a contrastive learning model, and the unsupervised learning loss value is calculated using the contrastive learning algorithm through an unsupervised learning method.

[0030] The semi-supervised learning loss value is calculated based on the supervised learning loss value and the unsupervised learning loss value, and the parameters of the contrastive learning model and the named entity model are optimized using the gradient descent algorithm.

[0031] Optionally, in one embodiment of this application, the supervised learning loss value is represented as:

[0032]

[0033] Among them, s real Indicates the actual path, s j This represents the predicted j-th path.

[0034] The unsupervised learning loss value is represented as:

[0035]

[0036] Where q represents the original text, k i k represents the positive and negative samples after data augmentation. + This only represents positive samples, where τ represents the temperature coefficient.

[0037] The semi-supervised learning loss value is expressed as:

[0038] m θ inJ=J sup +λ*J unsup

[0039] Among them, J sup J represents the supervised learning loss value. unsup λ represents the unsupervised learning loss value, and λ represents the proportionality coefficient, where λ∈[0,1].

[0040] To achieve the above objectives, a second aspect of the present invention proposes a structured semantic analysis device for defense science and technology texts suitable for scenarios with few samples, comprising a data acquisition module, a training data generation module, an unsupervised learning training module, a joint training module, and a prediction module, wherein...

[0041] The data acquisition module is used to acquire defense science and technology text data, which includes defense science and technology text with existing labels and defense science and technology text without labels.

[0042] The training data generation module is used to automatically match templates for defense science and technology text data, and to obtain training data by manually annotating and filtering the matched templates.

[0043] The unsupervised learning training module is used to train BERT pairs in unsupervised learning using training data in a contrastive learning manner to obtain a contrastive learning model. The contrastive learning model includes a text generation module, BERT, and a similarity judgment module.

[0044] The joint training module is used to perform semi-supervised joint training of the contrastive learning model and the named entity model using training data to obtain a trained named entity model.

[0045] The prediction module is used to acquire unlabeled defense science and technology texts to be predicted, input the defense science and technology text data into the named entity recognition model, and output BIOS tags as the recognition results.

[0046] Optionally, in one embodiment of this application, a data preprocessing module is further included, used to preprocess the defense science and technology text data using a word segmentation tool after obtaining the defense science and technology text data. Specifically, the data preprocessing module is used for:

[0047] Use regular expressions to remove HTML tags, garbled characters, and meaningless symbols from defense science and technology text data;

[0048] The defense science and technology text data is divided into several sentences according to punctuation marks, and each sentence is further segmented into several words using word segmentation technology.

[0049] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0050] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0051] Figure 1 This is a flowchart illustrating a structured semantic analysis method for defense science and technology texts applicable to scenarios with few samples, provided in Embodiment 1 of this application.

[0052] Figure 2 This is a schematic diagram of the cluster search algorithm according to an embodiment of this application;

[0053] Figure 3 This is a directed acyclic graph of the words in the embodiments of this application;

[0054] Figure 4 This is a flowchart illustrating the operation of the comparative learning model in an embodiment of this application.

[0055] Figure 5 This is a structural diagram of the named entity recognition model according to an embodiment of this application;

[0056] Figure 6 This is a schematic diagram of a structured semantic analysis device for defense science and technology texts suitable for scenarios with few samples, provided in an embodiment of this application. Detailed Implementation

[0057] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0058] The following describes, with reference to the accompanying drawings, a method and apparatus for structured semantic analysis of defense science and technology texts applicable to scenarios with few samples.

[0059] Figure 1 This is a flowchart illustrating a structured semantic analysis method for defense science and technology texts applicable to scenarios with few samples, provided in Embodiment 1 of this application.

[0060] like Figure 1 As shown, this structured semantic analysis method for defense science and technology texts suitable for scenarios with few samples includes the following steps:

[0061] Step 101: Obtain defense science and technology text data, wherein the data preprocessing of defense science and technology text data includes defense science and technology text with existing labels and defense science and technology text without labels;

[0062] Step 102: Perform automatic template matching on the preprocessed defense science and technology text data, and manually annotate and filter the matched templates to obtain training data;

[0063] Step 103: Using contrastive learning, the BERT pairs are trained in unsupervised semantics using preprocessed data to obtain a contrastive learning model. The preprocessed contrastive learning model includes a text generation module, BERT, and a similarity judgment module.

[0064] Step 104: Use the data preprocessing training data to perform semi-supervised joint training on the data preprocessing contrastive learning model and the data preprocessing named entity model to obtain the trained named entity model.

[0065] Step 105: Obtain the unlabeled defense science and technology text to be predicted, input the defense science and technology text data into the named entity recognition model, and output the BIOS tag as the recognition result.

[0066] The present application provides a structured semantic analysis method for defense science and technology texts applicable to low-sample scenarios. Seed data is obtained through automatic template matching, and training data is obtained through manual correction. Finally, a semi-supervised learning method is used to jointly train a contrastive learning and named entity recognition model, thereby achieving structured semantic analysis of defense science and technology texts in low-sample scenarios.

[0067] Optionally, in one embodiment of this application, after obtaining the defense science and technology text data, the method further includes using a word segmentation tool to preprocess the defense science and technology text data, wherein the data preprocessing includes:

[0068] Use regular expressions to remove HTML tags, garbled characters, and meaningless symbols from defense science and technology text data;

[0069] The defense science and technology text data is divided into several sentences according to punctuation marks, and each sentence is further segmented into several words using word segmentation technology.

[0070] Optionally, in one embodiment of this application, automatic template matching of text data includes:

[0071] Using a fixed-size sliding window, all sentences are traversed, and the word frequency and co-occurrence frequency of each word are counted.

[0072] For the two words a and b in the sentence window, add an edge from a to b and build a directed acyclic graph;

[0073] To filter out low-confidence edges and reduce the computational complexity of the graph, the mutual information of points is calculated based on word frequency and co-occurrence frequency, and a threshold is set to divide the graph into several subgraphs. For each subgraph, the optimal n paths, i.e. the generated sentence template, are found using a bundle algorithm based on dynamic programming.

[0074] Point mutual information is a measure of the interdependence between two words. The formula for calculating point mutual information is shown below:

[0075]

[0076] Here, x and y are any two words within the sliding window, p(x) and p(y) represent the probabilities of words x and y appearing respectively, and p(x,y) represents the probability of words x and y appearing together.

[0077] Bundle search is an improvement on the greedy strategy. At each time step, it retains the n outputs with the highest current scores. When n=1, bundle search becomes a greedy search. As shown in the diagram, each time step has 5 possible outputs (A, B, C, D, E). In the diagram, n=2, and at each time step, it retains the two sequences with the best conditional probabilities up to the current step.

[0078] Figure 2 This is a schematic diagram of the beam search algorithm. Figure 2 As can be seen, in the first time step, A and C are the two optimal word nodes, thus yielding two results, A and C. The second time step continues to generate based on these two results. In the branch A, five candidate word nodes can be obtained: AA, AB, AC, AD, and AE. Similarly, five candidate word nodes are obtained for C. At this point, these 10 paths are sorted, and the two optimal paths, AB and CE, are retained again. The third time step is similar, finally yielding two results, ABD and CED.

[0079] For example, if the input text is "Country A and Country B are cooperating in the development of new weapons and equipment using artificial intelligence technology," after text data preprocessing, directed acyclic graph construction, and threshold filtering, the result is as follows: Figure 3 The network of the words shown.

[0080] The following templates can be automatically extracted from a word network using a bundle search algorithm:

[0081] 1. Country A X Country B

[0082] 2. X and X

[0083] 3. Utilizing X technology

[0084] 4. X and X utilize X technology X

[0085] 5. X utilizes X technology X.

[0086] Optionally, in one embodiment of this application, training data is obtained by manually annotating and filtering the matched templates, including:

[0087] Sentence templates are manually annotated according to entity type;

[0088] Based on the labeled sentence templates, the defense science and technology text data is traversed, and the entity types are labeled by combining sentence templates and rules to obtain training data.

[0089] Specifically, after automatically extracting sentence templates, a small amount of manual annotation can be used to identify named entities of various types, such as research tasks, research methods / techniques, location information, and weapon names, through the "slots" in the templates. The generated templates and the entity types corresponding to the "slots" are then used to automatically annotate the defense science and technology text database, annotating all entities that conform to the template rules. Then, a named entity recognition method based on semi-supervised learning is used to learn these annotated data, thereby achieving structured semantic analysis in scenarios with a small number of samples.

[0090] Optionally, in one embodiment of this application, unsupervised semantic training employs contrastive learning to fine-tune the BERT model parameters. The contrastive learning training input data includes labeled training data and a large amount of unlabeled data. The main goal of this step is to utilize the contrastive learning algorithm to learn text features, so as to transfer the text features learned from unlabeled data to the named entity recognition model, thereby achieving the goal of few-shot learning.

[0091] First, the paper data is input and augmented using data augmentation methods. Then, the model is trained using contrastive learning. For each batch of training data, noisy data is generated using data augmentation. The input text is encoded using a BERT with shared parameters, so that the encoded representations of similar samples within the same batch are close to each other, while dissimilar samples are far apart. Finally, the contrastive learning loss and the BERT mask prediction loss are added together to pre-train the entire model.

[0092] The operation process of the contrastive learning model is as follows: Figure 4 As shown:

[0093] Positive samples for contrastive learning training are generated through data augmentation, with common methods including random insertion, random swapping, and random deletion. For positive samples in defense science and technology texts, the aim is to change the sentence's expression and length without altering its semantics. However, randomly inserting words into sentences may introduce noise, changing the sentence's meaning, while randomly deleting words will also distort the original meaning. Therefore, this patent employs "word repetition" and "word replacement" methods to augment the paper data. "Word repetition" refers to repeating high-weight words in the defense science and technology text, while "word replacement" replaces low-weight words in the defense science and technology text with words from the defense science and technology paper corpus that have lower IDF weights.

[0094] For an input sequence x = {x1, x2, ..., xN}, the length of repeated or replaced words is defined as follows:

[0095] L dup ∈[0, max(2, floor(R)] dup *N))]

[0096] Where Ldup is the number of random samples of a word in the input sequence, N is the sequence length, Rdup is the maximum repetition rate (set empirically), floor is the floor function, and max is the maximum value function. After determining the number of random samples of words, word weights are calculated based on features such as word frequency, document frequency, and positional information, and words with higher weights are selected from the input sequence for repetition processing. In the defense science and technology paper data, the positional features such as title, abstract, and keywords, in descending order of their textual expressive power, are: title, abstract, keywords, first sentence of the first paragraph, last sentence of the first paragraph, last paragraph, and other positions, totaling eight positional features. i，j The weights at the k-th position in the text are as follows:

[0097]

[0098] Where i represents the position of the word in the sequence, j represents the position of the document in the defense science and technology corpus, and tf i，j，k The word t i，j At the word frequency at position k, df i，j，k The word t i，j The number of documents containing the word at position k, C represents the number of papers in the defense science and technology text database, and L... i represents the length of the j-th document, and avgl represents the average length of the text in the defense science and technology corpus.

[0099] word t i，j The formula for calculating the weights at all locations is as follows:

[0100]

[0101] Where the position weighting coefficient λ k Satisfying λ k+1 ≥λ k k = 1, 2, ..., 7; and λ k The values ​​were set based on experience. In the experiment, the weighting coefficients were 0.22, 0.17, 0.15, 0.14, 0.12, 0.11, 0.10, and 0.09.

[0102] After data augmentation of the text, K+ positive examples corresponding to the text are obtained, and K- irrelevant negative examples are obtained through random sampling. Then, BERT is used to encode the original text, positive examples, and negative examples. Finally, the InfoNCE loss function is used to calculate the error of the BERT encoding. The definition of the InfoNCE loss function is as follows:

[0103]

[0104] Where q represents the paper, ki represents the positive and negative examples after data augmentation, and k+ represents only positive examples. τ represents the temperature coefficient, a scalar used to smooth the posterior probability of the BERT output.

[0105] Optionally, in one embodiment of this application, the error of BERT encoding is represented as:

[0106]

[0107] Where q represents the original text, k i k represents the positive and negative samples after data augmentation. + This only represents positive samples, and τ represents the temperature coefficient.

[0108] Optionally, in one embodiment of this application, the structured semantic analysis of defense science and technology text can be regarded as a named entity recognition task. The BERT-CRF method is used for supervised learning of labeled data. Considering that the training sample data is small, contrastive learning is used to learn text features. By combining supervised learning and unsupervised learning methods, the unlabeled data is fully utilized to improve the effect of supervised learning.

[0109] The BERT-CRF algorithm is a combined model that combines BERT and CRF. First, BERT is used to encode the input text. Then, the emission probability from the paper data to the predicted label is calculated through the hidden layer. The transition probability between labels is calculated using the label transition matrix. Finally, the CRF layer is used to calculate the final BIOS label.

[0110] The BERT-CRF model structure is as follows: Figure 5As shown, BERT is a pre-trained language model built on top of Transformer, possessing powerful language representation and feature extraction capabilities. It is typically used to fine-tune various downstream tasks. Transformer is a deep learning model based on a self-attention mechanism, utilizing attention to improve the model's feature encoding ability and training speed. Because it is suitable for parallel computing scenarios, its inherent model complexity results in higher accuracy and performance than recurrent neural networks.

[0111] The Transformer consists of an encoder and a decoder. The encoder is composed of a self-attention module and a feedforward neural network, and the decoder uses the same structure. The decoder first calculates a self-attention score on the output. The difference is that after the self-attention mechanism is applied, the output of the self-attention module is compared with the output of the decoder module to calculate the attention score again before entering the feedforward neural network module.

[0112] The core of Transformer is the self-attention module, and the self-attention calculation steps are as follows:

[0113] 1. First, convert the input words into embedding vectors;

[0114] 2. Obtain three vectors q, k, and v (all three vectors are identical) based on the embedding vector;

[0115] 3. Calculate the attention score: score = q·k;

[0116] 4. To make the gradient more stable, the Transformer scales the score, that is, the score is divided by a scaling factor;

[0117] 5. Use the softmax activation function to normalize the scores by weights, so that all of them sum to 1;

[0118] 6. Multiply the result of the softmax step by the vector v to obtain the weighted score of each input vector;

[0119] 7. Sum the weighted v vectors together to get the output of the self-attention module at the current position.

[0120] To further refine the self-attention mechanism layer, the Transformer adds a "multi-head attention mechanism," which expands the model's ability to focus on different locations and allows the self-attention layer to have multiple "representation subspaces," enabling the Transformer to encode richer semantic information. The "multi-head attention mechanism" reuses the self-attention module for encoding (without sharing parameters) and then concatenates the results together.

[0121] BERT only uses the Transformer encoder, and its pre-training includes a Language Modeling (MLM) task and a Next Sentence Prediction (NSP) task. The MLM task replaces a word in a sequence with a placeholder [mask], then uses the model to predict the word that was replaced by the [mask] in the sequence, finally calculating only the loss between the predicted and the true value. The NSP task mainly determines whether the second sentence is truly the next sentence after the first sentence when two sentences are input simultaneously into the model. Typically, the vector corresponding to the NSP is used as the vector representation of the sentence.

[0122] Conditional random fields (CRFs) are discriminative probabilistic models commonly used for labeling or analyzing sequence data, such as natural language text or biological sequences. A CRF is a conditional probability distribution model P(Y|X), representing a Markov random field that, given a set of input random variables X, outputs a set of random variables Y.

[0123] BERT can only learn the contextual relationships of features, while Conditional Random Fields (CRFs) can learn the contextual relationships of labels. Therefore, a combined BERT-CRF model is often used to train named entity recognition models, enabling the simultaneous learning of the contextual relationships of features and labels. CRFs contain two types of feature functions: one for the correspondence between observed sequences and states, and the other for the relationships between states. In the BERT+CRF model, the former type of feature function is given by the posterior probability of the BERT output, while the latter type is given by the label transition matrix. BERT-CRF uses BERT to encode the paper data, takes the CLS vector from the NSP task output, then uses a hidden layer neural network to convert the BERT output into the emission probabilities of the labels, and finally calculates the corresponding probabilities of the output labels using a CRF.

[0124] The decoding strategy of Conditional Random Fields (CRFs) is to find the path with the highest probability and best performance among all possible paths. This label sequence is the output of the model. Assuming the number of labels is k and the text length is n, there are N = kn paths. If Si represents the score of the i-th path, we can calculate the probability of a label sequence appearing as follows:

[0125]

[0126] Where exp represents the exponential function, and the same applies below.

[0127] Now we have a real path, which is the sequence we expect the CRF to decode. Its score can be represented as Sreal, and its probability of occurrence is:

[0128]

[0129] The goal of named entity recognition algorithm modeling and learning is to continuously improve the probability value of P(Sreal). Therefore, the negative log-likelihood function is used to model the loss function of the BERT-CRF model:

[0130]

[0131] The unsupervised learning part uses the same contrastive learning method as the unsupervised learning training process described above. The difference is that only labeled data is used as the input to the model, and the unsupervised learning output loss value is Junsup.

[0132]

[0133] The overall objective function for semi-supervised learning is:

[0134]

[0135] Where λ∈[0,1], is set based on experience.

[0136] After training the named entity recognition model described above, the process also includes predicting unlabeled data labels and updating the training data. Specifically:

[0137] The trained named entity recognition model is used to predict task-method entity pairs in unlabeled paper data, and the BIOS labels of the entities are output. Test samples with labels above a threshold are selected, and the predicted labels are used as the ground truth labels. These samples are then added to the training dataset for repeated unsupervised and semi-supervised training until there is no more unlabeled data that meets the criteria.

[0138] The strategy for the model to predict unlabeled paper data and select new labeled samples is as follows: if the predicted label for each word in the unlabeled paper data is credible (i.e., higher than a pre-set threshold), then the prediction result of the paper data is considered credible, and it is added to the training data for the next round of model training; otherwise, the prediction result of the paper data is considered unreliable, and the next round of label prediction continues.

[0139] To achieve the above embodiments, this application also proposes a structured semantic analysis device for defense science and technology texts suitable for scenarios with few samples.

[0140] Figure 6 This is a schematic diagram of a structured semantic analysis device for defense science and technology texts suitable for scenarios with few samples, provided in an embodiment of this application.

[0141] like Figure 6 As shown, this structured semantic analysis device for defense science and technology texts suitable for few-sample scenarios includes a data acquisition module, a training data generation module, an unsupervised learning training module, a joint training module, and a prediction module.

[0142] The data acquisition module is used to acquire defense science and technology text data, which includes defense science and technology text with existing labels and defense science and technology text without labels.

[0143] The training data generation module is used to automatically match templates for defense science and technology text data, and to obtain training data by manually annotating and filtering the matched templates.

[0144] The unsupervised learning training module is used to train BERT pairs in unsupervised learning using training data in a contrastive learning manner to obtain a contrastive learning model. The contrastive learning model includes a text generation module, BERT, and a similarity judgment module.

[0145] The joint training module is used to perform semi-supervised joint training of the contrastive learning model and the named entity model using training data to obtain a trained named entity model.

[0146] The prediction module is used to acquire unlabeled defense science and technology texts to be predicted, input the defense science and technology text data into the named entity recognition model, and output BIOS tags as the recognition results.

[0147] Optionally, in one embodiment of this application, a data preprocessing module is further included, used to preprocess the defense science and technology text data using a word segmentation tool after obtaining the defense science and technology text data. Specifically, the data preprocessing module is used for:

[0148] Use regular expressions to remove HTML tags, garbled characters, and meaningless symbols from defense science and technology text data;

[0149] The defense science and technology text data is divided into several sentences according to punctuation marks, and each sentence is further segmented into several words using word segmentation technology.

[0150] It should be noted that the foregoing explanation of the embodiment of the structured semantic analysis method for defense science and technology text applicable to few-sample scenarios also applies to the structured semantic analysis device for defense science and technology text applicable to few-sample scenarios in this embodiment, and will not be repeated here.

[0151] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0152] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0153] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0154] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0155] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0156] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0158] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A structured semantic analysis method for defense science and technology texts suitable for scenarios with few samples, characterized in that, Includes the following steps: Acquire defense science and technology text data, wherein the defense science and technology text data includes defense science and technology texts with existing labels and defense science and technology texts without labels; The defense science and technology text data is automatically template matched, and the matched templates are manually labeled and filtered to obtain training data. The BERT pair is trained unsupervised semantically using the training data in a contrastive learning manner to obtain a contrastive learning model, wherein the contrastive learning model includes a text generation module, BERT, and a similarity judgment module; The training data is used to perform semi-supervised joint training on the contrastive learning model and the named entity model to obtain a trained named entity model. Obtain the unlabeled defense science and technology text to be predicted, input the defense science and technology text data into the named entity recognition model, and output the BIOS tag as the recognition result; The automatic template matching of the text data includes: All sentences are traversed using a fixed-size sliding window, and the word frequency and co-occurrence frequency of each word are counted. While traversing the graph, a directed acyclic graph is built based on the words in the sentence window; The point mutual information is calculated based on the word frequency and the co-occurrence frequency. Based on the point mutual information and the set threshold, the directed acyclic graph is cut into several subgraphs. For each subgraph after cutting, a clustering algorithm based on dynamic programming is used to find several optimal paths as sentence templates. The process of manually labeling and filtering the matched templates to obtain training data includes: The sentence templates are manually annotated according to entity type; Based on the labeled sentence templates, the defense science and technology text data is traversed, and the entity types are labeled by combining the sentence templates and rules to obtain the training data; The step of using the training data to perform semi-supervised joint training on the contrastive learning model and the named entity model to obtain a trained named entity model includes: The training data is encoded and predicted using the named entity model, and the posterior probability of the BIOS label is output. The supervised learning loss value is obtained by calculating the difference between the predicted path and the real path. The training data is encoded using the contrastive learning model, and the unsupervised learning loss value is calculated using the contrastive learning algorithm through an unsupervised learning method. The semi-supervised learning loss value is calculated based on the supervised learning loss value and the unsupervised learning loss value, and the parameters of the contrastive learning model and the named entity model are optimized using the gradient descent algorithm.

2. The method as described in claim 1, characterized in that, After acquiring the defense science and technology text data, the method further includes using a word segmentation tool to preprocess the defense science and technology text data, wherein the data preprocessing includes: Regular expressions were used to remove HTML tags, garbled characters, and meaningless symbols from the defense science and technology text data. The defense science and technology text data is divided into several sentences according to punctuation marks, and each sentence is further segmented into several words using word segmentation technology.

3. The method as described in claim 1, characterized in that, The point mutual information is represented as follows: Here, x and y are any two words within the sliding window, p(x) and p(y) represent the probabilities of words x and y appearing respectively, and p(x,y) represents the probability of words x and y appearing together.

4. The method as described in claim 1, characterized in that, The method of using contrastive learning to perform unsupervised semantic training on BERT pairs using the training data to obtain a contrastive learning model includes: The training data is processed by the text generation module to obtain training text, wherein the training text includes original text, positive samples, and negative samples. The positive samples are obtained by data augmentation of the original text, and the negative samples are obtained by random sampling. The training text is encoded using BERT with shared parameters. The error of the BERT encoding is calculated by the similarity judgment module, and the parameters of BERT are optimized using the gradient descent algorithm.

5. The method as described in claim 4, characterized in that, The error of the BERT encoding is expressed as: Where q represents the original text, k i k represents the positive and negative samples after data augmentation. + This only represents positive samples, and τ represents the temperature coefficient.

6. The method as described in claim 1, characterized in that, The supervised learning loss value is expressed as: in, Indicates the actual path, This represents the predicted j-th path. The unsupervised learning loss value is expressed as: Where q represents the original text, k i k represents the positive and negative samples after data augmentation. + This only represents positive samples, where τ represents the temperature coefficient. The semi-supervised learning loss value is expressed as: in, This represents the supervised learning loss value. This represents the unsupervised learning loss value. This represents the proportionality coefficient. .

7. A structured semantic analysis device for defense science and technology texts suitable for scenarios with few samples, characterized in that, It includes a data acquisition module, a training data generation module, an unsupervised learning training module, a joint training module, and a prediction module. The data acquisition module is used to acquire defense science and technology text data, wherein the defense science and technology text data includes defense science and technology text with existing labels and defense science and technology text without labels. The training data generation module is used to automatically match templates for the defense science and technology text data, and to obtain training data by manually annotating and filtering the matched templates. The unsupervised learning training module is used to train the BERT pair in unsupervised learning using the training data in a contrastive learning manner to obtain a contrastive learning model. The contrastive learning model includes a text generation module, BERT, and a similarity judgment module. The joint training module is used to perform semi-supervised joint training of the contrastive learning model and the named entity model using the training data to obtain a trained named entity model. The prediction module is used to obtain the unlabeled defense science and technology text to be predicted, input the defense science and technology text data into the named entity recognition model, and output the BIOS tag as the recognition result. The automatic template matching of the text data includes: All sentences are traversed using a fixed-size sliding window, and the word frequency and co-occurrence frequency of each word are counted. While traversing the graph, a directed acyclic graph is built based on the words in the sentence window; The point mutual information is calculated based on the word frequency and the co-occurrence frequency. Based on the point mutual information and the set threshold, the directed acyclic graph is cut into several subgraphs. For each subgraph after cutting, a clustering algorithm based on dynamic programming is used to find several optimal paths as sentence templates. The process of manually labeling and filtering the matched templates to obtain training data includes: The sentence templates are manually annotated according to entity type; Based on the labeled sentence templates, the defense science and technology text data is traversed, and the entity types are labeled by combining the sentence templates and rules to obtain the training data; The step of using the training data to perform semi-supervised joint training on the contrastive learning model and the named entity model to obtain a trained named entity model includes: The training data is encoded and predicted using the named entity model, and the posterior probability of the BIOS label is output. The supervised learning loss value is obtained by calculating the difference between the predicted path and the real path. The training data is encoded using the contrastive learning model, and the unsupervised learning loss value is calculated using the contrastive learning algorithm through an unsupervised learning method. The semi-supervised learning loss value is calculated based on the supervised learning loss value and the unsupervised learning loss value, and the parameters of the contrastive learning model and the named entity model are optimized using the gradient descent algorithm.

8. The apparatus as claimed in claim 7, characterized in that, It also includes a data preprocessing module, used to preprocess the defense science and technology text data using a word segmentation tool after acquiring the data. Specifically, the data preprocessing module is used for: Regular expressions were used to remove HTML tags, garbled characters, and meaningless symbols from the defense science and technology text data. The defense science and technology text data is divided into several sentences according to punctuation marks, and each sentence is further segmented into several words using word segmentation technology.

Images