Text analysis method based on natural language processing and related device

Abstract

The invention provides a natural language processing-based text analysis method and a related device. The natural language processing-based text analysis method comprises the following steps of: obtaining a first training text comprising preset text contents and a prior distribution position of each core text segmented word in the first training text; according to the prior distribution position of each core text segmented word, performing character mask operation on one or more text segments in the first training text to obtain a text segment mask; carrying out representation vector extraction on the text fragment mask through a BERT model to be adjusted, and obtaining a target text feature vector; performing decoding mapping on the masked text fragment in the text fragment mask according to the target text feature vector through a first reduction mapping model to obtain a decoded text; and according to a first training cost value between the first training text and the decoded text, carrying out iterative optimization on model internal parameters of the BERT model to obtain an optimized BERT model.

Description

Technical Field

[0001] The present application relates to the field of text processing, and in particular to a text analysis method and related devices based on natural language processing. Background Art

[0002] With the rise of deep learning technology, neural network-based text analysis methods have gradually become a hot topic of research. Deep learning models such as recurrent neural networks (RNN), long short-term memory networks (LSTM), and gated recurrent units (GRU) can automatically learn the sequence features and contextual information of text, and have achieved remarkable results in tasks such as text classification, sentiment analysis, and machine translation. However, deep learning models usually require a lot of computing resources and time for training, and have low processing efficiency for large-scale text data. Moreover, deep learning models have poor interpretability, and it is difficult to understand the decision-making process and basis of the model, which is limited in some application scenarios with high interpretability requirements. In addition, in existing text analysis technologies, it is still difficult to accurately locate and analyze the core content in the text. When processing complex text data, it is often difficult to accurately identify and extract the core text segmentation and its distribution, which affects the effect and application value of text analysis. Summary of the invention

[0003] In view of this, the embodiment of the present application provides a text analysis method based on natural language processing and a related device. The technical solution of the present application is implemented as follows: On the one hand, the present application provides a text analysis method based on natural language processing, the method comprising: obtaining a first training text including preset text content and a prior distribution position of each core text segment in the first training text; performing a character masking operation on one or more text segments in the first training text according to the prior distribution position of each core text segment to obtain a text segment mask; extracting a representation vector from the text segment mask through a BERT model to be adjusted to obtain a target text feature vector; performing decoding mapping on the masked text segment in the text segment mask according to the target text feature vector through a first restoration mapping model to obtain a decoded text; iteratively optimizing the model internal parameters of the BERT model according to a first training cost value between the first training text and the decoded text to obtain the optimized BERT model.

[0004] On the other hand, the present application provides a text analysis device, which includes: a data acquisition module, which is used to acquire a first training text including preset text content and a prior distribution position of each core text segment in the first training text; a text masking module, which is used to perform character masking operations on one or more text segments in the first training text according to the prior distribution position of each core text segment to obtain a text segment mask; a feature extraction module, which is used to extract representation vectors from the text segment mask through a BERT model to be adjusted to obtain a target text feature vector; a decoding mapping module, which is used to decode and map the masked text segments in the text segment mask according to the target text feature vector through a first restoration mapping model to obtain a decoded text; a model adjustment module, which is used to iteratively optimize the model internal parameters of the BERT model according to a first training cost value between the first training text and the decoded text to obtain the optimized BERT model.

[0005] In an embodiment of the present disclosure, a first training text including preset text content and a prior distribution position of each core text segment in the first training text are obtained; based on the prior distribution position of each core text segment, a character masking operation is performed on one or more text segments in the first training text to obtain a text segment mask; through a BERT model to be adjusted, a representation vector is extracted for the text segment mask to obtain a target text feature vector; through a first restoration mapping model, a masked text segment in the text segment mask is decoded and mapped based on the target text feature vector to obtain a decoded text; based on a first training cost value between the first training text and the decoded text, the model internal parameters of the BERT model are iteratively optimized to obtain an optimized BERT model. Based on this, firstly, because the text segment mask is obtained by performing a character masking operation on one or more text segments in the first training text according to the prior distribution position of each core text segment, the internal model parameters of the BERT model are iteratively optimized according to the first training cost value between the first training text and the decoded text, guiding the BERT model to learn the semantics associated with the core text segment, so that the features of the core text segment extracted by the BERT model are more accurate, so as to improve the reliability of downstream tasks performed by the optimized BERT model; in addition, the first training text used for debugging the BERT model is a text including preset text content, and the first training text includes the prior distribution position of each core text segment. In addition, in downstream tasks, the training text used for model debugging generally also includes preset text content and the text including the prior distribution position of each core text segment. Therefore, the same training text can be used when pre-calibrating the BERT model and in the model calibration link after migrating the pre-calibrated BERT model to the downstream task, thereby reducing the number of training texts, saving costs and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A schematic diagram of the implementation flow of a text analysis method based on natural language processing provided in an embodiment of the present application.

[0007] Figure 2 A schematic diagram of the composition structure of a text analysis device provided in an embodiment of the present application.

[0008] Figure 3 A hardware entity diagram of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0009] The embodiment of the present application provides a text analysis method based on natural language processing, which can be executed by a processor of a computer system. The computer system can refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities.

[0010] In the field of natural language processing, the positioning of core text segmentation words and semantic association modeling are the basis of text understanding tasks (such as named entity recognition, keyword extraction, semantic role labeling, etc.). Its goal is to accurately identify the distribution positions of core segmentation words with specific semantic functions (such as entity words, action subjects or key descriptive phrases) from a given text and capture the semantic dependencies between them. Traditional methods usually implement text feature extraction based on recurrent neural networks (RNN) or convolutional neural networks (CNN). However, such methods have two limitations: 1. High data dependence: Existing models need to rely on massive general texts (such as Wikipedia, BookCorpus) for pre-training, while the annotated data of vertical fields or specific tasks (such as legal documents, medical reports) is limited, resulting in a large amount of domain data fine-tuning when migrating models, which is costly. 2. Insufficient semantic association modeling: The semantic function of text segmentation not only depends on the local context, but also needs to be combined with global semantic constraints (such as the subordinate relationship between entities and the logical order of action chains). Traditional sequence models have difficulty in effectively modeling long-distance dependencies, and there is a deviation between pre-training tasks (such as masked language modeling) and downstream word segmentation positioning goals, resulting in insufficient semantic understanding of the model for core word segmentations.

[0011] In recent years, Transformer-based pre-trained language models (such as BERT) have significantly improved text representation capabilities through internal attention mechanisms, and have been tried to be applied to word segmentation and localization tasks. However, the direct application of such general models faces challenges: Poor task adaptability: General pre-training tasks (such as random mask prediction) are not optimized for the semantic features of core word segmentations, and the model has difficulty distinguishing the difference between key word segmentations and ordinary vocabulary. Lack of inductive bias: The original Transformer structure lacks explicit modeling of semantic constraints between word segmentations (such as parent-child relationships in dependency trees and entity co-occurrence rules), and needs to rely on additional modules (such as neural networks in the figure) to introduce prior knowledge, which reduces the versatility of the model.

[0012] Based on this, the embodiment of the present application proposes a pre-training and fine-tuning framework centered on core word segmentation. Through a semantically guided masking strategy and a hierarchical attention decoding mechanism, the following are achieved: 1. Reduce data dependence: Use task-related text (such as domain data with annotated word segmentation positions) to directly pre-train the model to avoid relying on a general corpus, while enhancing the model's sensitivity to core word segmentation through semantically driven masking. 2. Enhanced semantic modeling: In the pre-training stage, dynamically mask adjacent text fragments based on word segmentation position priors, forcing the model to recover masked key semantics from the context; in the fine-tuning stage, capture constraints between word segmentations through internal attention and fuse global text features across attention to achieve end-to-end word segmentation positioning optimization.

[0013] While maintaining the universality of the Transformer structure, this solution achieves customized learning of task-related semantics, providing an efficient and lightweight solution for cross-domain text analysis tasks. Figure 1 A flowchart of a text analysis method based on natural language processing provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: the method includes: Step S100: obtaining a first training text including preset text content and a priori distribution position of each core text segment in the first training text; the preset text content is also a preset semantic unit.

[0014] In step S100, a first training text including preset text content and a prior distribution position of each core text segmentation word in the first training text are obtained. The preset text content refers to a preset semantic unit, such as a semantic role, a dependency syntactic center word or a named entity, etc., such as the agent, action, and patient all belong to the semantic unit. The core text segmentation word is a segmentation word with a specific semantic function, such as an entity word, an action subject, or a key descriptive phrase, etc., and the prior distribution position is the actual position of these core text segmentations in the first training text.

[0015] For example, suppose there is a first training text: "The teacher explained the mathematical theorem in class." In this example, the semantic units corresponding to the preset text content can be the agent, action, and patient. Among them, "teacher" is the agent, "explain" is the action, and "mathematical theorem" is the patient. These words are the core text segmentation words. Their prior distribution positions are the 1st to 2nd characters in the text (teacher), the 6th to 8th characters (explain), and the 10th to 14th characters (mathematical theorem).

[0016] When implementing step S100, for obtaining the first training text, data can be read from a local database, a network data source or a file system. If it is obtained from a local database, an SQL query statement can be used to filter out text data containing preset text content. For example, in an education-related database, a table for storing course descriptions is set, and the relevant text can be obtained through an SQL statement. For obtaining the prior distribution position of the core text segmentation, a combination of manual annotation and natural language processing technology can be used. Manual annotation refers to the marking of the first training text by professionals to clearly indicate the position of each core text segmentation. This method has high accuracy, but low efficiency and is suitable for small-scale data. Natural language processing technology can be implemented through some existing tools and algorithms. For example, using a named entity recognition (NER) tool, it can identify named entities in the text, such as names of people, places, names of organizations, etc., and give their location information. For the "teacher" and "mathematical theorem" in the above example, the NER tool can identify them as named entities and determine their positions in the text.

[0017] In addition, you can use dependency syntax analysis tools to analyze the dependency relationship between words in the text, find the dependency syntax core word, and then determine the core text segmentation and its position. Taking "The teacher explained the mathematical theorem in class" as an example, the dependency syntax analysis tool can analyze that "explain" is the core verb of the sentence, "teacher" is its agent, and "mathematical theorem" is its object, and give their position information.

[0018] In practical applications, natural language processing technology can be used to preliminarily process the text to obtain the approximate location of the core text segmentation, and then manually review and correct it to improve the accuracy of the location information. At the same time, in order to improve processing efficiency, parallel computing can be used to process multiple texts at the same time. For example, using a distributed computing framework, large-scale text data can be distributed to multiple computing nodes for processing. Each node is responsible for processing a part of the text, and finally the processing results are summarized.

[0019] Through the above process, the first training text including the preset text content and the prior distribution position of each core text segmentation can be effectively obtained, laying the foundation for the subsequent model training steps. The data obtained in this way can provide accurate information for the model, enabling the model to better learn the semantics associated with the core text segmentation, thereby improving the performance and reliability of the model in text analysis tasks.

[0020] Step S200: performing a character masking operation on one or more text segments in the first training text according to the prior distribution position of each core text segment, to obtain a text segment mask.

[0021] In step S200, according to the prior distribution positions of each core text token in the first training text, character masking operations are performed on one or more text segments in the first training text, thereby obtaining text segment masks. The key to this step is to use the prior distribution positions of core text tokens as a guide to specifically mask text segments, prompting the model to pay more attention to the semantic information of core text tokens and their surroundings during subsequent learning.

[0022] For example, assume the first training text is "The teacher explained the complex mathematical theorem in detail in class, and the students listened attentively". Among them, the prior distribution positions of the core text tokens "teacher", "explained", "mathematical theorem", "students", and "listened" are the 1st - 2nd characters, 7th - 9th characters, 12th - 16th characters, 18th - 21st characters, and 24th - 26th characters in the text respectively. Based on these prior distribution positions, determine which text segments need to be masked.

[0023] When implementing step S200, a sliding window - based method can be adopted. Specifically, set a sliding window with a fixed size, and the window size needs to be determined according to the specific task and data characteristics. For example, if the window size is set to 5 characters, centered on the position of each core text token, a certain length will be extended forward and backward to determine the window range. For the core text token "teacher" whose position is the 1st - 2nd characters, the window range will be from the 1st - 5th characters, and the text segment "The teacher in class" within this range may be masked.

[0024] After determining the window range, mask the characters within the window. The masking method can be to replace the characters with a specific mask symbol, such as "[MASK]". In this way, the original text segment "The teacher in class" will become "[MASK][MASK][MASK][MASK][MASK]". Through this masking operation, the model needs to try to recover the masked content from the remaining text information during subsequent training, which makes the model have to pay attention to the semantic associations of core text tokens and their surroundings, thereby improving the ability to understand the semantics of core text tokens.

[0025] In addition, a semantic - similarity - based method can be adopted. Calculate the semantic similarity between each text segment in the text and the core text token. For example, use a pre - trained word vector model to convert the text segment and the core text token into vector representations, and then measure the semantic similarity by calculating the cosine similarity between the vectors. When the semantic similarity between a certain text segment and the core text token exceeds a certain threshold, mask that text segment.

[0026] For the text in the above example, we found through calculation that the semantic similarity between "detailed explanation" and the core text segmentation word "explain" is very high, so the text segment "detailed explanation" is masked and turned into "[MASK][MASK][MASK][MASK]". This masking operation based on semantic similarity can more accurately locate text segments related to the semantics of the core text segmentation word, allowing the model to learn more effective semantic information.

[0027] In practical applications, the size of the sliding window and the threshold of semantic similarity can be determined through multiple experiments and adjustments to determine the optimal values. By masking the text fragments according to the prior distribution positions of the core text segmentation words, targeted data is provided for subsequent model training. This helps the model better capture the semantic dependencies between core text segmentation words, improves performance and accuracy in text analysis tasks, and lays the foundation for realizing efficient and lightweight cross-domain text analysis solutions.

[0028] Step S300: extracting a representation vector of the text segment mask through the BERT model to be adjusted to obtain a target text feature vector.

[0029] In step S300, the BERT model to be adjusted is used to extract the representation vector of the text segment mask, thereby obtaining the target text feature vector. This step uses the powerful text representation ability of the BERT model to convert the masked text segment into a vector representation with semantic information, so that the subsequent model can perform further analysis and processing based on these vectors.

[0030] For example, suppose the text segment mask obtained after processing in step S200 is "[MASK][MASK][MASK][MASK][MASK] in class, and the students [MASK][MASK][MASK] [MASK][MASK]", which is the result of the mask operation on "The teacher explained the complex mathematical theorem in detail in class, and the students listened attentively". The BERT model to be tuned will be used to process this text segment mask.

[0031] The BERT model is a pre-trained language model based on the Transformer architecture. It can capture the semantic associations between different positions in the text through an internal attention mechanism. When extracting representation vectors, the BERT model first performs word segmentation on the input text fragment mask and splits it into tokens. For the above text fragment mask, it may be segmented into tokens such as "[MASK]", "[MASK]", "in", "classroom", "on", "[MASK]", "[MASK]", "[MASK]", "[MASK]", "[MASK]", "," "students", "[MASK]", "[MASK]", "[MASK]", "ground", "[MASK]", "[MASK]" and other tokens.

[0032] Then, the BERT model assigns a corresponding embedding vector to each word, which contains the semantic information of the word. At the same time, the model considers the position information of the word in the text and adds a position embedding to each word to represent its order in the text sequence. Next, these embedding vectors are input into the multi-layer Transformer encoder of the BERT model for processing. In the Transformer encoder, the internal attention mechanism calculates the correlation between each word and other words to capture the long-distance dependencies in the text. Specifically, the internal attention mechanism determines the degree of attention of each word to other words by calculating the similarity between the three matrices of query (Query), key (Key), and value (Value). Assuming that the query vector of a word is Q and the key vector of other words is K, the similarity between them can be obtained by dot product operation, that is In order to avoid the similarity value being too large, it will be scaled, that is, , where d k is the dimension of the key vector. Then, the similarity value is converted into a probability distribution through the softmax function, that is, Finally, the attention distribution is weighted and summed with the value vector V to obtain the output vector of the word unit, that is, .

[0033] After being processed by the multi-layer Transformer encoder, each word unit will get an updated vector representation. These vector representations are integrated, for example, the vectors of each word unit are concatenated in order to form an overall vector sequence, which is the target text feature vector. For the above text segment mask, the final target text feature vector contains the semantic information of the text segment after mask processing, which can reflect the semantic relationship between the core text segment and its context.

[0034] When implementing step S300, a general deep learning framework, such as PyTorch or TensorFlow, can be used to implement the call and reasoning of the BERT model. When extracting the representation vector, a batch processing method can be used to input multiple text fragment masks into the model for processing at the same time to improve processing efficiency. At the same time, in order to reduce memory usage, a gradient accumulation method can be used to perform a back propagation update model parameters after multiple forward propagations. Through step S300, the text fragment mask can be converted into a target text feature vector, which provides an important data basis for subsequent decoding mapping and model optimization. These target text feature vectors contain the semantic information of the core text segmentation and its context, which helps the model better understand the semantic structure of the text, thereby improving performance and accuracy in text analysis tasks.

[0035] Step S400: using a first restoration mapping model, according to a target text feature vector, decoding and mapping are performed on the masked text segment in the text segment mask to obtain a decoded text.

[0036] In step S400, the masked text segment in the text segment mask is decoded and mapped according to the target text feature vector by the first restoration mapping model (this model can be understood as a text decoder), so as to obtain the decoded text. This step is based on the target text feature vector obtained in step S300, and the first restoration mapping model is used to restore these vector information to specific text content. Its purpose is to test the model's understanding and recovery ability of the masked text segment semantics, and provide a basis for the optimization of subsequent models. For example, it is assumed that the target text feature vector obtained by step S300 is extracted based on the text segment mask "[MASK][MASK] in the classroom [MASK][MASK] [MASK][MASK], and the students [MASK][MASK][MASK] [MASK] [MASK]". The text segment mask is the result of the masking operation of "the teacher explained the complex mathematical theorem in detail in the classroom, and the students listened attentively". The first restoration mapping model will be used to try to restore the masked text segment according to the target text feature vector.

[0037] The core function of the first restoration mapping model is to map the target text feature vector back to the text space. It consists of multiple neural network layers, such as fully connected layers, Softmax layers, etc. The fully connected layer is used to perform a linear transformation on the target text feature vector and adjust the dimension and feature representation of the vector; the Softmax layer is used to convert the linearly transformed vector into a probability distribution to determine the probability of each word in the vocabulary. Assuming that the target text feature vector is X, the weight matrix of the fully connected layer is W, and the bias vector is b, the vector Y obtained after the linear transformation of the fully connected layer can be expressed as Y=XW+b. Then, Y is converted into a probability distribution P through the Softmax function, and the formula is P=softmax(Y), where the softmax function is defined as , is the i-th element in vector Y, and n is the dimension of vector Y.

[0038] During the decoding process, the first restoration mapping model selects the word with the highest probability as the prediction result according to the probability distribution P. For example, for a masked position, the model calculates the probability of each word in the vocabulary. If the word "teacher" has the highest probability, "teacher" is used as the prediction result for that position. All masked positions in the text segment mask are predicted in turn, and finally the decoded text is obtained.

[0039] When implementing step S400, a general deep learning framework, such as PyTorch or TensorFlow, can be used to build and train the first reduction mapping model. During the decoding process, strategies such as greedy search or beam search can be used to select the prediction result. Greedy search selects the word with the highest probability as the prediction result at each time step. This method is simple and efficient, but it may fall into a local optimal solution; beam search retains multiple candidate word units with higher probabilities at each time step, and then comprehensively considers the probabilities and contextual information of these candidate word units in subsequent steps to select the optimal prediction sequence. This method can improve the accuracy of the prediction, but the computational complexity is high.

[0040] In addition, in order to improve the decoding performance of the model, an attention mechanism can be used when training the first restoration mapping model to enhance the model's attention to different parts of the target text feature vector. The attention mechanism can dynamically adjust the model's weights for each element in the target text feature vector based on the contextual information of the masked position, thereby capturing semantic information more accurately.

[0041] Through step S400, the target text feature vector can be restored to the decoded text. This decoded text reflects the model's understanding and recovery ability of the masked text segment semantics. Subsequent steps will compare the decoded text with the original first training text, calculate the training cost value, and thus optimize the internal parameters of the BERT model to improve the model's learning and understanding of the core text segmentation and its semantic relationship.

[0042] Step S500: According to the first training cost value between the first training text and the decoded text, iteratively optimize the model internal parameters of the BERT model to obtain an optimized BERT model.

[0043] In step S500, the model internal parameters of the BERT model are iteratively optimized according to the first training cost value between the first training text and the decoded text to obtain the optimized BERT model. This step is a key link in the entire model training process. Its core purpose is to adjust the parameters of the BERT model so that the model can better learn the semantic information of the core text segmentation from the text and improve the performance of the model in subsequent text analysis tasks.

[0044] For example, assuming that the first training text is "the teacher explained the complex mathematical theorem in detail in class, and the students listened attentively", the decoded text obtained after step S400 is "the professor roughly explained the simple physical principle in the classroom, and the students listened attentively". It is necessary to calculate the first training cost between the two texts.

[0045] The first training cost is measured using a loss function. A feasible loss function is, for example, the cross entropy loss function, which is used to measure the difference between two probability distributions. In text processing, it can measure the difference between the word unit probability distribution predicted by the model and the actual word unit distribution. Assuming that the probability distribution of the actual word unit is p and the word unit probability distribution predicted by the model is q, the calculation formula of the cross entropy loss function L is: , where n is the size of the vocabulary and p i and q i are the probabilities of the real word unit and the predicted word unit at the i-th position in the vocabulary, respectively. In this example, the first training text and the decoded text are converted into corresponding word unit probability distributions, and then the cross entropy loss function is used to calculate the difference between them to obtain the first training cost value.

[0046] After calculating the first training generation value, an optimization algorithm is used to iteratively optimize the internal parameters of the BERT model. Optimization algorithms include wield stochastic gradient descent (SGD) and its variants, such as Adagrad, Adadelta, Adam, etc. Taking the Adam algorithm as an example, it can adaptively adjust the learning rate of each parameter. The Adam algorithm updates the parameters according to the first-order moment estimate (mean) and second-order moment estimate (variance) of the gradient in each iteration. Specifically, the first-order moment estimate of the gradient is first calculated. and the second moment estimate The formulas are and ,in is the gradient of the current iteration, and are hyperparameters, usually set to 0.9 and 0.999 respectively. Then, the first-order moment estimate and the second-order moment estimate are bias-corrected to obtain and Finally, the parameters are updated based on the modified first-order moment estimate and the second-order moment estimate. , the formula is ,in is the learning rate, is a very small constant used to avoid the denominator being zero.

[0047] During the iterative optimization process, the internal parameters of the BERT model are continuously adjusted to gradually reduce the value of the first training generation. Each iteration calculates the gradient based on the current first training generation value and uses the optimization algorithm to update the parameters. As the number of iterations increases, the performance of the model will gradually improve until the preset stop condition is reached, such as the number of iterations reaches the upper limit or the first training generation value converges to a smaller value.

[0048] Through step S500, the internal parameters of the BERT model can be optimized according to the difference between the first training text and the decoded text, so that the model can better learn the semantic information of the core text segmentation and improve the accuracy and reliability of the model in the text analysis task. The optimized BERT model will provide more powerful support for subsequent text analysis tasks.

[0049] As an implementation mode, step S200, based on the prior distribution position of each core text segment, performs a character masking operation on one or more text segments in the first training text to obtain a text segment mask, including: Step S210: performing mobile segmentation on the first training text according to a pre-set text segmentation window to obtain a text segment set; Step S220: determining the target text segment where each core text segment is located in the text segment set according to the prior distribution position of each core text segment; Step S230: for each text segment in the text segment set, determine the masked confidence corresponding to the text segment according to the similarity between the text segment and each target text segment; wherein, the greater the maximum similarity between the text segment and each target text segment, the greater the masked confidence corresponding to the text segment; Step S240: performing a character masking operation on one or more text segments in the first training text according to the masked confidence corresponding to each text segment, to obtain a text segment mask.

[0050] The purpose of steps S210-S240 is to perform character masking operations on one or more text segments in the first training text based on the prior distribution position of each core text segment in the first training text to obtain text segment masks, thereby providing more targeted data for subsequent model training, allowing the model to better learn the semantic information associated with the core text segment.

[0051] In step S210, the first training text is segmented according to the pre-set text segmentation window to obtain a set of text segments. The text segmentation window is a sliding window of fixed length, and its size is set according to the specific task and data characteristics, which is not limited in the embodiment of the present application. Assuming that the first training text is "The teacher explained the complex mathematical theorem in detail through vivid cases in the classroom, and the students took notes seriously", the pre-set text segmentation window size is 5 characters. Starting from the starting position of the text, a text segment of length 5 is intercepted each time, and then the window is moved back by one character, and the interception continues until the window moves to the end of the text. In this way, a series of text segments are obtained, such as "The teacher is in the classroom", "The teacher is in the classroom", "Communication in the classroom", etc., and these text segments constitute a set of text segments. When implementing this step, a technical means of loop traversal can be used, starting from the starting position of the first training text, and interception operations are performed according to a fixed window size and moving step length.

[0052] In step S220, according to the prior distribution positions of each core text word segmentation, the target text segments where each core text word segmentation is located are determined in the text segment set. In the above example, the prior distribution positions of the core text word segmentations "teacher", "explain", and "mathematical theorem" are the 1st - 2nd characters, the 12th - 14th characters, and the 17th - 21st characters respectively. These position information are compared with each text segment in the text segment set to find the text segments containing these core text word segmentations. For "teacher", the target text segment containing it is "teacher in the classroom"; for "explain", the target text segment may be "explained in detail"; for "mathematical theorem", the target text segment may be "complicated mathematical theorem". The target text segments can be determined by traversing the text segment set and checking whether each text segment contains the core text word segmentation.

[0053] In step S230, for each text segment in the text segment set, according to the similarity between the text segment and each target text segment, the masked confidence corresponding to the text segment is determined, where the greater the maximum similarity between the text segment and each target text segment, the greater the masked confidence corresponding to the text segment. Multiple methods can be used to calculate the similarity, such as Mahalanobis distance, Euclidean distance, cosine distance, etc. There is no specific limitation, and the formula can refer to the existing technology and will not be elaborated here. For example, after calculating the Euclidean distance between the text segment and each target text segment, it can be converted into similarity, for example, using the formula S = 1 / (1 + D O ), where S represents similarity and D O is the Euclidean distance. Then, for each text segment, the maximum similarity between it and all target text segments is found, and this maximum similarity is used to determine the masked confidence corresponding to the text segment. Taking the text segment "through vividly above" as an example, after calculating the similarity with each target text segment, the maximum similarity is 0.3, then the masked confidence corresponding to it will be determined according to this 0.3. This step can be implemented by using technical means of vector calculation and comparison, converting the text segment and the target text segment into vectors, and then performing similarity calculation and comparison.

[0054] In step S240, according to the masked confidence corresponding to each text segment, perform a character masking operation on one or more text segments in the first training text to obtain a text segment mask. After determining the masked confidence of each text segment, a threshold can be set. When the masked confidence of a certain text segment exceeds this threshold, a masking operation is performed on this text segment. The masking operation can be to replace the characters in the text segment with a specific mask symbol, such as "[MASK]". For example, if the threshold is set to 0.2 and the masked confidence of "shang tong guo sheng dong" is 0.3, which exceeds the threshold, then "shang tong guo sheng dong" is replaced with "[MASK][MASK][MASK][MASK][MASK]". By performing the masking operation on all eligible text segments, a text segment mask is finally obtained. Conditional judgment and string replacement techniques can be used to implement this step. According to the comparison result of the masked confidence and the threshold, the corresponding text segment is replaced.

[0055] In practical applications, steps S210 - S240 are interrelated and jointly complete the character masking operation on the first training text. Step S210 divides the first training text into multiple text segments, providing a basis for subsequent processing; step S220 determines the target text segments containing the core text segmentation, clarifying the text parts that need to be focused on; step S230 provides a quantitative basis for the masking operation by calculating the similarity and determining the masked confidence; step S240 performs the masking operation according to the masked confidence to obtain the final text segment mask. The implementation of these steps helps to process the first training text more targeted, enabling the model training to better focus on the core text segmentation and its related semantic information, thereby improving the performance and accuracy of the model in text analysis tasks.

[0056] As an implementation manner, step S230 determines the masked confidence corresponding to the text segment according to the similarity between the text segment and each target text segment, including: Step S231: Determine the similarity between the text segment and each target text segment; Step S232: For each target text segment, determine an alternative confidence corresponding to the text segment according to the similarity between the text segment and the target text segment. The greater the similarity between the text segment and the target text segment, the greater the alternative confidence corresponding to the text segment; Step S233: Determine the masked confidence corresponding to the text segment according to the maximum value among each alternative confidence corresponding to the text segment.

[0057] In step S231, the similarity between the text segment and each target text segment is determined. Similarity is an indicator measuring the degree of similarity between two text segments in terms of semantics, structure, etc. Common calculation methods include Mahalanobis distance, Euclidean distance, cosine similarity, etc.

[0058] In step S232, for each target text segment, based on the similarity between the text segment and the target text segment, an alternative confidence level corresponding to the text segment is determined, and the greater the similarity between the text segment and the target text segment, the greater the alternative confidence level corresponding to the text segment. After calculating the similarity between the text segment and each target text segment, it is necessary to convert the similarity into an alternative confidence level. One conversion method is to directly use the similarity as the alternative confidence level. For example, after calculating the Mahalanobis distance between "through vividly above" and "the teacher in the classroom" and converting it into a similarity of 0.2, the corresponding alternative confidence level is 0.2; the similarity with "explained in detail" is 0.3, and the corresponding alternative confidence level is 0.3; the similarity with "complicated mathematical theorem" is 0.1, and the corresponding alternative confidence level is 0.1. Technical means of assignment can be used to directly assign the calculated similarity to the corresponding alternative confidence level.

[0059] In step S233, based on the maximum value among each alternative confidence level corresponding to the text segment, the masked confidence level corresponding to the text segment is determined. After obtaining the alternative confidence levels corresponding to the text segment and each target text segment, the maximum value among these alternative confidence levels will be found and used as the masked confidence level corresponding to the text segment. For the text segment "through vividly above", its alternative confidence levels are 0.2, 0.3, and 0.1 respectively, and the maximum value is 0.3, so the masked confidence level corresponding to "through vividly above" is 0.3. Technical means of comparison and selection can be used to traverse all alternative confidence levels, find the maximum value among them, and determine it as the masked confidence level.

[0060] Through the effective implementation of steps S231 - S233, an appropriate masked confidence level can be determined for each text segment, laying a foundation for accurately performing character masking operations on the first training text subsequently, thereby improving the effect of the model in learning the semantic information of core text word segmentation.

[0061] As an implementation manner, in step S500, after iteratively optimizing the internal model parameters of the BERT model based on the first training cost value between the first training text and the decoded text and obtaining the optimized BERT model, the method further includes: Step S600: obtaining a second training text including a preset text content and a prior distribution position of each core text segmentation in the second training text, and a first core text segmentation vector set, wherein each core text segmentation vector in the first core text segmentation vector set corresponds to a core text segmentation of the preset text content; Step S700: extracting a representation vector from the second training text using the BERT model to obtain a second text feature vector; Step S800: using the first distribution position inference model, based on the first core text word vector set and the second text feature vector, inferring the distribution position of the core text word in the second training text, and obtaining the inferred distribution position of each core text word in the second training text; Step S900: Based on the second training cost value between the prior distribution position of each core text segmentation in the second training text and the inferred distribution position of each core text segmentation in the second training text, optimize one or more of the model internal parameters of the BERT model, the model internal parameters of the first distribution position inference model, and the first core text segmentation vector set to obtain an optimized BERT model, an optimized first distribution position inference model, and an optimized first core text segmentation vector set.

[0062] In step S600, a second training text including preset text content and a prior distribution position of each core text segmentation in the second training text, as well as a first core text segmentation vector set are obtained, and each core text segmentation vector in the first core text segmentation vector set corresponds to a core text segmentation of the preset text content. For example, assuming that the second training text is "The professor demonstrated a complex physical experiment using advanced equipment in the laboratory, and the students carefully observed and recorded", the semantic units corresponding to the preset text content are the agent, action, and patient, and the core text segmentations "professor", "demonstration", and "physical experiment" are the agent, action, and patient, respectively, and their prior distribution positions are the 1st-2nd characters, the 7th-9th characters, and the 13th-17th characters, respectively. The first core text segmentation vector set is a set composed of the core text segmentations such as "professor", "demonstration", and "physical experiment" converted into vector form. The prior distribution positions of the second training text and the core text segmentations can be obtained from a local database, a network data source, or a file system, and the core text segmentations are converted into vectors using a pre-trained word vector model, thereby obtaining a first core text segmentation vector set.

[0063] In step S700, the representation vector of the second training text is extracted through the BERT model optimized in step S500 to obtain a second text feature vector. After preliminary optimization, the BERT model can better capture the semantic information in the text. For the above-mentioned second training text, it is input into the optimized BERT model. The model will perform word segmentation on the text, assign an embedding vector to each word element and consider the position information, and then process it through a multi-layer Transformer encoder, and finally output the second text feature vector. This vector set contains the semantic information of the second training text and can reflect the semantic relationship between the core text segmentation and its context. A deep learning framework, such as PyTorch or TensorFlow, can be used to implement the call and reasoning of the BERT model, and batch processing can be used to improve processing efficiency.

[0064] In step S800, the distribution position of the core text segmentation in the second training text is inferred through the first distribution position reasoning model based on the first core text segmentation vector set and the second text feature vector, and the inferred distribution position of each core text segmentation in the second training text is obtained. The function of the first distribution position reasoning model is to combine the vector information of the core text segmentation and the feature vector of the second training text to infer the position of the core text segmentation in the text. In this process, the model will consider the semantic dependencies between the core text segmentations and their association with the text context. For example, based on the vector information of "professor", "demonstration" and "physical experiment" and the feature vector of the second training text, the model will infer that "professor" is at the beginning of the text, "demonstration" is somewhere in the middle, and "physical experiment" is at the back. A neural network model can be used as the first distribution position reasoning model, and the trained parameters can be used for reasoning calculations.

[0065] In step S900, based on the second training cost value between the prior distribution position of each core text segmentation in the second training text and the inferred distribution position of each core text segmentation in the second training text, one or more of the model internal parameters of the BERT model, the model internal parameters of the first distribution position inference model, and the first core text segmentation vector set are optimized to obtain an optimized BERT model, an optimized first distribution position inference model, and an optimized first core text segmentation vector set. The second training cost value is usually measured using a loss function, such as a cross-entropy loss function. Assuming that the probability distribution of the true core text segmentation distribution position is p, and the probability distribution of the distribution position inferred by the model is q, the calculation formula of the cross-entropy loss function L is: , where n is the number of possible distribution positions. The second training generation value is calculated based on this loss function, and then an optimization algorithm, such as stochastic gradient descent (SGD) or its variants Adagrad, Adadelta, Adam, etc., is used to update the parameters of the BERT model, the first distribution position inference model, and the first core text segmentation vector set. For example, if the inferred distribution position of "professor" is significantly different from the prior distribution position, the value of the loss function will be larger. According to this loss value, the model parameters and vector set are adjusted to make the next inference more accurate. The gradient descent technique can be used to update the parameters and vector set according to the gradient of the loss function, and batch processing and validation set monitoring methods can be used to improve optimization efficiency and avoid overfitting.

[0066] The whole process is continuously iterated, so that the model can more accurately locate the position of core text segmentation when processing different texts, thereby improving the accuracy and reliability in text analysis tasks. At the same time, when implementing these steps, it is necessary to reasonably adjust the structure and parameters of the model according to the specific tasks and data characteristics, and select appropriate loss functions and optimization algorithms. For example, if the data scale is small, a simple model structure and regularization method can be selected to avoid overfitting; if the data distribution is complex, a more powerful optimization algorithm can be used to speed up convergence.

[0067] As an implementation mode, the first distribution position inference model includes a second restoration mapping model based on the Transformer model and a distribution position mapping model. Based on this, step S800, through the first distribution position inference model, according to the first core text segmentation vector set and the second text feature vector, the distribution position of the core text segmentation in the second training text is inferred to obtain the inferred distribution position of each core text segmentation in the second training text, including: Step S810: optimizing the first core text word segmentation vector set through the second restoration mapping model based on the Transformer model and combining the internal attention strategy to obtain the second core text word segmentation vector set, and optimizing the second core text word segmentation vector set based on the cross-attention strategy and the second text feature vector to obtain the first core text word segmentation feature of each core text word segmentation; Step S820: Using the distribution position mapping model, based on each first core text segmentation feature, the distribution position of the core text segment in the second training text is inferred to obtain the inferred distribution position of each core text segment in the second training text.

[0068] In step S810, the first core text segmentation vector set is optimized by combining the second reduction mapping model based on the Transformer model with the internal attention strategy to obtain the second core text segmentation vector set, and the second core text segmentation vector set is optimized according to the second text feature vector based on the cross-attention strategy to obtain the first core text segmentation feature of each core text segmentation. In the example of the education field, it is assumed that the first core text segmentation vector set contains vectors corresponding to "professor", "demonstration" and "physical experiment", and the second text feature vector is extracted by the BERT model from "the professor demonstrated a complex physical experiment in the laboratory using advanced equipment, and the students carefully observed and recorded". The internal attention strategy focuses on the semantic dependency between the vectors in the first core text segmentation vector set, just like analyzing the relationship between the core text segmentations of "professor", "demonstration" and "physical experiment" who initiates the action and who is the action object. Through the internal attention network, the correlation between the vectors is calculated, such as using the query, key and value matrix operations in the attention mechanism. Assuming that the vectors Q, K and V are query, key and value matrices respectively, the similarity sim=QK is first calculated. T , to avoid similarity values ​​being too large, scaling (d k is the key vector dimension), and then the attention distribution is obtained through the softmax function , and finally the weighted sum , in order to optimize the first core text word segmentation vector set to obtain the second core text word segmentation vector set. The cross-attention strategy focuses on the correlation between the second core text word segmentation vector set and the second text feature vector, considering the relationship between the core text word segmentation and the entire text context. The attention mechanism is also used to calculate the correlation between the two, and the second core text word segmentation vector set is optimized again according to the correlation, and finally the first core text word segmentation feature of each core text word segmentation is obtained. This feature integrates the semantic information of the core text word segmentation itself and the text context. The second reduction mapping model can be built using a deep learning framework, and the operation is performed according to the calculation process of the above-mentioned attention mechanism, and the optimization and feature extraction of the vector set are realized through matrix operations and function calls.

[0069] In step S820, the distribution position of the core text segmentation in the second training text is inferred based on each first core text segmentation feature through the distribution position mapping model to obtain the inferred distribution position of each core text segmentation in the second training text. The distribution position mapping model, such as a fully connected network, softmax, a linear mapping layer, etc., is used to map the first core text segmentation feature to a specific position of the text. Taking the fully connected network as an example, it will linearly transform the input first core text segmentation feature and adjust the feature dimension to match the representation of the text position. Assuming that the input first core text segmentation feature vector is X, the weight matrix of the fully connected layer is W, and the bias vector is b, the vector Y=XW+b obtained after the linear transformation of the fully connected layer. Next, use the softmax function to convert Y into a probability distribution, each position corresponds to a probability value, and the position with the largest probability is the inferred distribution position of the core text segmentation. For the first core text segmentation features of "professor", "demonstration", and "physical experiment", after being processed by the distribution position mapping model, their inference positions in the second training text will be obtained respectively, for example, "professor" is determined at the beginning of the text, "demonstration" is somewhere in the middle, and "physical experiment" is at the end. The distribution position mapping model can be constructed using a deep learning framework, and forward propagation calculations can be performed based on the structure and parameters of the model to convert the first core text segmentation features into inference distribution positions.

[0070] Step S810 generates features that integrate internal and contextual semantic information for the core text segmentation, and step S820 accurately infers the position of the core text segmentation in the text based on these features. The entire process makes full use of the attention mechanism of the Transformer model and the mapping ability of the distribution position mapping model, so that the core text segmentation can be located more accurately and the performance of the model in text analysis tasks can be improved. At the same time, when implementing these steps, it is necessary to reasonably adjust the structure and parameters of the model according to the specific tasks and data characteristics, such as the number of attention heads, the number of neurons in the fully connected layer, etc., in order to achieve the best reasoning effect. The model can be optimized through multiple experiments and evaluation of the validation set to continuously improve the accuracy of the distribution position reasoning of the core text segmentation.

[0071] As an implementation method, the second restoration mapping model includes an internal attention network, a cross-attention network and a multi-layer perceptron network. Based on this, step S810 optimizes the first core text word segmentation vector set through the second restoration mapping model based on the Transformer model, combined with the internal attention strategy, to obtain the second core text word segmentation vector set, and optimizes the second core text word segmentation vector set based on the cross-attention strategy and the second text feature vector to obtain the first core text word segmentation feature of each core text word segmentation, including: Step S811: optimizing the first core text word segmentation vector set through the internal attention network and the internal attention strategy to obtain the second core text word segmentation vector set; Step S812: optimizing the second core text word segmentation vector set based on the cross-attention network and the cross-attention strategy according to the second text feature vector to obtain a third core text word segmentation vector set; Step S813: Map each core text word segmentation vector in the third core text word segmentation vector set to a core text word segmentation feature of the corresponding core text word segmentation through a multi-layer perceptron network.

[0072] In step S811, the first core text word segmentation vector set is optimized by the internal attention network and the internal attention strategy to obtain the second core text word segmentation vector set. The internal attention strategy focuses on the semantic dependency between the vectors within the first core text word segmentation vector set. Taking the example of the education field, it is assumed that the first core text word segmentation vector set contains vectors corresponding to "teacher", "explain" and "course content". In a complete semantic scene, "teacher" is the executor of the action of "explain", and "course content" is the object of "explain", and there is a clear semantic dependency between them. Use the internal attention network to capture these relationships. Then, the scaled similarity is converted into an attention distribution using the softmax function, and finally, the attention distribution is weighted and summed with the value vector V to obtain the updated representation of the core text word segmentation vector. By performing such an operation on each vector in the first core text word segmentation vector set, a first correlation matrix can be obtained, which reflects the strength of the correlation between the core text word segmentation vectors. Based on this first correlation matrix, the first core text word segmentation vector set is optimized to obtain the second core text word segmentation vector set. The vectors in this set better reflect the semantic associations between core text segmentations, providing richer and more accurate information for subsequent processing.

[0073] In step S812, the second core text word segmentation vector set is optimized based on the cross-attention strategy and the second text feature vector through the cross-attention network to obtain the third core text word segmentation vector set. The cross-attention strategy focuses on the semantic dependency between the second core text word segmentation vector set and the second text feature vector. Continuing with the above example, the second text feature vector is extracted by the BERT model from the second training text containing "the teacher explained the course content in detail in class", which contains the semantic information of the entire text. The cross-attention network is used to analyze the relationship between the second core text word segmentation vector set and the second text feature vector. Similarly, query, key and value matrices are generated for the second core text word segmentation vector set and the second text feature vector respectively. By calculating the similarity between them, scaling, converting to attention distribution and weighted summation, a second correlation matrix is ​​obtained, which reflects the correlation between the second core text word segmentation vector set and the second text feature vector. The specific calculation process is similar to step S811, except that the input vector set is different. According to the second correlation matrix, the second core text word segmentation vector set is optimized to obtain the third core text word segmentation vector set. The vectors in this set not only contain the semantic associations between the core text segmentations, but also incorporate the semantic information of the core text segmentations and the entire text context, so that the vectors can more accurately reflect the semantic role and position of the core text segmentations in the text. You can use the deep learning framework to build a cross-attention network to achieve matrix operations and vector optimization.

[0074] In step S813, each core text word segmentation vector in the third core text word segmentation vector set is mapped to the core text word segmentation feature of the corresponding core text word segmentation through a multi-layer perceptron network. The multi-layer perceptron network is a feedforward neural network, which is composed of an input layer, a hidden layer and an output layer. It can perform nonlinear transformation on the input vector and extract more advanced features. For each core text word segmentation vector in the third core text word segmentation vector set, the multi-layer perceptron network will input it into the input layer, and after a series of neurons in the hidden layer are calculated and processed by the activation function, the corresponding core text word segmentation feature is finally output in the output layer. The activation function can adopt a common function such as ReLU, whose formula is f(x)=max(0, x), which can introduce nonlinear factors and enhance the expression ability of the network. Through the processing of the multi-layer perceptron network, each core text word segmentation vector is converted into a more representative and discriminative core text word segmentation feature. These features contain the semantic information of the core text word segmentation and the association information with the text context, which provides strong support for the subsequent reasoning of the distribution position of the core text word segmentation in the text through the distribution position mapping model. A deep learning framework can be used to build a multi-layer perceptron network, and parameters such as the number of network layers, number of neurons, and activation functions can be adjusted according to specific tasks.

[0075] In practical applications, the whole process gradually and deeply processes the core text segmentation vector, which enables a better understanding of the semantic role and position of the core text segmentation in the text, and lays a solid foundation for accurately reasoning about the distribution position of the core text segmentation. At the same time, when implementing these steps, it is necessary to reasonably adjust the structure and parameters of the internal attention network, cross-attention network, and multi-layer perceptron network according to the specific tasks and data characteristics to achieve the best processing effect. For example, when adjusting the internal attention network, the number of attention heads can be changed to capture semantic relationships at different levels; when building a multi-layer perceptron network, the number of hidden layers or neurons can be increased to improve the ability of feature extraction. In addition, batch processing can be used to input multiple core text segmentation vectors into the network for processing at the same time to improve processing efficiency. Through continuous optimization and adjustment, the position of the core text segmentation in the text can be more accurately located, and the performance and accuracy of the model in text analysis tasks can be improved.

[0076] As an implementation method, step S811, combining the internal attention strategy, optimizes the first core text word segmentation vector set to obtain the second core text word segmentation vector set, including: Step S8111: extracting the semantic dependency between the core text word segmentation vectors in the first core text word segmentation vector set in combination with the internal attention strategy to obtain a first correlation matrix; Step S8112: Optimize the first core text word segmentation vector set based on the first correlation matrix to obtain a second core text word segmentation vector set.

[0077] In step S8111, combined with the internal attention strategy, the semantic dependency relationship between each core text word segmentation vector in the first core text word segmentation vector set is extracted to obtain the first correlation matrix. The internal attention strategy is based on the attention mechanism in the Transformer architecture, which allows the model to dynamically focus on elements at different positions in the sequence when processing the input sequence. For example, assume that the first core text word segmentation vector set contains three core text word segmentation vectors corresponding to "student", "learning" and "knowledge". At the semantic level, "student" is the executor of the action of "learning", and "knowledge" is the object of "learning", and there is a clear semantic dependency relationship between them.

[0078] In order to extract these semantic dependencies, three matrices, query, key, and value, are generated for each vector in the first core text segmentation vector set. Let the query vector of a core text segmentation vector be Q, and the key vector of other core text segmentation vectors be K. First, the similarity between them is calculated by dot product operation. In order to avoid the similarity value being too large, it is scaled. After scaling, the scaled similarity is converted into attention distribution using the softmax function. The softmax function maps all similarity values ​​to the interval [0, 1] and makes the sum of all values ​​1, so that each value can represent the degree of attention of the vector to other vectors. By performing the above operation on each vector in the first core text segmentation vector set, a matrix can be obtained. Each element in this matrix represents the degree of attention of a core text segmentation vector to another core text segmentation vector, that is, the correlation between them. This matrix is ​​the first correlation matrix, which comprehensively reflects the semantic dependency between the vectors in the first core text segmentation vector set. For example, in the example of "student", "learning" and "knowledge", the first correlation matrix may show that the correlation between the "student" vector and the "learning" vector is high because "student" is the subject of "learning"; the correlation between the "learning" vector and the "knowledge" vector is also high because the object of "learning" is "knowledge".

[0079] In step S8112, the first core text word segmentation vector set is optimized according to the first correlation matrix to obtain the second core text word segmentation vector set. The first correlation matrix reflects the semantic dependency between the core text word segmentation vectors, and this matrix is ​​used to adjust each vector in the first core text word segmentation vector set to better reflect these semantic relationships.

[0080] For each vector in the first core text word segmentation vector set, the value vector V of other vectors is weighted summed according to the correlation between the vector and other vectors in the first correlation matrix. After all vectors in the first core text word segmentation vector set are updated in this way, the second core text word segmentation vector set is obtained. The vectors in this set more accurately reflect the semantic dependency relationship between each core text word segmentation, providing more valuable information for subsequent cross-attention network processing and the final core text word segmentation distribution position reasoning.

[0081] When implementing this step, the deep learning framework can also be used to update the vector through matrix multiplication and vector operations. The first correlation matrix and the value vector of the first core text word segmentation vector set are used as input, and the calculation is performed according to the above update formula to output the second core text word segmentation vector set.

[0082] As an implementation mode, in step S812, based on the cross-attention strategy and according to the second text feature vector, the second core text word segmentation vector set is optimized to obtain a third core text word segmentation vector set, including: Step S8121: Based on the cross-attention strategy, the semantic dependency relationship between the second core text word segmentation vector set and the second text feature vector is extracted to obtain a second correlation matrix; Step S8122: Optimize the second core text word segmentation vector set based on the second correlation matrix to obtain a third core text word segmentation vector set.

[0083] Steps S8121-S8122 are the key links for optimizing the second core text word segmentation vector set through the cross-attention network to obtain the third core text word segmentation vector set. The cross-attention network captures the semantic dependency between the second core text word segmentation vector set and the second text feature vector, so that the core text word segmentation vector can fully integrate the text context information, providing strong support for the subsequent accurate reasoning of the distribution position of the core text word segmentation in the text.

[0084] In step S8121, the semantic dependency relationship between the second core text word segmentation vector set and the second text feature vector is extracted based on the cross-attention strategy to obtain a second correlation matrix. The cross-attention strategy is a mechanism used in the Transformer architecture to handle the relationship between different sequences, allowing the model to focus on relevant information in another sequence when processing one sequence. For example, assuming that the second core text word segmentation vector set contains vectors corresponding to "teacher", "teaching", and "course content", the second text feature vector is extracted by the BERT model from "the teacher teaches vividly at the podium and explains the rich course content". At the semantic level, "teacher" is related to the subject described in the text, "teaching" is related to the action description in the text, and "course content" is related to the specific objects mentioned in the text. They have a close semantic dependency relationship with the second text feature vector.

[0085] In order to extract these semantic dependencies, query, key, and value matrices are generated for the second core text segmentation vector set and the second text feature vector. Let the query vector of a vector in the second core text segmentation vector set be Q, and the key vector of the second text feature vector be K. First, the similarity between them is calculated by dot product operation. In order to avoid the similarity value being too large, it is scaled. After scaling, the scaled similarity is converted into attention distribution using the softmax function. The softmax function maps all similarity values ​​to the interval [0, 1] and makes the sum of all values ​​1, so that each value can represent the degree of attention of the second core text segmentation vector to each part of the second text feature vector.

[0086] By performing the above operation on each vector in the second core text word segmentation vector set, a matrix can be obtained. Each element in this matrix represents the degree of attention of a second core text word segmentation vector to a certain part of the second text feature vector, that is, the correlation between them. This matrix is ​​the second correlation matrix, which comprehensively reflects the semantic dependency between the second core text word segmentation vector set and the second text feature vector. For example, in the example of "teacher", "teaching", and "course content", the second correlation matrix may show that the "teacher" vector has a high correlation with the part of the second text feature vector that describes the subject, the "teaching" vector has a high correlation with the part that describes the action, and the "course content" vector has a high correlation with the part that describes the object.

[0087] In step S8122, the second core text word segmentation vector set is optimized according to the second correlation matrix to obtain a third core text word segmentation vector set. The second correlation matrix reflects the semantic dependency between the second core text word segmentation vector set and the second text feature vector, and this matrix is ​​used to adjust each vector in the second core text word segmentation vector set to better integrate the text context information.

[0088] For each vector in the second core text word segmentation vector set, a weighted sum is performed on the value vector V of the second text feature vector according to the correlation between the vector and the second text feature vector in the second correlation matrix. Each vector will incorporate the relevant part of the information in the second text feature vector to obtain an updated vector.

[0089] Taking the "teacher" vector as an example, according to the second correlation matrix, if it has a high correlation with the part of the subject described in the second text feature vector, then when updating the "teacher" vector, the information of this part of the feature vector will be considered more. After performing such an update operation on all vectors in the second core text word segmentation vector set, the third core text word segmentation vector set is obtained. The vectors in this set more accurately reflect the semantic association between the core text word segmentation and the text context, and provide more valuable information for subsequent multi-layer perceptron network processing and the final core text word segmentation distribution position reasoning. The deep learning framework can also be used to update the vector through matrix multiplication and vector operations. Take the second correlation matrix and the value vector of the second text feature vector as input, calculate according to the above update formula, and output the third core text word segmentation vector set.

[0090] Through these two steps, we can have a deeper understanding of the semantic relationship between core text segmentation and text context, and improve the accuracy of the model's reasoning about the distribution position of core text segmentation in text analysis tasks.

[0091] As an implementation mode, the distribution position mapping model includes a first mapping model and a second mapping model, and the inferred distribution position includes a first distribution situation in the text sequence and a second distribution situation in the semantic level. Based on this, step S820, through the distribution position mapping model, according to each first core text segmentation feature, the distribution position of the core text segmentation in the second training text is inferred to obtain the inferred distribution position of each core text segmentation in the second training text, including: Step S821: using the first mapping model, based on the features of each core text segmentation, inferring the distribution position of the core text segmentation in the second training text in the text sequence, and obtaining a first distribution situation of each core text segmentation in the second training text; Step S822: Using the second mapping model, based on the features of each core text segmentation, infer the distribution position of the core text segmentation in the second training text in the semantic hierarchy to obtain a second distribution of each core text segmentation in the second training text.

[0092] In the implementation of step S820, steps S821-S822 are specific operation steps of inferring the distribution position of the core text segmentation in the second training text according to the features of each core text segmentation through the distribution position mapping model, so as to obtain the inferred distribution position of each core text segmentation in the second training text. The distribution position mapping model is composed of a first mapping model and a second mapping model. The inferred distribution position includes a first distribution in the text sequence and a second distribution in the semantic level. These two steps infer the distribution position of the core text segmentation from different angles.

[0093] In step S821, the first mapping model is used to infer the distribution position of the core text segmentation in the text sequence of the second training text based on each core text segmentation feature, and the first distribution of each core text segmentation in the second training text is obtained. The main function of the first mapping model is to associate the core text segmentation features with the text sequence position of the second training text, so as to determine the specific position of the core text segmentation in the text. For example, assuming that the second training text is "In class, the teacher carefully explained the mathematical knowledge, and the students actively thought about it", the core text segmentation is "teacher", "explanation", "mathematical knowledge", and their respective core text segmentation features are obtained through the previous steps. The first mapping model will receive these core text segmentation features as input.

[0094] The first mapping model can usually adopt structures such as a fully connected network and a linear mapping layer. A fully connected network is a neural network structure in which each neuron is connected to all neurons in the previous layer. Assuming that the input core text segmentation feature vector is X, the weight matrix of the fully connected layer is W, and the bias vector is b, the vector Y obtained after the linear transformation of the fully connected layer can be expressed as Y=XW+b. This linear transformation maps the core text segmentation feature vector to a new vector space whose dimension is related to the length of the second training text. Then, the vector Y is converted into a probability distribution through the softmax function, with the formula P=softmax(Y). This probability distribution represents the probability that each position is the position where the core text segmentation appears.

[0095] For the core text segmentation word "teacher", the first mapping model will calculate the probability of occurrence at each position in the second training text based on its core text segmentation features. The position with the highest probability is the inferred distribution position of "teacher" in the text sequence, which is the first distribution situation. For example, the calculation result may show that the probability is highest at the 3rd-4th character position of the text, so the first distribution situation of "teacher" in the text sequence is the 3rd-4th character position.

[0096] In step S822, the distribution position of the core text segmentation in the second training text in the semantic hierarchy is inferred based on each core text segmentation feature through the second mapping model to obtain the second distribution of each core text segmentation in the second training text. The distribution position in the semantic hierarchy focuses on the role and hierarchical relationship of the core text segmentation in the semantic structure, such as agent, action, patient, etc. The role of the second mapping model is to associate the core text segmentation features with these semantic hierarchy information.

[0097] Taking "In class, the teacher carefully explained the mathematical knowledge, and the students actively thought about it" as an example, "teacher" is the agent in the semantic hierarchy, "explanation" is the action, and "mathematical knowledge" is the object. The second mapping model receives the core text segmentation features as input, and determines the position of each core text segmentation in the semantic hierarchy through a series of calculations and mappings. The second mapping model can also adopt a structure of a fully connected network, a linear mapping layer and a softmax function. Assume that the input core text segmentation feature vector is X', the weight matrix of the second mapping model is W', and the bias vector is b'. After linear transformation, the vector Y'=X'W'+b' is obtained, and then it is converted into a probability distribution P'=softmax(Y') through the softmax function. This probability distribution represents the probability that each core text segmentation belongs to a different semantic level.

[0098] For the core text segmentation word "explanation", the second mapping model will calculate the probability of it belonging to different semantic levels such as agent, action, and patient based on its core text segmentation features. The semantic level with the highest probability is the inferred distribution position of "explanation" in the semantic level, which is the second distribution situation. For example, the calculation results show that the probability of "explanation" belonging to the action semantic level is the highest, so the second distribution situation of "explanation" in the semantic level is the action level. The second mapping model can be constructed using a deep learning framework, with the core text segmentation features as input, and the model's calculation outputs the inferred distribution position of each core text segmentation in the semantic level.

[0099] Through these two steps, the distribution positions of the core text segmentation words in the second training text can be inferred more comprehensively and accurately, providing richer and more accurate information for subsequent text analysis tasks.

[0100] Through the above steps, the present application also provides a text analysis method using the above model, which specifically includes the following steps: Step S10: obtaining a target text including preset text content and a fourth core text word segmentation vector set set in advance, wherein each core text word segmentation vector in the fourth core text word segmentation vector set corresponds to a core text word segmentation of the preset text content; Step S20: extracting a representation vector from the target text through the optimized BERT model to obtain a third text feature vector; wherein the BERT model is obtained by training based on the above-mentioned embodiment; Step S30: Using the optimized second distribution position inference model, based on the fourth core text word segmentation vector set and the third text feature vector, the positions of the core text words in the target text are inferred to obtain the inferred distribution position of each core text word in the target text.

[0101] In step S10, a target text including preset text content and a fourth core text segmentation vector set set in advance are obtained, wherein each core text segmentation vector in the fourth core text segmentation vector set corresponds to a core text segmentation of the preset text content. The preset text content refers to a text containing a specific semantic unit, and the core text segmentation is a vocabulary with a key semantic function. In the field of education, assuming that the target text is "the researcher used professional knowledge in the laboratory to carry out cutting-edge physical experiments, and the students watched and learned on the side", the semantic units corresponding to the preset text content are the agent, action, and object, and the core text segmentation "researcher", "carry out", and "physical experiment" correspond to the agent, action, and object respectively. The fourth core text segmentation vector set is a set composed of these core text segmentations converted into vector form. The target text can be obtained from various data sources, such as local databases, network resources, or file systems, and the core text segmentation is converted into a vector using a pre-trained word vector model, such as Word2Vec, GloVe, etc., to obtain the fourth core text segmentation vector set.

[0102] In step S20, the representation vector of the target text is extracted through the optimized BERT model to obtain the third text feature vector. The optimized BERT model is obtained through iterative optimization in the previous training phase, and it can capture the semantic information in the text more accurately. For the above target text, it is input into the optimized BERT model. The BERT model will first perform word segmentation on the target text, split it into individual word units, assign corresponding embedding vectors to each word unit, and consider the position information of the word unit in the text to add position embedding. Then, these embedding vectors will be input into the multi-layer Transformer encoder of the BERT model for processing. In the Transformer encoder, the internal attention mechanism calculates the correlation between each word unit and other word units to capture the long-distance dependency in the text. After being processed by the multi-layer Transformer encoder, each word unit will get an updated vector representation, and these vector representations will be spliced ​​together in order to form an overall vector sequence. This vector sequence is the third text feature vector, which contains the semantic information of the target text and can reflect the semantic relationship between the core text word segmentation and its context.

[0103] In step S30, the optimized second distribution position inference model is used to infer the position of the core text segmentation in the target text based on the fourth core text segmentation vector set and the third text feature vector, and the inferred distribution position of each core text segmentation in the target text is obtained. The optimized second distribution position inference model is obtained after multiple optimizations in the training stage. It can accurately infer the position of the core text segmentation in the text based on the core text segmentation vector and the text feature vector. The reasoning process of this model is similar to the first distribution position inference model in the training stage, and may include a reduction mapping model based on the Transformer model and a distribution position mapping model. First, the reduction mapping model based on the Transformer model will combine the internal attention strategy and the cross-attention strategy to optimize the fourth core text segmentation vector set, and interact with the third text feature vector to extract the features of the core text segmentation. Then, the distribution position mapping model will infer the distribution position of the core text segmentation in the target text based on these features. For example, for the core text words "researcher", "carry out", and "physical experiment", the model will infer that "researcher" is at the beginning of the text, "carry out" is somewhere in the middle, and "physical experiment" is at the end based on the fourth core text word vector set and the third text feature vector. The optimized second distribution position inference model can be built using a deep learning framework, with the fourth core text word vector set and the third text feature vector as input, and the inference distribution position of each core text word in the target text can be output through the forward propagation calculation of the model.

[0104] The embodiment of the present application also includes a derivative scheme for event recognition of the target text, which specifically includes the following steps: Step S1000: obtaining text segments to be recognized corresponding to a plurality of target text segments in a target text, wherein the target text is a text describing a target event; Step S1100: performing event recognition on each to-be-recognized text segment respectively through an event recognition model to obtain an abnormal event confidence of each to-be-recognized text segment, where the abnormal event confidence represents the confidence that the to-be-recognized text segment includes an abnormal event; Step S1200: determining an event recognition result of the target text at least according to the confidence level of the abnormal event of each text segment to be recognized, the event recognition result including the abnormal event or not including the abnormal event.

[0105] In step S1000, the target text is obtained to obtain the text segments to be identified that correspond to the multiple target text segments in the target text, where the target text is a text that describes the target event. The target text can be divided into multiple text segments first, and then these text segments are subjected to preliminary event recognition to obtain the event recognition results of each text segment. Based on these results, multiple target text segments are determined from all the text segments, and the corresponding text segments to be identified are selected from the target text segments. For example, assuming that the target text is a report on campus activities, the report is first divided into multiple text segments by sentences or paragraphs, and then each segment is preliminarily judged, such as judging whether the segment involves key information of the activity. If a segment mentions the performance link in the activity, and the link is considered to be related to the subsequent abnormal event recognition, then this segment may be determined as a target text segment, and then the text segment to be identified is selected from it. In order to achieve this step, text segmentation technology in natural language processing can be used, such as rule-based segmentation or machine learning-based segmentation method.

[0106] In step S1100, event recognition is performed on each text segment to be recognized by the event recognition model, and the confidence of the abnormal event of each text segment to be recognized is obtained, and the confidence represents the confidence that the text segment to be recognized includes an abnormal event. In the specific implementation, the text segment to be recognized is first represented by text features to obtain an event representation vector. This can be converted into a vector form by techniques such as word embedding and sentence embedding. For example, each word in the text is mapped to a vector using a pre-trained word vector model, and then these word vectors are combined into a sentence or paragraph vector by a certain method. Next, the event representation vector is classified and mapped to obtain a first confidence that the text segment to be recognized includes an abnormal event and a second confidence that the event is included. Classification mapping can use classification algorithms in machine learning, such as support vector machines, neural networks, etc. Finally, the confidence of the abnormal event is determined based on the first confidence or the second confidence. For example, if a neural network is used for classification mapping, the output layer of the network can have two nodes, which respectively represent the probability of including an abnormal event and the probability of including an event, and the final confidence of the abnormal event can be determined based on these two probability values. Assume that a text segment to be recognized describes a sudden equipment failure during a campus activity. Through the above steps, the confidence that the segment includes an abnormal event is 0.8, which means that there is an 80% probability that the segment contains an abnormal event.

[0107] In step S1200, at least according to the abnormal event confidence of each text segment to be identified, the event recognition result of the target text is determined, and the result is including abnormal events or not including abnormal events. First, multiple categories of detection indicators are obtained, wherein one or more categories of detection indicators are determined according to the abnormal event confidence of the text segment to be identified. These detection indicators may include first-class detection indicators and / or second-class detection indicators. The first-class detection indicator may include the number of target text segments, the first indicator of the event classification confidence of all text segments to be identified, and the second indicator of the abnormal event confidence of all text segments to be identified, etc. The event classification confidence characterizes the confidence of the event of the set event type in the text segment to be identified; the second-class detection indicator may be the percentage of the number of text segments to be identified that may include abnormal events to the total number of target text segments, and the text segments to be identified that may include abnormal events are determined based on comparing the abnormal event confidence of the text segments to be identified with the reference confidence. For example, assuming that there are 10 target text segments in the target text, and the abnormal event confidence of 3 text segments to be identified exceeds the reference confidence of 0.5, then the second-class detection indicator is 30%.

[0108] Determine the event recognition result of the target text based on multiple categories of detection indicators. When multiple categories of detection indicators include the first category of detection indicators and the second category of detection indicators, if both the first category of detection indicators and the second category of detection indicators meet the corresponding event recognition requirements, or the first category of detection indicators or the second category of detection indicators meet the corresponding event recognition requirements, determine that the event recognition result of the target text does not include abnormal events. For different detection indicators, there are different evaluation methods. When the second category of detection indicators is a percentage, based on evaluating whether the percentage meets the set percentage interval, determine whether the second category of detection indicators meets the corresponding event recognition requirements; when the first category of detection indicators includes the second indicator, based on evaluating whether the second indicator is in the first set indicator interval, determine whether the first category of detection indicators meets the corresponding event recognition requirements; when the first category of detection indicators includes the first indicator, based on evaluating whether the first indicator is in the second set indicator interval, determine whether the first category of detection indicators meets the corresponding event recognition requirements; when the first category of detection indicators includes the first indicator, based on evaluating whether the first indicator is in the second set indicator interval, determine whether the first category of detection indicators meets the corresponding event recognition requirements; when the first category of detection indicators includes the number of target text segments, based on evaluating whether the number of target text segments is in the third set indicator interval, determine whether the first category of detection indicators meets the corresponding event recognition requirements. For example, the percentage interval of the second type of detection indicators is set to [0, 20%]. If the calculated percentage is 30%, the second type of detection indicators do not meet the corresponding event recognition requirements.

[0109] As an implementation mode, step S1200 determines the event recognition result of the target text at least according to the confidence of the abnormal event of each text segment to be recognized, including: Step S1210: acquiring detection indicators of multiple categories, wherein the detection indicators of one or more categories are determined according to the confidence of the abnormal event of the text segment to be identified; Step S1220: Determine the event recognition result of the target text based on the detection indicators of multiple categories, wherein when the detection indicators of any category meet the corresponding event recognition requirements, the event recognition result of the target text does not include abnormal events, or when the detection indicators of all categories meet the corresponding event recognition requirements, the event recognition result of the target text does not include abnormal events.

[0110] In step S1210, multiple categories of detection indicators are obtained, wherein one or more categories of detection indicators are determined based on the confidence of the abnormal event of the text segment to be identified. These detection indicators are the key basis for evaluating whether the target text contains abnormal events, and different categories of detection indicators reflect the characteristics of the events in the target text from different angles.

[0111] The detection indicators of multiple categories obtained may include first-class detection indicators and / or second-class detection indicators. The first-class detection indicators include the number of target text segments, the first indicator of the event classification confidence of all the text segments to be identified, and the second indicator of the abnormal event confidence of all the text segments to be identified. The event classification confidence characterizes the confidence of the event that the text segment to be identified includes the set event type. For example, in the report of campus activities, the set event type can be performance, competition, etc., and the event classification confidence of a certain text segment to be identified about the performance link is the possibility of the segment describing the performance event. The second-class detection indicator is the percentage of the number of text segments to be identified that may include abnormal events to the total number of target text segments. The text segments to be identified that may include abnormal events are determined based on comparing the abnormal event confidence of the text segments to be identified with the reference confidence. For example, if the reference confidence is 0.5 and the abnormal event confidence of a text segment to be identified is 0.6, then the segment is considered to be likely to include an abnormal event.

[0112] In order to obtain these detection indicators, statistical analysis methods can be used. The number of target text segments can be counted directly; the event classification confidence and abnormal event confidence can be counted and calculated through the probability values ​​output by the event recognition model. For example, if there are 10 text segments to be recognized, and the abnormal event confidence of 3 of them exceeds the reference confidence of 0.5, then the second category of detection indicators, i.e., the percentage, is 30%.

[0113] After obtaining detection indicators of multiple categories, in step S1220, the event recognition result of the target text is determined based on these detection indicators. A judgment is made based on whether the detection indicators of different categories meet the corresponding event recognition requirements. When the detection indicators of any category meet the corresponding event recognition requirements, or when the detection indicators of all categories meet the corresponding event recognition requirements, the event recognition result of the target text is determined to include no abnormal events. Different categories of detection indicators have different event recognition requirements. For example, for the second category of detection indicators, if the percentage interval is set to [0, 20%], when the calculated percentage is 30%, the detection indicator does not meet the requirements; for the second indicator in the first category of detection indicators, if the first set indicator interval is set to [0.1, 0.3], when the calculated second indicator value is 0.4, the indicator does not meet the requirements.

[0114] By comprehensively evaluating detection indicators of multiple categories, it is possible to comprehensively and accurately determine whether the target text contains abnormal events, providing a reliable basis for subsequent decision-making and processing.

[0115] As an embodiment, multiple categories of detection indicators include first-category detection indicators and / or second-category detection indicators; wherein, the first-category detection indicators include one or more of the first indicators of the number of target text segments and the event classification confidence of all text segments to be identified, and the second indicator of the abnormal event confidence of all text segments to be identified, and the event classification confidence characterizes the confidence that the text segments to be identified include events of set event types; and / or, the second-category detection indicators are the percentage of the number of text segments to be identified that may include abnormal events to the total number of target text segments, and the text segments to be identified that may include abnormal events are determined based on comparing the abnormal event confidence of the text segments to be identified with the reference confidence.

[0116] When performing event recognition on the target text, multiple categories of detection indicators are used, including first-category detection indicators and / or second-category detection indicators, which provide a basis for judging whether the target text contains abnormal events from different dimensions. The first category of detection indicators includes one or more of the first indicators of the number of target text segments, the event classification confidence of all the text segments to be recognized, and the second indicator of the abnormal event confidence of all the text segments to be recognized. The event classification confidence is used to characterize the confidence of the events of the set event type included in the text segments to be recognized, which reflects the degree of association between the text segments and the specific event types. Machine learning classification algorithms, such as support vector machines, naive Bayes classifiers, etc., can be used to train and classify text segments to obtain event classification confidence. The first indicator of the event classification confidence of all the text segments to be recognized can be statistics such as the average and median of these confidences. By analyzing these statistics, the overall distribution of various types of events in the text can be understood.

[0117] The second indicator of the confidence of abnormal events of all text segments to be identified is a comprehensive measure of the possibility of abnormal events in the text. The confidence of abnormal events refers to the confidence that the text segment to be identified includes abnormal events. Each text segment to be identified is analyzed through the event recognition model to obtain the probability that the segment contains abnormal events. The number of target text segments is also part of the first type of detection indicators, which intuitively reflects the size of the target text after being segmented. The second type of detection indicator is the percentage of the number of text segments to be identified that may include abnormal events to the total number of target text segments. The text segments to be identified that may include abnormal events are determined based on comparing the confidence of abnormal events of the text segments to be identified with the reference confidence. The reference confidence is a pre-set threshold. When the confidence of abnormal events of the text segment to be identified exceeds this threshold, the segment is considered to be likely to include abnormal events.

[0118] As an implementation mode, when multiple categories of detection indicators include second-category detection indicators, and the second-category detection indicators are percentages, whether the second-category detection indicators meet the corresponding event identification requirements is determined based on whether the evaluation percentage is in a set percentage interval; when multiple categories of detection indicators include first-category detection indicators, and the first-category detection indicators include second indicators, whether the first-category detection indicators meet the corresponding event identification requirements is determined based on whether the evaluation second indicators are in a first set indicator interval; when multiple categories of detection indicators include first-category detection indicators, and the first-category detection indicators include first indicators, whether the first-category detection indicators meet the corresponding event identification requirements is determined based on whether the evaluation first indicators are in a second set indicator interval; when multiple categories of detection indicators include first-category detection indicators, and the first-category detection indicators include the number of target text segments, whether the first-category detection indicators meet the corresponding event identification requirements is determined based on whether the evaluation number of target text segments is in a third set indicator interval.

[0119] As an implementation manner, the multiple categories of detection indicators include first category detection indicators and second category detection indicators; step S1220, determining the event recognition result of the target text according to the multiple categories of detection indicators, includes: Step S1221: When both the first category detection indicators and the second category detection indicators meet the corresponding event recognition requirements, the event recognition result of the target text is determined to include no abnormal events; or, when the first category detection indicators or the second category detection indicators meet the corresponding event recognition requirements, the event recognition result of the target text is determined to include no abnormal events.

[0120] The first type of detection indicators include the number of target text segments, the first indicator of event classification confidence of all text segments to be identified, and the second indicator of abnormal event confidence of all text segments to be identified, etc. These indicators reflect the characteristics of events in the target text from different aspects. The second type of detection indicators is the percentage of the number of text segments to be identified that may include abnormal events to the total number of target text segments. Each detection indicator has a corresponding event recognition requirement, that is, a pre-set reasonable range, such as a set percentage interval, a set indicator interval, etc.

[0121] As an implementation manner, the number of the first type of detection indicators is one or more; wherein, in step S1221, when the first type of detection indicators or the second type of detection indicators meet the corresponding event recognition requirements, determining that the event recognition result of the target text does not include abnormal events includes: Step S12211: evaluating whether each first-category detection indicator meets the event identification requirements corresponding to each first-category detection indicator; Step S12212: when each of the first-category detection indicators meets the event recognition requirements corresponding to each of the first-category detection indicators, determining that the event recognition result of the target text does not include an abnormal event; Step S12213: when any of the first-category detection indicators does not meet the corresponding event identification requirements, evaluating whether the second-category detection indicators meet the event identification requirements corresponding to the second-category detection indicators; Step S12214: When the second-category detection indicator meets the event recognition requirement corresponding to the second-category detection indicator, it is determined that the event recognition result of the target text does not include abnormal events.

[0122] In step S12211, it is evaluated whether each first-category detection indicator meets the event recognition requirements corresponding to each first-category detection indicator. The first-category detection indicators include the number of target text segments, the first indicator of the event classification confidence of all text segments to be recognized, and the second indicator of the abnormal event confidence of all text segments to be recognized. Each first-category detection indicator has a corresponding event recognition requirement, that is, a pre-set reasonable range, such as a set indicator interval, a set percentage interval, etc.

[0123] If it is determined in step S12211 that each of the first-category detection indicators meets the event recognition requirements corresponding to each of the first-category detection indicators, then according to step S12212, it will be determined that the event recognition result of the target text does not include abnormal events.

[0124] However, if in step S12211, it is found that any of the first-class detection indicators does not meet the corresponding event identification requirements, step S12213 is entered to evaluate whether the second-class detection indicators meet the event identification requirements corresponding to the second-class detection indicators. The second-class detection indicator is the percentage of the number of text segments to be identified that may include abnormal events to the total number of target text segments. The event identification requirement corresponding to this indicator is also a pre-set reasonable range, for example, the percentage interval is set to 0-10%. Assume that when evaluating the first-class detection indicator, it is found that the average value of the abnormal event confidence of all text segments to be identified exceeds the reasonable range, but at this time, the percentage of the number of text segments to be identified that may include abnormal events to the total number of target text segments will be further calculated. If the percentage is calculated to be 8%, within the set percentage interval of 0-10%, then the second-class detection indicator meets the corresponding event identification requirements; if the percentage is calculated to be 15%, which exceeds the range of 0-10%, then the second-class detection indicator does not meet the corresponding event identification requirements. The evaluation of the second type of detection indicators is completed by first counting the number of text segments to be identified that may include abnormal events and the total number of target text segments, then calculating the percentage of the two and comparing it with the set percentage range.

[0125] When it is determined in step S12213 that the second category of detection indicators meets the event identification requirements corresponding to the second category of detection indicators, according to step S12214, it will be determined that the event identification results of the target text do not include abnormal events. In other words, even if there are situations in the first category of detection indicators that do not meet the requirements, if the second category of detection indicators performs well and is within a reasonable range, then it is still considered that the target text as a whole does not contain abnormal events. For example, in a project progress report text, the number of target text segments in the first category of detection indicators exceeds a reasonable range, but the percentage of the number of text segments to be identified that may include abnormal events to the total number of target text segments is within the set percentage range, and it will be determined that the target text does not include abnormal events.

[0126] As an implementation manner, step S1000, obtaining the to-be-recognized text segments respectively corresponding to the plurality of target text segments in the target text, includes: Step S1010: performing event recognition on each text segment included in the target text to obtain an event recognition result for each text segment; Step S1020: determining a plurality of target text segments from all the text segments according to the event recognition results of the respective text segments, and selecting corresponding text segments to be recognized from the target text segments.

[0127] In step S1010, event recognition is performed on each text segment included in the target text to obtain the event recognition results of each text segment. Then, in step S1020, multiple target text segments are determined from all text segments based on the event recognition results of each text segment, and corresponding text segments to be recognized are selected from the target text segments. The target text segments are those text segments that are closely related to the subsequent event recognition tasks, and the segments that meet specific conditions will be screened out as target text segments according to the event recognition results. Then, among these target text segments, the corresponding text segments to be recognized are further selected. The text segments to be recognized are text segments used for subsequent detailed analysis and judgment of the confidence of abnormal events, and the selection method can be based on factors such as the importance of the segment and the significance of the event. For example, in the determined target text segment, the most detailed and critical part describing the moment of scoring is selected as the text segment to be recognized. The determination of the target text segment and the selection of the text segment to be recognized can be completed by setting thresholds, sorting, etc. For example, based on the confidence of the event in the event recognition result, a text segment with a confidence higher than a certain threshold is selected as the target text segment, and then a segment with moderate length and rich information is selected from the target text segment as the text segment to be recognized.

[0128] As an implementation mode, in step S1010, event recognition is performed on each to-be-recognized text segment through an event recognition model, and the abnormal event confidence of each to-be-recognized text segment is obtained, including: Step S1011: performing text feature representation on the text segment to be recognized to obtain an event representation vector; Step S1012: classify and map the event representation vector to obtain a first confidence level that the to-be-recognized text segment includes an abnormal event and a second confidence level that the to-be-recognized text segment includes an event; Step S1013: Determine the confidence level of the abnormal event according to the first confidence level or the second confidence level.

[0129] In step S1011, the text segment to be identified is represented by text features to obtain an event representation vector. The text segment to be identified is the part selected from the target text for detailed analysis, and the text feature representation is to convert the text information into a vector form that can be processed by the computer. This vector is the event representation vector, which contains the semantics, grammar and other key information of the text segment to be identified, which is helpful for subsequent classification and analysis. A variety of technical means can be used for text feature representation, such as word embedding technology, such as Word2Vec, GloVe and other models.

[0130] In step S1012, the event representation vector is classified and mapped to obtain a first confidence that the text segment to be identified includes an abnormal event and a second confidence that it includes an event. Classification mapping is a process of mapping an event representation vector to probabilities of different categories, where the categories are mainly "includes abnormal events" and "includes events". A variety of classification models can be used to complete this task, and common ones include logistic regression models, support vector machine models, neural network models, etc. Taking the neural network model as an example, the event representation vector is input, and feature transformation and nonlinear processing are performed through the hidden layer of the network. Finally, the probability value of each category is obtained through the activation function of the output layer (such as the softmax function).

[0131] In step S1013, the confidence of the abnormal event is determined based on the first confidence or the second confidence. The method for determining the confidence of the abnormal event can be selected according to specific business needs and scenarios. One way is to directly use the first confidence as the confidence of the abnormal event, because the first confidence itself represents the probability that the text segment to be identified includes an abnormal event. In some cases, the first confidence and the second confidence can also be combined for calculation.

[0132] Based on the foregoing embodiments, the embodiments of the present application provide a text analysis device, and each unit included in the device, and each module included in each unit, can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0133] Figure 2 A schematic diagram of the structure of a text analysis device provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the text analysis device 200 includes: A data acquisition module 210 is used to acquire a first training text including a preset text content and a prior distribution position of each core text segmentation word in the first training text; A text masking module 220, configured to perform a character masking operation on one or more text segments in the first training text according to the prior distribution position of each core text segment, to obtain a text segment mask; A feature extraction module 230 is used to extract a representation vector from the text segment mask through the BERT model to be adjusted to obtain a target text feature vector; A decoding mapping module 240 is used to perform decoding mapping on the masked text segment in the text segment mask according to the target text feature vector through a first restoration mapping model to obtain a decoded text; The model tuning module 250 is used to iteratively optimize the model internal parameters of the BERT model according to the first training cost value between the first training text and the decoded text to obtain the optimized BERT model.

[0134] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0135] Figure 3 A hardware entity diagram of a computer system provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the hardware entity of the computer system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

[0136] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A text analysis method based on natural language processing, characterized in that: The method comprises: Acquire a first training text including preset text content and a prior distribution position of each core text segmentation word in the first training text; According to the prior distribution position of each core text segment, a character masking operation is performed on one or more text segments in the first training text to obtain a text segment mask; Extracting a representation vector from the text segment mask through the BERT model to be adjusted to obtain a target text feature vector; By using a first restoration mapping model, according to the target text feature vector, decoding mapping is performed on the masked text segment in the text segment mask to obtain a decoded text; According to the first training cost value between the first training text and the decoded text, the model internal parameters of the BERT model are iteratively optimized to obtain the optimized BERT model.

2. The method according to claim 1, characterized in that The step of performing a character masking operation on one or more text segments in the first training text according to the prior distribution position of each core text segment to obtain a text segment mask includes: According to a pre-set text segmentation window, the first training text is segmented by movement to obtain a set of text segments; According to the prior distribution position of each core text segment, determining the target text segment where each core text segment is located in the text segment set; For each of the text segments in the text segment set, the masked confidence corresponding to the text segment is determined according to the similarity between the text segment and each of the target text segments; wherein the greater the maximum similarity between the text segment and each target text segment, the greater the masked confidence corresponding to the text segment; According to the masked confidence corresponding to each of the text segments, a character masking operation is performed on one or more text segments in the first training text to obtain a text segment mask.

3. The method according to claim 2, characterized in that Determining the masked confidence corresponding to the text segment according to the similarity between the text segment and each of the target text segments includes: Determining a similarity between the text segment and each target text segment; For each of the target text segments, determining an alternative confidence level corresponding to the text segment according to the similarity between the text segment and the target text segment, wherein the greater the similarity between the text segment and the target text segment, the greater the alternative confidence level corresponding to the text segment; The masked confidence corresponding to the text segment is determined according to the maximum value of each candidate confidence corresponding to the text segment.

4. The method according to any one of claims 1 to 3, characterized in that After iteratively optimizing the model internal parameters of the BERT model according to the first training cost value between the first training text and the decoded text to obtain the optimized BERT model, the method further includes: Obtaining a second training text including preset text content and a prior distribution position of each core text segmentation in the second training text, and a first core text segmentation vector set, wherein each core text segmentation vector in the first core text segmentation vector set corresponds to a core text segmentation of the preset text content; Using the BERT model, extracting a representation vector from the second training text to obtain a second text feature vector; Inferring the distribution positions of the core text segmentation words in the second training text according to the first core text segmentation word vector set and the second text feature vector through the first distribution position inference model, and obtaining the inferred distribution position of each core text segmentation word in the second training text; According to the second training cost value between the prior distribution position of each core text segment in the second training text and the inferred distribution position of each core text segment in the second training text, one or more of the model internal parameters of the BERT model, the model internal parameters of the first distribution position inference model and the first core text segmentation vector set are optimized to obtain an optimized BERT model, an optimized first distribution position inference model and an optimized first core text segmentation vector set.

5. The method according to claim 4, characterized in that The first distribution position inference model includes a second restoration mapping model based on a Transformer model and a distribution position mapping model; the first distribution position inference model is used to infer the distribution position of the core text segmentation in the second training text according to the first core text segmentation vector set and the second text feature vector to obtain the inferred distribution position of each core text segmentation in the second training text, including: The first core text word segmentation vector set is optimized by combining the internal attention strategy with the second restoration mapping model based on the Transformer model to obtain the second core text word segmentation vector set, and the second core text word segmentation vector set is optimized according to the second text feature vector based on the cross-attention strategy to obtain the first core text word segmentation feature of each core text word segmentation; Through the distribution position mapping model, the distribution position of the core text segmentation in the second training text is inferred according to each of the first core text segmentation features, so as to obtain the inferred distribution position of each of the core text segmentation in the second training text.

6. The method according to claim 5, characterized in that The second restoration mapping model includes an internal attention network, a cross-attention network and a multi-layer perceptron network; The first core text word segmentation vector set is optimized by combining the internal attention strategy with the second restoration mapping model based on the Transformer model to obtain the second core text word segmentation vector set, and the second core text word segmentation vector set is optimized based on the cross-attention strategy according to the second text feature vector to obtain the first core text word segmentation feature of each core text word segmentation, including: By using the internal attention network and the internal attention strategy, the first core text word segmentation vector set is optimized to obtain a second core text word segmentation vector set; By using the cross-attention network, based on the cross-attention strategy and according to the second text feature vector, the second core text word segmentation vector set is optimized to obtain a third core text word segmentation vector set; Through the multi-layer perceptron network, each core text word segmentation vector in the third core text word segmentation vector set is mapped to a core text word segmentation feature of the corresponding core text word.

7. The method according to claim 6, characterized in that The combining of the internal attention strategy to optimize the first core text word segmentation vector set to obtain the second core text word segmentation vector set includes: In combination with the internal attention strategy, the semantic dependency relationship between each core text word segmentation vector in the first core text word segmentation vector set is extracted to obtain a first correlation matrix; According to the first correlation matrix, optimizing the first core text word segmentation vector set to obtain a second core text word segmentation vector set; The method of optimizing the second core text word segmentation vector set based on the cross-attention strategy and the second text feature vector to obtain a third core text word segmentation vector set includes: Based on the cross-attention strategy, the semantic dependency relationship between the second core text word segmentation vector set and the second text feature vector is extracted to obtain a second correlation matrix; The second core text word segmentation vector set is optimized according to the second correlation matrix to obtain a third core text word segmentation vector set.

8. The method according to any one of claims 5 to 7, characterized in that: The distribution position mapping model includes a first mapping model and a second mapping model, and the inferred distribution position includes a first distribution situation in a text sequence and a second distribution situation in a semantic level; The method of inferring the distribution position of the core text segmentation in the second training text according to each of the first core text segmentation features through the distribution position mapping model to obtain the inferred distribution position of each of the core text segmentation in the second training text includes: Using the first mapping model, according to each of the core text segmentation features, inferring the distribution position of the core text segmentation in the second training text in the text sequence, to obtain a first distribution of each of the core text segmentation in the second training text; Through the second mapping model, based on the features of each core text segmentation, the distribution position of the core text segmentation in the second training text in the semantic level is inferred to obtain the second distribution of each core text segmentation in the second training text.

9. A text analysis method based on natural language processing, characterized in that: The method comprises: Acquire a target text including preset text content and a fourth core text word segmentation vector set set in advance, wherein each core text word segmentation vector in the fourth core text word segmentation vector set corresponds to a core text word segmentation of the preset text content; Through the optimized BERT model, a representation vector is extracted from the target text to obtain a third text feature vector; wherein the BERT model is trained based on the text analysis method based on natural language processing according to any one of claims 1 to 8; Using the optimized second distribution position inference model, based on the fourth core text segmentation vector set and the third text feature vector, the positions of the core text segmentations in the target text are inferred to obtain the inferred distribution position of each core text segmentation in the target text; The fourth core text word segmentation vector set includes a first core text word segmentation vector set obtained based on the text analysis method based on natural language processing as described in any one of claims 4 to 8, and the second distribution position inference model includes a first distribution position inference model obtained based on the text analysis method based on natural language processing as described in any one of claims 4 to 8.

10. A text analysis device, characterized in that: The device comprises: A data acquisition module, used to acquire a first training text including a preset text content and a prior distribution position of each core text segmentation word in the first training text; A text masking module, configured to perform a character masking operation on one or more text segments in the first training text according to a prior distribution position of each core text segment, so as to obtain a text segment mask; A feature extraction module, used to extract a representation vector from the text segment mask through the BERT model to be adjusted, so as to obtain a target text feature vector; A decoding mapping module, configured to perform decoding mapping on the masked text segment in the text segment mask according to the target text feature vector through a first restoration mapping model to obtain a decoded text; The model tuning module is used to iteratively optimize the model internal parameters of the BERT model according to the first training cost value between the first training text and the decoded text to obtain the optimized BERT model.

Images