Text sample equalization processing method and system introducing attention mechanism

Abstract

The invention discloses a text sample equalization processing method and system introducing an attention mechanism, and relates to the field of data equalization processing, and the method comprises the steps: receiving a text sample set, and carrying out the feature extraction of the text sample set, and obtaining a vector matrix; the text sample set is input into a Transform network structure, an attention output value is obtained, and the Transform network structure comprises a multi-head attention mechanism; according to the attention output value and the vector matrix, carrying out importance evaluation on each text sample to obtain an importance score; performing text enhancement on the text samples with the importance scores smaller than a preset threshold value, and updating the text sample set; adversarial samples are added into the updated text sample set, and a new text sample set is obtained; and adjusting the sample quantity of each sample category by adopting an adaptive sampling strategy to obtain a final text sample set for representing the text sample set. The text sample can be equalized, and the overall performance of the model is effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data equalization processing, and in particular to a text sample equalization processing method and system that introduces an attention mechanism. Background Art

[0002] In today's digital age, the rapid development of the Internet and social media platforms has led to an explosive growth in user-generated content (UGC). This content covers various forms, including text, images, videos, etc., providing users with unprecedented opportunities for expression and communication. However, this comes with huge challenges in content security and quality control. The spread of bad, illegal, and harmful content may have serious consequences for individuals, society, and businesses. Therefore, effectively managing and reviewing this massive amount of content has become a top priority for platform operators and regulators.

[0003] Content safety big models and intelligent review big models are key technical solutions to address this challenge. These models are usually based on deep learning technology, especially in the fields of natural language processing (NLP) and computer vision (CV). The main application scenarios of the models include: 1. Social media platforms: Real-time review of user posts, comments and private messages to filter spam, malicious speech and illegal content. 2. News portals: Automatically detect and mark news articles that may contain false information, inflammatory speech or inappropriate content. 3. E-commerce platforms: Review product descriptions and user reviews to identify false advertisements, infringing information and inappropriate speech. 4. Online education platforms: Monitor student submissions and discussion content to ensure compliance with academic integrity standards and maintain a good learning environment. 5. Online game communities: Review the content of communication between players to prevent bullying, harassment and other inappropriate behaviors. 6. Internal communication systems of enterprises: Monitor internal emails and chat records to prevent the leakage of commercial secrets and workplace harassment. 7. Government agencies: Analyze public opinions and online public opinion to promptly discover and deal with speech that may cause social problems. By learning from large amounts of labeled data, these models can automatically identify and classify various types of content, including but not limited to various types of illegal content.

[0004] Although these models have achieved remarkable results in the field of content review, in actual applications, since on most platforms, the vast majority of content is compliant and illegal content only accounts for a small proportion, the ratio of normal content to illegal content is unbalanced. This will cause an imbalance in the text samples input into the model, which in turn causes the model to become biased, overfitting common categories, and insufficient recognition of rare but potentially more serious illegal content, which seriously affects the overall performance of the model and even causes serious consequences, such as missing important illegal content or mistakenly judging normal content as illegal.

[0005] There is currently no better solution to the above problems. Summary of the invention

[0006] The embodiments of the present application provide a text sample equalization processing method and system that introduces an attention mechanism, which is used to equalize text samples and thereby effectively improve the overall performance of the model.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a method for equalizing text samples by introducing an attention mechanism is provided, the method comprising:

[0009] In response to receiving the text sample set, preprocessing the text sample set, and performing feature extraction on the preprocessed text sample set to obtain a vector matrix;

[0010] Inputting the text sample set into a Transformer network structure to obtain an attention output value, wherein the text sample set includes at least one text sample, and the Transformer network structure includes a multi-head attention mechanism;

[0011] According to the attention output value and the vector matrix, an importance evaluation is performed on each of the text samples in the text sample set to obtain an importance score for each of the text samples;

[0012] Performing text enhancement on the text samples whose importance scores are less than a preset threshold, and updating the text sample set according to the text samples after text enhancement;

[0013] Add adversarial samples to the updated text sample set to obtain a new text sample set;

[0014] A preset classifier is used to determine the sample categories of the new text sample set and the number of samples in each sample category, and an adaptive sampling strategy is used to adjust the number of samples in each sample category to obtain a final text sample set for representing the text sample set.

[0015] In a possible implementation manner of the first aspect, the Transformer network structure also includes a feedforward neural network and a residual connection.

[0016] In another possible implementation manner of the first aspect, inputting the text sample set into a Transformer network structure to obtain an attention output value includes:

[0017] Pass the text sample set through the multi-head attention mechanism to obtain a first output;

[0018] The first output is respectively subjected to a first residual connection, a first layer normalization and a feedforward neural network to obtain a second output;

[0019] The second output is subjected to a second residual connection and a second layer normalization to obtain an attention output value, wherein the attention output value is a vector sequence, and each vector in the vector sequence corresponds to a text sample in the text sample set.

[0020] In another possible implementation of the first aspect, the step of passing the text sample set through the multi-head attention mechanism to obtain a first output includes:

[0021] Projecting the text sample set into three different vector spaces of query, key and value respectively through linear transformation;

[0022] In each attention head of the multi-head attention mechanism, a dot product similarity between a query and a key is calculated, and the dot product similarity is normalized to obtain at least one attention weight;

[0023] Multiply each of the attention weights of each of the attention heads by a value and sum them up to generate an output value of each of the attention heads;

[0024] The output values ​​of all the attention heads are concatenated, and the concatenated output values ​​are linearly transformed to obtain a first output.

[0025] In another possible implementation manner of the first aspect, performing importance evaluation on each of the text samples in the text sample set according to the attention output value and the vector matrix to obtain an importance score for each of the text samples includes:

[0026] For each of the text samples in the text sample set, determining the original vector of the text sample in the vector matrix;

[0027] Calculate the cosine similarity between the attention output value and the original vector;

[0028] When the cosine similarity is greater than a preset similarity threshold, a preset importance scoring formula is used to calculate the importance score of the text sample according to the cosine similarity.

[0029] In another possible implementation manner of the first aspect, performing text enhancement on the text sample whose importance score is less than a preset threshold, and updating the text sample set according to the text sample after text enhancement, includes:

[0030] Taking the text sample whose importance score is less than a preset threshold as a target sample, and identifying the sequence length of the target sample;

[0031] When the sequence length is greater than a preset length threshold, segmenting the target sample into a plurality of overlapping short text segments;

[0032] Inputting each of the short text segments into a pre-trained language model to obtain an output text segment corresponding to each of the short text segments, wherein the semantics of each of the short text segments is consistent with the corresponding output text segment;

[0033] According to a preset knowledge graph, all the output text fragments are sorted, and all the output text fragments are connected in the sorted order to obtain an output sample;

[0034] Identifying the word order structure of the output sample, and adjusting the output sample according to the word order structure of the output sample to obtain the text sample after text enhancement;

[0035] When the sequence length is not greater than the length threshold, determining non-stop words in the target sample, and replacing the non-stop words with synonyms of the non-stop words according to a synonym database to obtain the text sample after text enhancement;

[0036] The text sample set is updated according to the text sample after text enhancement.

[0037] In another possible implementation manner of the first aspect, identifying the word order structure of the output sample includes:

[0038] A preset dependency syntax analysis algorithm is used to identify the word order structure of the output sample.

[0039] In another possible implementation manner of the first aspect, the adaptive sampling strategy includes:

[0040] According to the number of samples of each sample category, the new text sample set is divided into three class samples, wherein the class samples include minority class samples, mean class samples and majority class samples;

[0041] Obtaining the sampling ratio of each class sample by the preset model;

[0042] For any of the class samples, determining whether the class sample is oversampled or undersampled;

[0043] In the case where the class samples are oversampled, interpolating the class samples using a preset interpolation algorithm to increase the sample quantity of the class samples;

[0044] In the case where the class samples are under-sampled, a random under-sampling algorithm is used to delete the class samples to reduce the number of samples of the class;

[0045] The oversampling or the undersampling is determined based on the sampling ratio of the class samples and the number of samples.

[0046] In a second aspect, the present application provides a text sample equalization processing system that introduces an attention mechanism, comprising:

[0047] a memory configured to store instructions; and

[0048] The processor is configured to call the instructions from the memory and implement the above-mentioned text sample equalization processing method introducing the attention mechanism when executing the instructions.

[0049] In a third aspect, the present application provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the above-mentioned text sample equalization processing method introducing an attention mechanism.

[0050] Through the above technical solutions, the text sample set is first preprocessed and feature extracted, laying the foundation for subsequent equalization processing. The introduction of multi-head attention mechanism and Transformer architecture effectively captures the complex relationship and contextual information between text samples, and improves the ability to identify rare illegal content. The importance evaluation based on vector sequence and vector matrix enables the model to identify and focus on samples that are more critical to the classification task, thereby alleviating the problem of overfitting of common categories. Text enhancement is performed on samples with importance scores below the preset threshold, which not only expands the number of samples in rare categories, but also improves the model's learning effect on these categories. The introduction of adversarial sample strategies further enhances the robustness of the model, enabling it to better cope with various potential adversarial inputs. Finally, the adaptive sampling strategy is used to adjust the sampling ratio of different categories, effectively balancing the contribution of various samples in the training process, thereby reducing the model's bias towards normal content that is dominant in number. Not only does it effectively solve the problem of sample imbalance, but it also improves the model's recognition accuracy for various illegal content and reduces missed and false positives. In this way, the overall performance and generalization capabilities of the content security big model and the intelligent review big model are significantly improved, enabling them to handle various types of content more fairly and accurately, effectively responding to challenges in the fields of content security and quality control, and providing more reliable technical support for platform operators and regulators.

[0051] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1A flowchart of a method for equalizing text samples by introducing an attention mechanism provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a Transformer network structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0056] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0057] Figure 1 The following schematically shows a flow chart of a method for equalizing text samples by introducing an attention mechanism according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a text sample equalization processing method introducing an attention mechanism, and the method may include the following steps.

[0058] S110, in response to receiving the text sample set, preprocessing the text sample set, and performing feature extraction on the preprocessed text sample set to obtain a vector matrix;

[0059] S120, inputting the text sample set into the Transformer network structure to obtain an attention output value, wherein the text sample set includes at least one text sample, and the Transformer network structure includes a multi-head attention mechanism;

[0060] S130, performing importance evaluation on each text sample in the text sample set according to the attention output value and the vector matrix to obtain an importance score for each text sample;

[0061] S140, performing text enhancement on text samples whose importance scores are less than a preset threshold, and updating the text sample set according to the text samples after text enhancement;

[0062] S150, adding adversarial samples to the updated text sample set to obtain a new text sample set;

[0063] S160: using a preset classifier to determine the sample categories of the new text sample set and the number of samples in each sample category, and using an adaptive sampling strategy to adjust the number of samples in each sample category, to obtain a final text sample set for representing the text sample set.

[0064] First, the text sample set is preprocessed, including but not limited to the following operations: removing special characters and punctuation marks, converting all text to lowercase, word segmentation, removing stop words, etc. Preprocessing aims to clean and standardize text data for subsequent processing and analysis.

[0065] After preprocessing, feature extraction is performed, mainly using word embedding technology. Word embedding is a technology that maps words to a high-dimensional vector space. Common methods include Word2Vec, GloVe, etc. In this step, each word is converted into a vector of fixed dimension, which can capture the semantic relationship between words. For each text sample, all word vectors in it are combined to form a matrix, namely the vector matrix. The vector matrix not only retains the semantic information of the original text, but also converts the text into a numerical form that can be directly processed by the machine learning algorithm. In this way, the text data is converted into structured numerical data, laying the foundation for subsequent analysis and processing, and converting the original, unstructured text data into a numerical representation with rich semantic information, so that the subsequent Transformer network can process this data more effectively and extract deeper features and relationships.

[0066] The preprocessed text sample set is input into the Transformer network structure to obtain the attention output value. Transformer is a neural network architecture based on the self-attention mechanism, and its core component is the multi-head attention mechanism. Specifically, each sample in the text sample set is first converted into a sequence of word embeddings. Then, these sequences are input into the encoder of the Transformer. The encoder contains multiple identical layers, each of which has a multi-head self-attention sublayer and a feedforward neural network sublayer. The working principle of the multi-head attention mechanism is that for each input word embedding, its attention weight with all other word embeddings is calculated. This is achieved by linearly projecting the input vector into three different vector spaces: query, key, and value. In each attention head, the attention weight is obtained by calculating the dot product between the query and the key, and then normalizing it with the softmax function. These weights are multiplied by the value and summed to generate the output of each attention head. The outputs of multiple attention heads are spliced ​​together and passed through a linear transformation to obtain the final attention layer output. This output is then passed through a feedforward neural network, coupled with residual connections and layer normalization to form the output of a complete Transformer layer. Multiple such layers are stacked together to form a complete Transformer encoder. Ultimately, the output of the Transformer network is the attention output value, which is a sequence of vectors, each of which corresponds to a text sample in the input text sample set. Through the multi-head attention mechanism, the model can capture long-distance dependencies and complex contextual information in the text, thereby better understanding the semantics and structure of the text and providing high-quality feature representation for subsequent importance evaluation.

[0067] The importance of each text sample in the text sample set is evaluated using the attention output value and the previously obtained vector matrix. The core idea is to evaluate the importance of each sample by comparing the similarity between the sample representation after the attention mechanism and the original sample representation. The specific implementation is as follows: First, for each text sample in the text sample set, find its corresponding original vector representation in the vector matrix. Then, calculate the similarity between the vector corresponding to the sample in the attention output value and the original vector. Cosine similarity or Euclidean distance can be used as a metric. Cosine similarity calculates the cosine value of the angle between two vectors, ranging from -1 to 1. The larger the value, the more similar the two vectors are. Euclidean distance calculates the straight-line distance between two vectors in a high-dimensional space. The smaller the distance, the more similar they are. When cosine similarity is selected, if the similarity is greater than a preset threshold (for example, 0.8), it is considered that the sample retains a lot of original information after being processed by the attention mechanism and may be more important. Subsequently, a preset importance scoring formula is used to calculate an importance score for each sample based on the calculated similarity. The importance scoring formula can be a linear mapping, the purpose of which is to convert the similarity value into a standardized importance score. For example, a sigmoid function can be used to map similarity to a score between 0 and 1. In summary, it is possible to identify samples that still retain a large amount of original information after being processed by the attention mechanism. These samples usually contain more key information or are more representative, and are more important for subsequent classification tasks. In this way, the model can pay more attention to samples that have a large amount of information and are more critical to the task, thereby improving the learning efficiency and performance of the model.

[0068] Text enhancement is performed for text samples with importance scores lower than a preset threshold, and the enhanced samples are used to update the text sample set. The purpose of this step is to improve the quality and diversity of low-importance samples, thereby improving the balance of the overall sample set. The specific implementation method is as follows: First, set an importance score threshold, such as 0.5. For samples with scores lower than this threshold, they are regarded as target samples that need to be enhanced. Then, different enhancement strategies are adopted according to the length of the target sample. For samples with a length exceeding a preset threshold (such as 100 words), a segmented enhancement method is adopted: the sample is divided into multiple overlapping short text segments, each of which may contain 50 words, and adjacent segments overlap by 20 words. These short text segments are respectively input into the pre-trained language model (such as the GPT series model) to generate new texts that are semantically consistent with each segment. The generated new text segments are sorted using the preset knowledge graph to ensure their semantic coherence and logic. Then, these new text segments are connected in the sorted order to form a complete new sample. Finally, the dependency syntactic analysis algorithm is used to identify the word order structure of the new sample, and it is adjusted as needed to ensure the correctness of the grammar and the naturalness of the expression. For samples whose length does not exceed the preset threshold, the word replacement method is used: first identify non-stop words in the sample, then use a synonym database (such as WordNet) to find synonyms of these words, and randomly select some words for replacement. This method can increase the diversity of samples while maintaining the original semantics. After the text enhancement is completed, the enhanced samples are added to the original text sample set to update the sample set. By enhancing samples with low importance, the quality and information content of these samples are improved, so that they can provide more valuable learning materials for the model. At the same time, the enhanced samples increase the diversity of the data set, which helps to improve the generalization ability of the model, especially for categories with a small number of samples. This approach can not only balance the sample distribution, but also improve the model's understanding and processing capabilities of various different expressions.

[0069] Add adversarial samples to the updated text sample set to generate a new text sample set. Adversarial samples are specially designed inputs designed to deceive or confuse machine learning models, thereby exposing the model's weaknesses and improving its robustness. In the text field, adversarial sample generation methods include but are not limited to the following: word replacement method, replacing some words in the original text with synonyms, near-synonyms or spelling errors, but keeping the overall semantics unchanged; insertion method, inserting some irrelevant words or phrases that do not affect the semantics into the original text; deletion method, deleting some non-key words in the original text, but keeping the main semantics; rearrangement method, changing the sentence structure but keeping the semantics. In specific implementation, the following steps can be adopted: First, randomly select a part of the original samples as the basis. For each selected sample, randomly select one or more of the above methods to generate adversarial samples. For example, when using the word replacement method, the word embedding model can be used to find several words that are most similar to the target word, and then randomly select one to replace. When using the insertion method, words can be randomly selected from a predefined list of irrelevant words and inserted into the original text. When using the deletion method, non-keywords can be identified through part-of-speech tagging, and a part can be randomly deleted. When using the rearrangement method, the sentence structure can be reorganized through dependency syntactic analysis. After generating adversarial samples, the original classifier is used to predict these samples. If the prediction result of the classifier for the adversarial sample is different from the original sample, the adversarial sample is considered to be valid and added to the sample set. Finally, these valid adversarial samples are merged with the updated text sample set to form a new text sample set. By introducing adversarial samples, the robustness and generalization ability of the model can be significantly improved. Adversarial samples can help the model learn more subtle feature differences and improve the model's sensitivity to text details. At the same time, these samples can also expose the potential weaknesses of the model, prompting the model to learn various variants and expressions of the text more comprehensively during the training process. This can not only improve the performance of the model in the face of various abnormal or edge cases, but also enhance the model's ability to resist malicious attacks, so that it is more robust and reliable in practical applications.

[0070] A preset classifier is used to determine the sample categories of the new text sample set and the number of samples in each category, and an adaptive sampling strategy is used to adjust the number of samples in each category, so as to finally obtain a final text sample set representing the original text sample set. The specific implementation method is as follows: First, a pre-trained classifier (such as a support vector machine, a random forest or a neural network) is used to classify each sample in the new text sample set to determine the category to which it belongs. At the same time, the number of samples in each category is counted. Then, the categories are divided into three categories according to the number of samples: minority class (the number of samples is significantly less than the average level), average class (the number of samples is close to the average level) and majority class (the number of samples is significantly more than the average level). Next, the ideal sampling ratio of each category for the preset model is obtained. The ratio can be determined according to business needs or statistical analysis, with the purpose of balancing the performance of each category in the final sample set. The actual number of samples is compared with the ideal sampling ratio to determine whether each category needs to be oversampled or undersampled. For categories that need to be sampled (usually minority classes), algorithms such as SMOTE (SyntheticMinority Over-sampling Technique) or ADASYN (Adaptive Synthetic) are used to generate new synthetic samples. These algorithms create new samples by interpolating between minority class samples, thereby increasing the number of samples in the minority class. For categories that need undersampling (usually the majority class), a random undersampling method can be used, that is, a portion of the samples are randomly deleted. Finally, the adjusted samples of each category are merged to form the final text sample set. Through the adaptive sampling strategy, the distribution of each category in the sample set is effectively balanced. This balance is not only reflected in the quantity, but also takes into account the quality and representativeness of the samples. For categories with a small number of samples, the synthesis of new samples increases their influence in the training process and improves the model's recognition ability for these categories. For categories with too many samples, the intelligent undersampling method reduces their dominance in training while retaining key information. This equalization process can significantly improve the overall performance of the model, especially when dealing with unbalanced data sets, allowing the model to handle various types of inputs more fairly and accurately.

[0071] This embodiment first preprocesses and extracts features from the text sample set, laying the foundation for subsequent equalization processing. The introduction of the multi-head attention mechanism and the Transformer architecture effectively captures the complex relationships and contextual information between text samples, and improves the ability to identify rare illegal content. The importance evaluation based on vector sequences and vector matrices enables the model to identify and focus on samples that are more critical to the classification task, thereby alleviating the problem of overfitting of common categories. Text enhancement is performed on samples with importance scores below the preset threshold, which not only expands the number of samples in rare categories, but also improves the learning effect of the model on these categories. The strategy of introducing adversarial samples further enhances the robustness of the model, enabling it to better cope with various potential adversarial inputs. Finally, the adaptive sampling strategy is used to adjust the sampling ratio of different categories, effectively balancing the contribution of various samples in the training process, thereby reducing the model's bias towards normal content that is dominant in number. Not only does it effectively solve the problem of sample imbalance, it also improves the model's recognition accuracy for various illegal content and reduces missed reports and false positives. In this way, the overall performance and generalization capabilities of the content security big model and the intelligent review big model are significantly improved, enabling them to handle various types of content more fairly and accurately, effectively responding to challenges in the fields of content security and quality control, and providing more reliable technical support for platform operators and regulators.

[0072] In one implementation of this embodiment, the Transformer network structure also includes a feedforward neural network and a residual connection.

[0073] The feedforward neural network and residual connection in the Transformer network structure are two key components, which together with the self-attention mechanism constitute the core architecture of the Transformer.

[0074] The Feed-Forward Neural Network (FFN) comes after the self-attention layer in the Transformer. It consists of two fully connected layers with a nonlinear activation function in between. The first fully connected layer usually expands the input dimension to a higher dimension, and then introduces nonlinearity through activation functions such as ReLU. The second fully connected layer compresses the dimension back to the original input dimension. This structure allows the network to capture more complex features and relationships.

[0075] Specifically, given the input x, the calculation process of the feedforward neural network can be expressed as:

[0076] FFN(x)=W2×ReLU(W1×x+b1)+b2;

[0077] Among them, W1 and W2 are weight matrices, b1 and b2 are bias vectors, and ReLU is the activation function. This process is applied independently to each position in the sequence, allowing the model to perform nonlinear transformations at each position, enhancing the model's expressive power.

[0078] Residual Connection is another important component. It works by establishing a short-circuit connection between the input and output of a sublayer (such as a self-attention layer or a feed-forward neural network layer). The main purpose of residual connection is to alleviate the gradient vanishing problem in deep neural network training and allow information to flow more directly in the network.

[0079] The calculation of the residual connection can be expressed as: Output = LayerNorm (x + Sublayer (x)).

[0080] Among them, x is the input of the sublayer, Sublayer(x) is the output of the sublayer (such as self-attention or feedforward network), and LayerNorm is the layer normalization operation. This structure allows the network to learn the residual function, that is, the difference between the input and the ideal output, instead of directly learning the complete transformation.

[0081] In practical implementation, feedforward neural networks and residual connections are usually combined in the following way:

[0082] 1. First, the input is processed through a multi-head self-attention mechanism.

[0083] 2. The output of self-attention is added to the original input through a residual connection and then normalized by the layer.

[0084] 3. The normalized result is used as the input of the feedforward neural network.

[0085] 4. The output of the feedforward network is again added to its input (i.e. the output of step 2) through a residual connection and layer normalization is performed again.

[0086] First, the feedforward neural network enhances the nonlinear modeling capabilities of the model, enabling it to capture more complex features and patterns. It operates independently at each position and can be viewed as fine-tuning the features of each tag. Second, the residual connection greatly improves the training stability and convergence speed of deep networks. It allows the gradient to flow through the network more easily during backpropagation, alleviating the gradient vanishing problem.

[0087] Layer normalization further enhances the stability of the network by standardizing the input of each layer and reducing the impact of internal covariate shift. This not only speeds up the training process but also improves the generalization ability of the model.

[0088] In this embodiment, the combination of feedforward neural networks and residual connections greatly enhances the performance and trainability of Transformer. The feedforward network provides additional nonlinear transformation capabilities, enabling the model to learn more complex feature representations. Residual connections make it possible to build and train deeper networks by promoting gradient flow and information transfer. This structural design enables Transformer to demonstrate better performance in various sequence modeling tasks, including but not limited to machine translation, text summarization, question-answering systems, etc. It not only improves the expressiveness and learning efficiency of the model, but also enhances the model's ability to capture long-term dependencies, which is particularly important when processing long sequence data.

[0089] In one implementation of this embodiment, a text sample set is input into a Transformer network structure to obtain an attention output value, including the following steps:

[0090] S210, passing the text sample set through a multi-head attention mechanism to obtain a first output;

[0091] S220, subjecting the first output to a first residual connection, a first layer normalization, and a feedforward neural network, respectively, to obtain a second output;

[0092] S230. The second output is subjected to a second residual connection and a second layer normalization to obtain an attention output value, wherein the attention output value is a vector sequence, and each vector in the vector sequence corresponds to a text sample in the text sample set.

[0093] The multi-head attention mechanism is one of the core components of the Transformer network, which allows the model to simultaneously focus on different parts of the input sequence and different representation subspaces. In this step, the text sample set is first converted into an embedding vector sequence and then processed by multi-head attention.

[0094] For example: suppose there is a text sample containing 5 words "The cat sat on mat", each word is embedded as a 64-dimensional vector. Use 8 attention heads, each with a dimension of 8 (64 / 8). Each head generates a 5x8 output matrix, and the outputs of the 8 heads are concatenated to obtain a 5x64 matrix. Finally, a linear transformation is performed to obtain the final 5x64 output matrix, which is the first output.

[0095] This embodiment allows the model to capture complex dependencies in the input sequence. Different attention heads can focus on different types of relationships, such as grammatical structure, semantic association, etc. By computing multiple attention heads in parallel, the model can consider information from multiple angles at the same time, greatly enhancing its ability to understand the sequence.

[0096] Figure 2A schematic diagram of a Transformer network structure provided in this embodiment is shown. Figure 2 As shown in Figure 1, after the multi-head attention mechanism, residual connections, layer normalization, and feedforward neural networks are introduced to further process and optimize feature representation. The specific implementation process is as follows:

[0097] 1. The first residual connection: Add the output of the multi-head attention to its input (i.e., the original embedding vector sequence). This short-circuit connection allows the gradient to flow directly back to the early layers, helping to alleviate the gradient vanishing problem in deep networks.

[0098] 2. First layer normalization: Perform layer normalization on the output of the residual connection. Layer normalization helps stabilize the training process of the deep network by calculating the mean and variance of each sample in the feature dimension and standardizing it.

[0099] Layer normalization calculation:

[0100] μ=mean(x);

[0101] σ=std(x);

[0102] y=γ×(x-μ) / (σ+ε)+β.

[0103] Here, γ and β are learnable scaling and bias parameters, and ε is a small constant for numerical stability.

[0104] 3. Feedforward neural network: The normalized output is passed into a two-layer feedforward neural network.

[0105] For example: Assume that the first output from the previous step is a 5x64 matrix (5 words, 64 dimensions each). First, this matrix is ​​added to the original input (residual connection). Then, each 5x1 column vector is layer normalized. Finally, through the feedforward network, it may be expanded from 64 dimensions to 256 dimensions, and then compressed back to 64 dimensions to obtain the final 5x64 second output matrix.

[0106] Residual connections promote the flow of gradients, making it easier for the network to learn identity mappings. Layer normalization reduces the impact of internal covariate shifts by standardizing the input of each layer, improving the stability and speed of training. Feedforward neural networks increase the nonlinear transformation capabilities of the model, enabling it to learn more complex feature representations. The combination of these three components greatly enhances the model's expressiveness and training effects.

[0107] The second output is connected through the second residual connection and normalized in the second layer to obtain the attention output value, where the attention output value is a vector sequence, and each vector in the vector sequence corresponds to a text sample in the text sample set. The specific implementation process is as follows:

[0108] 1. Second residual connection: Add the output of the feedforward neural network (second output) to its input (i.e. the normalized output of the first layer). This further strengthens the direct transfer of information and helps retain important features.

[0109] 2. Second layer normalization: The output of the second residual connection is normalized. This further stabilizes the feature distribution and helps with the processing of subsequent layers or tasks.

[0110] The final attention output value is a sequence of vectors, each of which corresponds to a text sample in the input text sample set. These vectors capture rich semantic information of the input text, including contextual relationships and long-range dependencies.

[0111] For example: suppose there is a set of 3 text samples, each sample has 5 words, and the feature dimension of each word is 64. Then:

[0112] 1. Input shape: (3,5,64), indicating 3 samples, 5 words in each sample, and 64 dimensions for each word.

[0113] 2. The second output shape after multi-head attention, the first residual connection, the first layer normalization and feedforward neural network processing is still: (3,5,64).

[0114] 3. After the second residual connection and the second layer normalization, the final attention output value shape is also: (3,5,64).

[0115] Each (5,64) matrix in (3,5,64) corresponds to the attention output value of a text sample. Each row (64-dimensional vector) corresponds to a word in the sample and contains a rich representation of the word after considering the entire sequence context.

[0116] This embodiment produces a highly contextualized feature representation. Through multiple residual connections and layer normalization, the model can continuously optimize and refine the feature representation while retaining the original information. The final attention output value not only contains the information of each word itself, but also incorporates the contextual information of the entire sequence. This rich representation enables the model to better understand the semantics and grammatical structure of the text, as well as the complex relationship between words, providing a basis for subsequent various natural language processing tasks (such as classification, translation, question answering, etc.).

[0117] The Transformer network in this embodiment achieves deep semantic understanding and feature extraction of the input text through a combination of multi-head attention mechanism, residual connection, layer normalization and feedforward neural network. The multi-head attention mechanism allows the model to focus on different semantic relationships at the same time and capture complex sequence dependencies. The residual connection ensures that information and gradients can be directly transmitted, effectively alleviating the gradient vanishing problem in deep networks. Layer normalization stabilizes the feature distribution, accelerates the training process and improves the generalization ability of the model. The feedforward neural network enhances the nonlinear modeling ability of the model. The synergy of these components enables Transformer to generate high-quality contextualized feature representations, and each text sample is converted into a rich vector sequence that accurately captures the meaning of the word in its context.

[0118] In one implementation of this embodiment, the text sample set is passed through a multi-head attention mechanism to obtain a first output, including the following steps:

[0119] S310, projecting the text sample set into three different vector spaces of query, key and value respectively through linear transformation;

[0120] S320, in each attention head of the multi-head attention mechanism, calculating the dot product similarity between the query and the key, and normalizing the dot product similarity to obtain at least one attention weight;

[0121] S330, multiplying each attention weight of each attention head by the value and summing them up to generate an output value of each attention head;

[0122] S340: Concatenate the output values ​​of all attention heads, and perform linear transformation on the concatenated output values ​​to obtain a first output.

[0123] In the multi-head attention mechanism, the first step is to convert the input text sample set into three different representations: query, key, and value. This process is achieved through linear transformation, that is, matrix multiplication of the input. Specifically, for each input vector x, three different weight matrices WQ, WK, and WV are multiplied respectively to obtain query q, key k, and value v. This can be expressed as: q = xWQ, k = xWK, v = xWV.

[0124] Linear transformation actually extracts and transforms the input features, mapping the original feature space to the new feature space. The purpose of mapping is to enable the model to understand and represent the input information from different perspectives. The three representations of query, key and value each have different functions: query is used to find relevant information, key is used to be matched, and value is the actual information content extracted.

[0125] For example, suppose the input text sample set contains 10 samples, each of which is represented as a 64-dimensional vector. If 8 attention heads are used, each with a dimension of 8, then the dimensions of WQ, WK, and WV are all 64×8. After linear transformation, each sample will get an 8-dimensional query, key, and value vector. For 10 samples, the dimensions of the query, key, and value matrices are all 10×8.

[0126] Finally, the original input is converted into three different representations to prepare for the subsequent attention calculation. Through this conversion, the model can more flexibly capture various features and relationships in the input, laying the foundation for the implementation of complex attention mechanisms. Its advantage is that it allows the model to learn different types of attention patterns, enhancing the model's ability to understand and process complex sequence data.

[0127] In each head of the multi-head attention mechanism, the next step is to calculate the similarity between the query and the key, which can be achieved through the dot product operation. The dot product operation is simple and intuitive, and can effectively measure the similarity of two vectors. Specifically, for the query matrix Q and the key matrix K, their dot product QK^T is calculated (where ^T represents the matrix transpose).

[0128] However, using the dot product directly may cause the variance of the result to be too large, especially when the vector dimension is high. To alleviate this problem, the dot product result is usually scaled by dividing it by the square root of the key vector dimension. This scaling operation can stabilize the gradient and make the training more stable. Therefore, the actual calculation formula is: where d_k is the dimension of the key vector.

[0129] Next, the softmax function is applied to the scaled dot product result for normalization. The softmax function converts the input into a probability distribution, ensuring that the sum of all weights is 1 and highlighting larger values. The output of this step is the attention weight, which reflects how much attention each position pays to all other positions.

[0130] For example, suppose there is a sequence of 5 words, and the dimension of each attention head is 8. Then the dimensions of Q and K are both 5×8, and the dimension of their dot product QK^T is 5×5. Softmax normalize this 5×5 matrix, and the resulting attention weight matrix is ​​also 5×5, where each row represents the attention distribution of the corresponding position to all positions.

[0131] Finally, a set of attention weights is generated, which determines how much attention should be paid to other positions in the sequence when generating the output of each position. In this way, the model can capture long-distance dependencies in the sequence. At the same time, since each attention head can learn different attention patterns, the multi-head mechanism allows the model to pay attention to multiple different types of relationships at the same time, greatly enhancing the model's expressive power.

[0132] After obtaining the attention weights, the multi-head attention mechanism uses these weights to aggregate the value vectors to generate the output of each attention head. This process can be regarded as a weighted summation operation, where the attention weight determines the contribution of each value vector to the final output.

[0133] Specifically, for each attention head, the previously calculated attention weight matrix is ​​multiplied with the value matrix V. In matrix multiplication, each row of the attention weight matrix is ​​dot-producted with each column of the value matrix V, and the result is a new matrix in which each element represents the weighted sum of all value vectors at that position.

[0134] For example, suppose there is a sequence of 5 words and the dimension of each attention head is 8. Then the dimension of the attention weight matrix is ​​5×5, and the dimension of the value matrix V is 5×8. The result of their multiplication is a 5×8 matrix, which is the output of the attention head. Each row represents the new representation of the corresponding position in the sequence. The new representation comprehensively considers the information of all positions in the sequence, but assigns different weights.

[0135] By weighted summation of attention weights, the model is able to generate a new representation for each position that not only contains the information of the position itself, but also incorporates information from other related positions in the sequence. This mechanism enables the model to dynamically focus on different parts of the input sequence, thereby capturing complex contextual dependencies.

[0136] For example, when processing a sentence, the new representation of a word may incorporate more information from other words that are semantically related to it, while considering less relevant words. This flexible way of aggregating information greatly enhances the model's ability to understand and express complex language structures, and is the key to the power of the attention mechanism.

[0137] The last step of the multi-head attention mechanism is to merge the outputs of all attention heads and perform final processing to get the first output. This step includes two main operations: concatenation and linear transformation.

[0138] First, the outputs of all attention heads are concatenated according to the feature dimension. Assuming there are h attention heads and the output dimension of each head is d_v, the concatenated matrix dimension will be sequence length × (h × d_v). This concatenation operation retains the different types of information captured by each attention head, allowing the model to utilize multiple attention modes simultaneously.

[0139] The concatenation operation can be expressed as: Concat(head_1,...,head_h), where head_i is the output of the i-th attention head.

[0140] Next, the concatenated matrix is ​​linearly transformed. Linear transformation is a matrix multiplication operation that maps the concatenated high-dimensional representation back to the original model dimension. It can be understood as a further fusion and compression of multi-head information.

[0141] The linear transformation can be expressed as: MultiHead(Q,K,V)=Concat(head_1,...,head_h)W_O, where W_O is a learnable weight matrix.

[0142] For example, suppose there is a sequence of 10 words, using 8 attention heads, and the output dimension of each head is 64. Then: 1. The output dimension of each attention head is 10×64; 2. The concatenated dimension is 10×(8*64)=10×512; 3. If the original dimension of the model is 256, then the final linear transformation will convert the 10×512 matrix into 10×256.

[0143] Through the concatenation operation, the model retains the unique information captured by each attention head, which may focus on different aspects of the input or features at different scales. The subsequent linear transformation allows the model to learn how to best combine this information, potentially strengthening certain important feature combinations while suppressing less important information.

[0144] This design enables the multi-head attention mechanism to capture multiple types of attention patterns at the same time, greatly enhancing the expressiveness and flexibility of the model. The final first output is a highly informative representation that incorporates the complex dependencies and multi-scale features in the input sequence, providing a rich foundation for subsequent processing.

[0145] This implementation first projects the input into three spaces: query, key, and value through linear transformation, preparing for information extraction from different angles. Then, by calculating the similarity between the query and the key and normalizing them, the attention weights that reflect the internal associations of the sequence are obtained. Next, these weights are used to perform weighted aggregation on the values ​​to generate a new representation that takes the context into account. Finally, the outputs of multiple attention heads are concatenated and linearly transformed to fuse different types of attention information. This enables the model to simultaneously focus on multiple relationships and features in the sequence, greatly enhancing its ability to understand complex language structures. The introduction of the multi-head mechanism allows the model to analyze input from multiple angles and scales, capturing richer semantic information.

[0146] In one implementation of this embodiment, the importance of each text sample in the text sample set is evaluated according to the attention output value and the vector matrix to obtain the importance score of each text sample, including the following steps:

[0147] S410, for each text sample in the text sample set, determining the original vector of the text sample in the vector matrix;

[0148] S420, calculating the cosine similarity between the attention output value and the original vector;

[0149] S430: When the cosine similarity is greater than a preset similarity threshold, a preset importance scoring formula is used to calculate the importance score of the text sample according to the cosine similarity.

[0150] For each text sample in the text sample set, the original vector of the text sample in the vector matrix is ​​determined, which can be understood as finding the corresponding original vector representation for each text sample in the text sample set. Specifically, the text data is mapped to a high-dimensional vector space so that the semantic information of the text can be expressed and processed in mathematical form. Usually, the mapping is achieved through a pre-trained language model.

[0151] Specifically, for each text sample, word segmentation is first performed. Word segmentation is the process of dividing continuous text into meaningful basic units (usually words or subwords). Then, for each word or subword, find its corresponding vector in the pre-trained model. These vectors are usually real number vectors of fixed dimensions, such as 300 dimensions or 768 dimensions, depending on the pre-trained model used.

[0152] If a text sample contains multiple words, then the vectors of these words need to be combined to form a vector representation of the entire text sample. Common combination methods include simple average and weighted average.

[0153] For example, suppose there is a text sample "Artificial intelligence is changing the world". Using the pre-trained language model, the vector dimension of each word is 100. First, the text is segmented into ["artificial intelligence", "is", "change", "world"]. Then, the vector representation of each word is found to obtain 4 100-dimensional vectors. Finally, a 100-dimensional vector can be obtained as the original vector representation of the entire text sample by calculating the average of these 4 vectors.

[0154] After determining the original vector representation of each text sample, the next step is to calculate the cosine similarity between the attention output value and these original vectors. Cosine similarity is an indicator that measures the degree of similarity between the directions of two vectors. Its value range is between -1 and 1, where 1 means the directions are exactly the same, -1 means the directions are completely opposite, and 0 means orthogonal (no correlation).

[0155] The calculation formula of cosine similarity is: cos(θ)=(A·B) / (||A||||B||), where A and B are two vectors, · represents the dot product operation, and ||A|| and ||B|| represent the Euclidean norms of vectors A and B respectively (i.e., the lengths of the vectors).

[0156] In specific implementation, for each text sample, the cosine similarity between its original vector and the attention output value needs to be calculated. The attention output value here is the vector representation obtained after the multi-head attention mechanism is processed.

[0157] After obtaining the cosine similarity, we first determine whether the similarity exceeds the preset threshold, and then use the preset formula to calculate the importance score of the samples that meet the conditions. This can be understood as screening and quantitative evaluation of text samples.

[0158] First, a similarity threshold is set to filter out samples that have low correlation with the attention output. The choice of this threshold needs to be determined according to the specific task and data characteristics, and the best value can usually be found through experiments. For example, the threshold may be set to 0.5 or 0.7.

[0159] For samples whose cosine similarity exceeds the threshold, the preset importance scoring formula is used to calculate their importance. The formula is as follows:

[0160] S=w1×cos_sim+w2×log(length);

[0161] Among them, S is the importance score, cos_sim is the cosine similarity, length is the text length, and w1 and w2 are weight coefficients.

[0162] This implementation first converts the text into a vector representation, and then quantifies the degree of semantic association between the original text and the representation after attention processing by calculating the cosine similarity. Finally, based on the similarity and other features, the importance score is calculated using a preset formula. This process not only takes into account the original semantics of the text, but also integrates the attention mechanism's understanding of the text, as well as other text features, to more comprehensively capture the importance of the text. At the same time, by setting a similarity threshold, less relevant samples can be effectively filtered out, improving processing efficiency.

[0163] In one implementation of this embodiment, text enhancement is performed on text samples whose importance scores are less than a preset threshold, and a text sample set is updated according to the text samples after text enhancement, including the following steps:

[0164] S510, taking a text sample whose importance score is less than a preset threshold as a target sample, and identifying the sequence length of the target sample;

[0165] S520, when the sequence length is greater than a preset length threshold, segmenting the target sample into a plurality of overlapping short text segments;

[0166] S530, inputting each short text segment into a pre-trained language model to obtain an output text segment corresponding to each short text segment, wherein each short text segment has semantic consistency with the corresponding output text segment;

[0167] S540, sorting all output text fragments according to a preset knowledge graph, and connecting all output text fragments in the sorted order to obtain an output sample;

[0168] S550, identifying the word order structure of the output sample, and adjusting the output sample according to the word order structure of the output sample to obtain a text sample after text enhancement;

[0169] S560, when the sequence length is not greater than the length threshold, determining non-stop words in the target sample, and replacing the non-stop words with synonyms of the non-stop words according to a synonym database to obtain a text sample after text enhancement;

[0170] S570: Update the text sample set according to the text samples after text enhancement.

[0171] First, set a preset threshold that represents the minimum standard for a text sample to be considered important enough. For example, if the importance score ranges from 0 to 1, the threshold can be set to 0.5 or 0.6.

[0172] For each text sample, its importance score is compared with this preset threshold. If a sample's score is lower than this threshold, it is marked as a "target sample", which means that this sample needs to be enhanced. It can be understood as dividing the dataset into two parts: samples that need to be enhanced and samples that do not need to be enhanced.

[0173] Next, for each text marked as a target sample, its sequence length needs to be identified. Sequence length usually refers to the number of words or characters in the text. For English text, the number of words can be counted; for Chinese text, the number of characters can be counted. The process of identifying sequence length includes the following steps: 1. Text preprocessing: remove extra blank characters, unify uppercase and lowercase characters, etc. 2. Word segmentation: If it is based on word counting, the text needs to be segmented into separate words. 3. Counting: Count the number of words or characters after word segmentation.

[0174] For longer text samples, break them down into smaller units that are easier to process. The length threshold is selected based on the requirements of the specific task and the characteristics of the language model used. For example, if the maximum input length of the language model used is 512 words, the length threshold can be set to 400 or 450 words.

[0175] For text samples that exceed the length threshold, they need to be segmented into multiple short text segments. Here, overlapping segmentation is used instead of simple sequential segmentation for two main reasons: 1. Maintaining contextual coherence: Overlapping can ensure that each segment contains sufficient contextual information, which helps to maintain semantic coherence during subsequent processing. 2. Avoiding information loss: Information may be broken at the boundaries of simple segmentation, while overlapping can retain this boundary information to the greatest extent.

[0176] The segmentation process can be performed as follows: 1. Determine the segment length: For example, the length of each segment can be set to 200 words. 2. Determine the overlap length: It is usually set to a certain percentage of the segment length, such as 25% or 50%. In this example, it can be set to 100 words. 3. Sliding window segmentation: Start from the text, take 200 words as a segment each time, then move back 100 words, take the next 200-word segment, and repeat this cycle until the end of the text.

[0177] This embodiment can convert a long text into a series of short text fragments with contextual associations. Not only does it make subsequent language model processing more efficient (because most pre-trained language models have input length restrictions), it also maintains the semantic integrity of the text. The overlapping parts ensure the information transfer between adjacent fragments, which helps to maintain the coherence of the overall text in subsequent processing. It is suitable for tasks that need to consider long-distance dependencies, such as text summarization or semantic analysis of long documents.

[0178] Each short text segment is input into a pre-trained language model to obtain an output text segment corresponding to each short text segment, wherein each short text segment is semantically consistent with the corresponding output text segment. Specifically, each short text segment is processed using a pre-trained language model to generate a new text segment with consistent semantics but possibly different expressions. Pre-trained language models are used here because these models have learned the deep representation and generation capabilities of language on a large-scale corpus, and can understand the semantics of the input text and generate related new text.

[0179] Commonly used pre-trained language models include BERT, GPT, T5, etc.

[0180] The processing process can be summarized in the following steps: 1. Convert each short text fragment into the input format required by the model. This usually includes adding special tags (such as [CLS], [SEP]) and converting the text into token IDs in the model's vocabulary. 2. Feed the processed input into the pre-trained model. The model encodes the input and understands its semantics. 3. Generate new text fragments based on the output of the model. And decode and convert the output of the model back into readable text. 4. Ensure that the generated new text fragment is semantically consistent with the original fragment. This can be achieved through semantic similarity calculation methods such as cosine similarity.

[0181] For example, suppose the original short text segment is: "Technology A is developing rapidly and affecting various industries." After being processed by the language model, the output text segment may be: "In recent years, technology A has made rapid progress, its application scope has continued to expand, and it is reshaping the future of multiple fields."

[0182] In summary, the rewriting or interpretation of the text is achieved, which increases the diversity of the text while maintaining the original core semantics. This can not only enrich the text expression, but also introduce some new and relevant information, thereby enhancing the content of the text.

[0183] According to the preset knowledge graph, all output text fragments are sorted, and all output text fragments are connected in the sorted order to obtain output samples, so as to reorganize multiple text fragments processed by the language model into a coherent whole. Using the preset knowledge graph for sorting can ensure that the final generated text remains coherent and reasonable in both logic and semantics. The knowledge graph contains the relationship information between concepts, which can help understand the semantic connection between text fragments.

[0184] Knowledge graphs are usually structured data consisting of entities (concepts) and relationships. In this step, the use of knowledge graphs can involve the following aspects: 1. Identify key entities or concepts from each text fragment. 2. Use the relationship information in the knowledge graph to infer the relationship between entities in different text fragments. 3. Use the hierarchy or causal relationship in the knowledge graph to analyze the logical order between text fragments.

[0185] The sorting process includes the following steps: 1. Calculate a score for each text fragment. The score can be calculated based on the importance of entities in the fragment, the semantic similarity between fragments, and the degree of logical connection with other fragments. 2. Sort the text fragments according to the calculated scores. 3. Consider the position information of the fragment in the original text and fine-tune the sorting result to maintain a certain degree of original order.

[0186] After sorting is complete, connect the text segments in the sorted order. During the connection process, some additional processing is required: 1. Add transition words or phrases to make the transition between adjacent segments more natural. 2. Remove duplicate content, especially in the original overlapping parts. 3. Adjust pronouns and demonstratives to ensure that they still refer to the correct objects in the new context.

[0187] For example, suppose there are three processed text fragments: 1. Artificial intelligence is changing the medical industry and improving diagnostic accuracy. 2. In the field of education, technology A is personalizing the learning experience. 3. The development of smart medicine makes telemedicine possible.

[0188] Through knowledge graph analysis, it is identified that "medical care" is a core topic, while "education" is a related but secondary topic. The sorted connection results may be:

[0189] "Artificial intelligence is changing the medical industry and improving diagnostic accuracy. Not only that, the development of smart medicine has also made telemedicine possible. In addition to medicine, in other fields such as education, A technology is also personalizing the learning experience."

[0190] In summary, multiple independent text fragments can be integrated into a logically coherent and semantically consistent whole. By utilizing the knowledge graph, the sorting process not only considers the surface information of the text, but also incorporates deeper semantics and logical relationships, making the enhanced text richer and more reasonable in structure and content.

[0191] The output samples generated in the previous step are further optimized to ensure that their word order structure is reasonable, so as to produce a more natural and fluent final text. The word order structure refers to the order of words, phrases and clauses in a sentence or paragraph, which has an important impact on the readability and expression of the text.

[0192] Identifying word order structures generally involves the following aspects: 1. Syntactic analysis: Using natural language processing tools to perform syntactic analysis on the text to identify sentence components such as subject, predicate, and object. 2. Topic structure analysis: Identifying the theme and focus of each sentence or paragraph. 3. Coherence analysis: Checking whether the logical connections and transitions between sentences are natural. 4. Information flow analysis: Evaluating whether the presentation order of information is logical and meets the reader's expectations.

[0193] Based on the identified word order structures, the following adjustments can be made: 1. Rearranging sentence order: If it is found that the positions of certain sentences are inappropriate, their positions in the paragraph can be adjusted. 2. Adjusting the internal structure of sentences: If the structure of a certain sentence is not clear enough, its internal components can be reorganized. 3. Adding transitional words: Appropriate connecting words can be added between sentences or paragraphs to enhance the fluency of the text. 4. Splitting or merging sentences: If some sentences are too long or too short, they can be split or merged with other sentences. 5. Adjusting information density: Ensuring that the amount of information in each paragraph is appropriate and avoiding some parts being too dense or sparse.

[0194] When the sequence length is not greater than the length threshold, non-stop words in the target sample are identified. Stop words usually refer to those words that frequently appear in the text but contribute little to the semantics, such as "de" (的), "shi" (是), "zai" (在), etc. It can be achieved using a predefined stop word list. The process of identifying non-stop words can be realized through part-of-speech tagging and word frequency analysis. For the identified non-stop words, their synonyms need to be found. The synonym database can be an existing dictionary such as WordNet. The method of synonym replacement can increase the lexical diversity of the text, which helps the model learn richer expression methods. At the same time, the original semantics is retained.

[0195] Add the enhanced text samples generated in the previous steps to the original text sample set. First, it is necessary to ensure that the newly generated samples do not repeat with the original samples, which can be quickly judged by calculating text similarity or using a hash function. When adding new samples, the balance of the sample set needs to be considered to ensure that the proportions of samples in different categories or topics are reasonable. Techniques such as stratified sampling or oversampling / undersampling can be used to adjust the sample distribution.

[0196] The adjustment of word order structures in this embodiment can help the model learn different expression methods and improve its understanding ability of various sentence patterns. Synonym replacement increases the richness of vocabulary, which helps the model establish a more flexible semantic understanding. Continuously updating the sample set ensures the freshness and representativeness of the data. This multi-angle data augmentation method can not only expand the scale of training data, but also improve the quality and diversity of the data, thus significantly enhancing the generalization ability and robustness of the model.

[0197] In one implementation of this embodiment, identifying the word order structure of the output sample includes the following steps:

[0198] S610: Use a preset dependency syntax analysis algorithm to identify the word order structure of the output sample.

[0199] The core idea of ​​dependency parsing is to determine the dependency relationship between words in a sentence, so as to construct the grammatical structure of the entire sentence. This process usually includes the following main steps: First, the input sentence is segmented and tagged with parts of speech. The continuous text is divided into independent word units, and each word is assigned a corresponding part of speech tag. Next, a pre-trained dependency parsing model is used to predict the dependency relationship between words. It can be a transfer-based model (such as MaltParser). The model usually uses deep learning techniques, such as recurrent neural networks (RNN) or attention mechanisms, to capture long-distance dependencies between words, and the output of the model is a dependency tree.

[0200] This implementation adopts a preset dependency syntax analysis algorithm to identify the word order structure of the output sample, which can deeply understand the grammatical structure of the sentence and the relationship between words. Through accurate word order structure recognition, it provides a reliable foundation for subsequent text enhancement operations, so that word order adjustment and word replacement can be performed under the premise of maintaining grammatical correctness and semantic coherence. It greatly improves the quality and diversity of text enhancement, thereby generating more natural and fluent text variants, effectively expanding the training data set, and improving the model's language understanding and generation capabilities.

[0201] In one implementation of this embodiment, the adaptive sampling strategy includes the following steps:

[0202] S710, dividing the new text sample set into three class samples according to the number of samples of each sample category, wherein the class samples include minority class samples, average class samples and majority class samples;

[0203] S720, obtaining the sampling ratio of each class sample of the preset model;

[0204] S730, for any class sample, determining whether the class sample is oversampled or undersampled;

[0205] S740, in the case where the class samples are oversampled, interpolate the class samples using a preset interpolation algorithm to increase the number of class samples;

[0206] S750, in the case of under-sampling of class samples, using a random under-sampling algorithm to delete class samples to reduce the number of class samples;

[0207] Among them, oversampling or undersampling is determined based on the sampling ratio and number of class samples.

[0208] The new text sample set is divided into three categories according to the number of sample categories. First, the number of samples in each category needs to be counted, and then classified according to the distribution of the number. In specific implementation, the mean and standard deviation of the number of samples in all categories can be calculated first. The category with a sample number lower than the mean minus one standard deviation is defined as the minority class, the category with a sample number higher than the mean plus one standard deviation is defined as the majority class, and the remaining categories are defined as the mean class. For example, suppose there are 5 categories with sample numbers of 100, 200, 500, 800, and 1000 respectively. The calculated mean value is 520 and the standard deviation is about 391. Then, the category with a sample number less than 129 (520-391) (i.e., 100 and 200) is the minority class, the category with a sample number greater than 911 (520+391) (i.e., 1000) is the majority class, and the rest (500 and 800) are the mean class. This division method can effectively identify the imbalance in the data set and provide a basis for subsequent sampling strategies.

[0209] Get the sampling ratio of each class sample of the preset model. The preset model is designed based on specific tasks or domain knowledge. The purpose is to determine the proportion of each category in the final data set. Therefore, the sampling ratio of each class sample of the model can be directly obtained through the preset model. Generally speaking, the sampling ratio is inversely proportional to the number of original samples. The specific calculation formula can be: sampling ratio = K / (number of samples) α , where K is a constant and α is a parameter between 0 and 1, which is used to adjust the sampling intensity. For example, for the five categories mentioned above, if K = 1000 and α = 0.5, the sampling ratios of each category are 3.16, 2.24, 1.41, 1.12, and 1.00 respectively.

[0210] Determine whether each class sample needs to be oversampled or undersampled based on the sampling ratio and actual number of samples obtained in S720. In specific implementation, a threshold value, such as 1.0, can be set. If the sampling ratio of a certain category is greater than this threshold, it is determined to be oversampled; if it is less than this threshold, it is determined to be undersampled; if it is equal to this threshold, it remains unchanged. Continuing with the previous example, the sampling ratio of the minority class (100 and 200) is greater than 1.0, and oversampling is required; the sampling ratio of the majority class (1000) is equal to 1.0 and remains unchanged; the sampling ratio of the mean class (500 and 800) is less than 1.0, and undersampling is required. This method can adopt corresponding sampling strategies for different categories, which helps to balance the data set.

[0211] For the categories that need oversampling, a preset interpolation algorithm is used to increase the number of samples. The interpolation algorithm can be SMOTE (Synthetic Minority Over-sampling Technique). The basic principle of SMOTE is to generate new synthetic samples between minority class samples. In specific implementation, for each minority class sample, a sample is randomly selected from its k nearest neighbors, and then a point is randomly selected on the line between the two samples as the new sample. Mathematically expressed as: new sample = original sample + rand (0,1) × (nearest neighbor sample - original sample).

[0212] For the categories that need undersampling, a random undersampling algorithm is used to reduce the number of samples. Random undersampling randomly removes a portion of samples from the majority class until the target number is reached. When implementing, a random number generator can be used to decide which samples to keep. For example, if there are 1,000 samples in a category and it needs to be reduced to 800, 1,000 random numbers between 0 and 1 can be generated, and samples with random numbers less than 0.8 are selected to be retained, and the rest are deleted.

[0213] The adaptive sampling strategy in this embodiment effectively solves the problem of class imbalance in the data set by fine-tuning samples of different categories. By dividing the samples into minority classes, average classes, and majority classes, and using oversampling or undersampling techniques in a targeted manner, the representativeness of minority class samples can be improved while maintaining the overall characteristics of the data set. This method not only increases the diversity of the data set, but also improves the model's ability to recognize minority classes, thereby improving the overall performance of the model. Especially when dealing with unbalanced data sets, this strategy can significantly improve the generalization ability of the model and reduce the risk of overfitting or underfitting. Ultimately, this adaptive sampling strategy can produce a more balanced and representative training data set, providing a better foundation for subsequent machine learning tasks.

[0214] The embodiment of the present application also provides a text sample equalization processing system that introduces an attention mechanism, including:

[0215] a memory configured to store instructions; and

[0216] The processor is configured to call instructions from the memory and implement the above-mentioned text sample equalization processing method introducing the attention mechanism when executing the instructions.

[0217] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned text sample equalization processing method introducing an attention mechanism.

[0218] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0219] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of the processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0220] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0221] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0222] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0223] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0224] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0225] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0226] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for equalizing text samples by introducing an attention mechanism, characterized in that: include: In response to receiving the text sample set, preprocessing the text sample set, and performing feature extraction on the preprocessed text sample set to obtain a vector matrix; Inputting the text sample set into a Transformer network structure to obtain an attention output value, wherein the text sample set includes at least one text sample, and the Transformer network structure includes a multi-head attention mechanism; According to the attention output value and the vector matrix, an importance evaluation is performed on each of the text samples in the text sample set to obtain an importance score for each of the text samples; Performing text enhancement on the text samples whose importance scores are less than a preset threshold, and updating the text sample set according to the text samples after text enhancement; Add adversarial samples to the updated text sample set to obtain a new text sample set; A preset classifier is used to determine the sample categories of the new text sample set and the number of samples in each sample category, and an adaptive sampling strategy is used to adjust the number of samples in each sample category to obtain a final text sample set for representing the text sample set.

2. The method according to claim 1, characterized in that The Transformer network structure also includes a feedforward neural network and a residual connection.

3. The method according to claim 2, characterized in that The step of inputting the text sample set into the Transformer network structure to obtain the attention output value includes: Pass the text sample set through the multi-head attention mechanism to obtain a first output; The first output is respectively subjected to a first residual connection, a first layer normalization and a feedforward neural network to obtain a second output; The second output is subjected to a second residual connection and a second layer normalization to obtain an attention output value, wherein the attention output value is a vector sequence, and each vector in the vector sequence corresponds to a text sample in the text sample set.

4. The method according to claim 3, characterized in that The step of passing the text sample set through the multi-head attention mechanism to obtain a first output includes: Projecting the text sample set into three different vector spaces of query, key and value respectively through linear transformation; In each attention head of the multi-head attention mechanism, a dot product similarity between a query and a key is calculated, and the dot product similarity is normalized to obtain at least one attention weight; Multiply each of the attention weights of each of the attention heads by a value and sum them up to generate an output value of each of the attention heads; The output values ​​of all the attention heads are concatenated, and the concatenated output values ​​are linearly transformed to obtain a first output.

5. The method according to claim 1, characterized in that The step of performing importance evaluation on each of the text samples in the text sample set according to the attention output value and the vector matrix to obtain an importance score for each of the text samples includes: For each of the text samples in the text sample set, determining the original vector of the text sample in the vector matrix; Calculate the cosine similarity between the attention output value and the original vector; When the cosine similarity is greater than a preset similarity threshold, a preset importance scoring formula is used to calculate the importance score of the text sample according to the cosine similarity.

6. The method according to claim 1, characterized in that The text sample having the importance score less than a preset threshold is subjected to text enhancement, and the text sample set is updated according to the text sample after text enhancement, including: Taking the text sample whose importance score is less than a preset threshold as a target sample, and identifying the sequence length of the target sample; When the sequence length is greater than a preset length threshold, segmenting the target sample into a plurality of overlapping short text segments; Inputting each of the short text segments into a pre-trained language model to obtain an output text segment corresponding to each of the short text segments, wherein the semantics of each of the short text segments is consistent with the corresponding output text segment; According to a preset knowledge graph, all the output text fragments are sorted, and all the output text fragments are connected in the sorted order to obtain an output sample; Identifying the word order structure of the output sample, and adjusting the output sample according to the word order structure of the output sample to obtain the text sample after text enhancement; When the sequence length is not greater than the length threshold, determining non-stop words in the target sample, and replacing the non-stop words with synonyms of the non-stop words according to a synonym database to obtain the text sample after text enhancement; The text sample set is updated according to the text sample after text enhancement.

7. The method according to claim 6, characterized in that The identifying the word order structure of the output sample includes: A preset dependency syntax analysis algorithm is used to identify the word order structure of the output sample.

8. The method according to claim 1, characterized in that The adaptive sampling strategy includes: According to the number of samples of each sample category, the new text sample set is divided into three class samples, wherein the class samples include minority class samples, mean class samples and majority class samples; Obtaining the sampling ratio of each class sample by the preset model; For any of the class samples, determining whether the class sample is oversampled or undersampled; In the case where the class samples are oversampled, interpolating the class samples using a preset interpolation algorithm to increase the sample quantity of the class samples; In the case where the class samples are under-sampled, a random under-sampling algorithm is used to delete the class samples to reduce the number of samples of the class; The oversampling or the undersampling is determined based on the sampling ratio of the class samples and the number of samples.

9. A text sample equalization processing system that introduces an attention mechanism, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and to implement the text sample equalization processing method introducing an attention mechanism according to any one of claims 1 to 8 when executing the instructions.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for enabling a machine to execute the text sample equalization processing method introducing an attention mechanism according to any one of claims 1 to 8.

Images