Fine-grained text uncertainty monitoring method and system based on semantic compression regularization

Abstract

The invention relates to the technical field of natural language processing, and particularly discloses a fine-grained text uncertainty monitoring method and system based on semantic compression regularization, and the method comprises the steps: segmenting a to-be-detected text into a plurality of Tokens, and carrying out the context fusion of each Token through a pre-training large language model, and obtaining the semantic representation of each Token; based on a semantic compression regularization technology, calculating the uncertainty contribution of semantic representation to the task, and obtaining the uncertainty weight of each Token; and binarizing the uncertainty weight according to a preset threshold value, marking the high-uncertainty Token as a shielding state, keeping the low-uncertainty Token in an available state, and obtaining an uncertainty monitoring result. According to the method, the false alarm rate is reduced, meanwhile, the monitoring accuracy and interpretability are remarkably improved, and a more reliable and robust solution is provided for financial risk control, content security, compliance examination and other scenes.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a fine-grained text uncertainty monitoring method and system based on semantic compression regularization. Background Technology

[0002] With the widespread application of large-scale language models in financial risk control, content compliance, and sensitive information monitoring, the demand for text uncertainty monitoring has increased dramatically. However, existing mainstream solutions often operate at the sentence, paragraph, or even entire document level, providing overall judgments through keyword matching, traditional machine learning classifiers, or direct fine-tuning of pre-trained language models. While these methods are easy to deploy, they often fail to accurately pinpoint the word segments that truly pose a risk due to a lack of detailed characterization of semantic differences and uncertainties at the token level. On the one hand, coarse-grained predictions are prone to false positives when encountering synonym substitutions or contextual interference; on the other hand, the model's amplification or neglect of local anomalous tokens can also lead to missed detections. To compensate for this deficiency, some studies have attempted to introduce overall confidence calibration or window-level sliding assessment, but these remain at the sample or short-segment level, making it difficult to target and suppress specific high-risk uncertainty tokens, thus limiting the interpretability and robustness of the monitoring system. Summary of the Invention

[0003] This invention aims to address the problems of coarse granularity, high false positive rate, and lack of interpretability in existing text uncertainty monitoring technologies. To this end, this invention provides a fine-grained text uncertainty monitoring method and system based on semantic compression regularization. First, the text is semantically encoded at the token level. Then, a semantic compression regularization module is used to evaluate the contribution of each token to the overall task uncertainty, dynamically generating uncertainty weights. Next, an attention mask is constructed to directly shield the influence of uncertain tokens. Finally, uncertainty discrimination is performed on the suppressed semantic representation. This framework combines the deep semantic understanding capabilities of LLM with the discriminative advantages of the semantic compression regularization module, achieving accurate isolation and interpretation of high-uncertainty tokens. Experimental and practical deployment results show that this invention significantly improves monitoring accuracy and interpretability while reducing the false positive rate, providing a more reliable and robust solution for scenarios such as financial risk control, content security, and compliance review.

[0004] This invention provides a fine-grained text uncertainty monitoring method based on semantic compression regularization, and the technical solution adopted is as follows: including the following steps: S1: Obtain the text to be tested; S2: Segment the text to be tested into several tokens, and use a pre-trained large language model to perform context fusion on each token to obtain the semantic representation of each token; S3: Based on semantic compression regularization technology, calculate the uncertainty contribution of semantic representation to the task and obtain the uncertainty weight of each token; S4: Binarize the uncertainty weights according to the preset threshold, mark the high uncertainty tokens as the masked state, and keep the low uncertainty tokens as the available state to obtain the uncertainty monitoring results.

[0005] Furthermore, in step S2, the pre-trained large language model adopts a multi-layer self-attention pre-trained language model.

[0006] Furthermore, in step S2, the pre-trained large language model performs forward propagation on the test text to obtain the hidden states of each layer; the hidden representations of each layer are concatenated in a weighted sum manner to form a token-level context representation, thereby obtaining the semantic representation of each token.

[0007] Furthermore, in step S3, uncertainty weights are calculated using a semantic compression regularization module, which includes an encoder and a decoder: In the encoder, the semantic representation is passed sequentially through the first fully connected layer, the first batch normalization, the first ReLU activation, random dropout, and two parallel fully connected branches, and then the latent variables are sampled using the reparameterization technique. In the decoder, the latent variables pass through the second fully connected layer, the second batch normalization, the second ReLU activation and random dropout, the third fully connected layer, and the Sigmoid function in sequence to obtain the uncertain weights.

[0008] Furthermore, two parallel fully connected branches are used to predict the mean and variance of the underlying distribution, respectively.

[0009] Furthermore, the closer the uncertainty weight is to 1, the higher the predictive uncertainty introduced by the token in the current context.

[0010] Furthermore, in step S3, the semantic compression regularization module minimizes the weighted joint loss of classification loss and regularization term as the loss during the training phase.

[0011] Furthermore, the loss function is: in, For loss function, For classification cross-entropy loss, To adjust the trade-off between semantic compression regularization and task discrimination, KL represents the Kullback-Leibler divergence. It follows a Gaussian posterior distribution. Let be a potential variable for the t-th Token. Let be the semantic representation vector of the t-th token. This is the prior distribution.

[0012] Furthermore, in step S4, if the uncertainty weight is greater than a preset threshold, the corresponding Token is marked as blocked; otherwise, the corresponding Token is marked as available.

[0013] This invention also provides a fine-grained text uncertainty monitoring system based on semantic compression regularization, the technical solution of which includes: The text acquisition module is used to acquire the text to be tested. The context semantic fusion module is used to segment the text to be tested into several tokens, and use a pre-trained large language model to perform context fusion on each token to obtain the semantic representation of each token. The semantic compression regularization module is used to calculate the uncertainty contribution of semantic representation to the task based on semantic compression regularization technology, and obtain the uncertainty weight of each token. The monitoring result calculation module is used to binarize the uncertainty weights according to a preset threshold, mark high uncertainty tokens as masked, and keep low uncertainty tokens as available, thereby obtaining the uncertainty monitoring results.

[0014] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: This invention assesses uncertainty at the token level and dynamically masks it, enabling precise identification of risk sources and overcoming the limitation of sentence / paragraph-level methods in failing to identify fine-grained risks. By calculating uncertainty weights through a semantic compression regularization module, it provides an intuitive basis for model decision-making, making the system sensitive to abnormal tokens and stable to normal tokens, facilitating algorithm visualization and manual verification. This invention significantly reduces false positives caused by contextual pseudo-relevance by actively masking the negative interference of high-uncertainty tokens on global decisions. This invention can be embedded as a plug-in module into existing large language model inference services or text processing pipelines without altering the main model structure, resulting in low deployment costs and applicability to various scenarios such as financial risk control and content security auditing.

[0015] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the method provided by the present invention.

[0018] Figure 2 This is a training diagram provided by the present invention.

[0019] Figure 3 This is a system structure block diagram provided by the present invention.

[0020] Figure label: 1. Text acquisition module; 2. Contextual semantic fusion module; 3. Semantic compression regular expression module; 4. Monitoring result calculation module. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0022] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0023] The following is combined Figures 1 to 3 The present invention will be further described in detail below, providing a fine-grained text uncertainty monitoring method and system based on semantic compression regularization: In this embodiment, as Figure 1As shown, a fine-grained text uncertainty detection method based on semantic compression regularization is provided, including the following steps: S1: Get the text to be tested.

[0024] S2: Segment the text to be tested into several tokens, and use a pre-trained large language model to perform context fusion on each token to obtain the semantic representation of each token.

[0025] The test text is segmented into several tokens, and then the test text is input into a pre-trained large language model. The pre-trained large language model tests the test text sequence. Forward propagation is performed to obtain the hidden states of each layer of the large language model; the hidden representations of each layer of the large language model are concatenated in a weighted sum manner to form a token-level context representation, thus obtaining the semantic representation of each token. , ,in, This represents the t-th token, where T is the sequence length. Let be the semantic representation vector of the t-th token. Let d be the real number field and d be the dimension. In this process, the model performs context fusion on each token to generate a high-dimensional semantic vector. This vector not only contains the semantic information of the token itself but also integrates the semantic relationships between its preceding and following text, providing a rich representational basis for subsequent uncertainty assessment. In this embodiment, the pre-trained large language model employs a multi-layer self-attention pre-trained language model.

[0026] S3: Based on semantic compression regularization technology, calculate the contribution of semantic representation to the uncertainty of the task, and obtain the uncertainty weight of each token.

[0027] This step is based on semantic compression regularization technology, which builds a semantic compression regularization module to learn the contribution of the potential representation of each token to the uncertainty of the task.

[0028] The semantic compression regular expression module contains an encoder and a decoder: In the encoder, the semantic representation is sequentially passed through the first fully connected layer, the first batch normalization, and the first ReLU activation to obtain latent spatial features. After random discarding, the distribution parameters of these features are predicted through two parallel fully connected branches, which are used to predict the mean and variance of the latent distribution, respectively. Then, the latent variables are sampled using a reparameterization technique. These latent variables can be regarded as a compression of the original semantic representation while retaining key information.

[0029] In the decoder, the latent variables are transformed sequentially through a second fully connected layer, a second batch normalization, a second ReLU activation, and random dropout; then, through a third fully connected layer and a sigmoid function, the uncertainty weights of the token are output. , The closer the uncertainty weight is to 1, the higher the prediction uncertainty introduced by the token in the current context, which is more likely to lead to classification errors; conversely, the lower the uncertainty weight, the more reliable the token content is.

[0030] To ensure that the uncertainty weights accurately reflect the true risk level of each token, the classification loss and the regularization term (KL divergence) are combined according to hyperparameters during the training phase. Weighted joint minimization. Through this joint optimization, the semantic compression regularization module can achieve a balance between compressing information and preserving discriminative ability, so that the uncertainty weight of high-uncertainty tokens is pushed to a higher level, while irrelevant or low-risk tokens maintain a lower uncertainty weight.

[0031] The model training process is as follows Figure 2 As shown. During the model training phase, given the token context representation, the semantic compression regularization module encoder outputs a Gaussian posterior distribution. , ,in, Let be a potential variable for the t-th Token. Let be the semantic representation vector of the t-th token. The mean is Covariance is The Gaussian distribution. Through reparameterization... Perform differentiable sampling. , The posterior mean is... For the posterior standard deviation, For element-wise multiplication, For noise sampled from a Gaussian distribution, It is a standard Gaussian distribution with a mean of 0 and a covariance of identity matrix I. The semantic compression regularization module decoder... The mapping is performed and the predicted value of the token-level uncertainty weights is obtained through the Sigmoid function. .

[0032] The training objective is , , .in, For encoder parameters, For decoder parameters, For distribution Expectations For the log-likelihood term, The uncertainty label for the overall text is y=1, which represents high uncertainty, and y=0, which represents low uncertainty. To adjust the trade-off between semantic compression regularization and task discrimination, KL represents the Kullback-Leibler divergence. This is the prior distribution.

[0033] To ensure that the uncertainty weights accurately reflect high-uncertainty tokens, this implementation weights and minimizes the classification cross-entropy loss together with a KL regularization term with a weight of 0.1: in, For loss function, For classification cross-entropy loss, =0.1. Configure Adam optimizer, learning rate 2×10. -5 The batch size is 32, the training is conducted for 200 epochs, and the F1-score is monitored on the validation set. The model is saved when the metric is optimal. Through this joint training, the semantic compression regularization module can effectively increase the weight of high-uncertainty tokens while maintaining discriminative ability, so that they are given priority in subsequent masking.

[0034] S4: Binarize the uncertainty weights according to the preset threshold, mark the high uncertainty tokens as the masked state, and keep the low uncertainty tokens as the available state to obtain the uncertainty monitoring results.

[0035] Each uncertainty weight is binarized based on a pre-set threshold to generate a dynamic mask. When the uncertainty weight exceeds a preset threshold, the corresponding token belongs to the high uncertainty category and needs to be hidden in the context. Therefore, this token is hidden. season The corresponding token will be marked as blocked. For the preset threshold, Let be the dynamic mask for the t-th token. When the uncertainty weight is less than or equal to a preset threshold, the corresponding token belongs to the low uncertainty category and will be directly retained, and the corresponding token will be marked as available. Applying the mask to the semantic representation of the token or injecting it into the attention weight of the downstream network can achieve fine-grained masking of high uncertainty tokens, thereby reducing their negative impact on the final judgment.

[0036] Finally, the representation sequence after masking is... Input a large language model. Simultaneously, confidence metrics and interpretability hints can be included to help users understand which tokens are considered highly uncertain and the extent of their impact. This end-to-end process is interconnected, achieving full-link uncertainty monitoring from semantic extraction to uncertainty assessment, dynamic masking, and final judgment.

[0037] To verify the effectiveness of this method, this embodiment first prepares a financial news dataset with 10,000 samples, with an average of about 100 tokens per text. The dataset is divided into a training set (8,000 samples), a test set (1,000 samples), and a validation set (1,000 samples). Each data point contains a text to be monitored and its corresponding label, with the label content being high uncertainty and low uncertainty.

[0038] This embodiment selects a 32-layer, 768-hidden-dimension multi-layer self-attention pre-trained language model to generate semantic representations. The model outputs a context representation matrix of shape [32, L, 768], where L is the number of tokens in the text. The semantic representation of the tokens output by the model includes both the semantics of each token itself and its contextual information, providing a reliable foundation for subsequent uncertainty assessment.

[0039] This embodiment establishes a semantic compression regular expression module, the structure of which is shown in Table 1.

[0040] Table 1. Structure of Semantic Compression Regular Expression Module

[0041] The semantic representation is input into the semantic compression regularization module. The semantic compression regularization module maps each 768-dimensional representation to 512 dimensions through a fully connected layer, and then performs BatchNorm, ReLU activation, and Dropout (ratio 0.5) sequentially to obtain smoother and more overfit-resistant features. Subsequently, it splits into two parallel fully connected branches, each predicting the mean and standard deviation of the latent distribution, and uses a reparameterization technique to sample and generate a 128-dimensional latent vector.

[0042] After obtaining the latent vectors, this embodiment calculates the KL divergence between the latent distribution and the prior distribution of each token, treating it as an uncertainty index. This value is then mapped to the [0,1] interval using a Sigmoid algorithm, producing uncertainty weights. For example, a token with a KL divergence of 1.2 will have a mapped value of approximately 0.77, indicating a strong risk tendency. After this step, a series of weights with the same length as the input are obtained, quantifying the potential contribution of each token to the overall uncertainty.

[0043] This implementation minimizes the classification cross-entropy loss and a KL regularization term with a weight of 0.1. The formula for calculating the loss function is as follows: .

[0044] Configure the Adam optimizer with a learning rate of 2×10. -5 The batch size is 32, the training is conducted for 200 epochs, and the F1-score is monitored on the validation set. The model is saved when the metric is optimal.

[0045] After training, this embodiment determines the optimal uncertainty threshold on the validation set using a grid search, and obtains... =0.6.

[0046] During inference, the uncertainty weight of each token is compared with a preset threshold. The comparison generates a binary mask: when the uncertainty weight is greater than 0.6, the token is masked; otherwise, the token is retained. The mask is then applied directly to the original 768-dimensional context representation or attention weights, so that high-uncertainty tokens are completely ignored in the downstream network, thereby achieving fine-grained removal of noise or misleading content.

[0047] Finally, the masked token representation sequence is input into a lightweight classification network (512→2 fully connected network), which outputs a "high uncertainty" or "low uncertainty" label and its corresponding confidence score. On the test set, this method achieved 89.5% accuracy, 80.1% recall, 84.6% precision, and 82.3% F1 score, respectively, which is an improvement of 4.3% accuracy, 7.6% recall, 4.5% precision, and 6.3% F1 score compared to the baseline method (without monitoring). It is also superior to sentence-level uncertainty monitoring methods, validating the effectiveness and reproducibility of this method. Test results are shown in Table 2.

[0048] Table 2 Monitoring Accuracy Results

[0049] This invention addresses the problems of coarse granularity, high false positive rate, and weak interpretability in existing text monitoring systems. It enables precise location and suppression of sources of high uncertainty at the token level, thereby improving the accuracy, robustness, and interpretability of monitoring. This invention also boasts excellent engineering embeddability, serving as a lightweight module to integrate with existing LLM inference and content review pipelines, making it suitable for scenarios such as financial risk control, sensitive information monitoring, and content compliance review.

[0050] This embodiment also provides a fine-grained text uncertainty monitoring system based on semantic compression regularization, such as... Figure 3 As shown, the technical solution adopted is as follows: including: Text acquisition module 1 is used to acquire the text to be tested; The context semantic fusion module 2 is used to segment the text to be tested into several tokens, and use a pre-trained large language model to perform context fusion on each token to obtain the semantic representation of each token; Semantic compression regularization module 3 is used to calculate the uncertainty contribution of semantic representation to the task based on semantic compression regularization technology, and obtain the uncertainty weight of each token. The monitoring result calculation module 4 is used to binarize the uncertainty weight according to the preset threshold, mark the high uncertainty token as the masked state, and keep the low uncertainty token in the available state to obtain the uncertainty monitoring result.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fine-grained text uncertainty monitoring method based on semantic compression regularization, characterized in that, Includes the following steps: S1: Obtain the text to be tested; S2: Segment the text to be tested into several tokens, and use a pre-trained large language model to perform context fusion on each token to obtain the semantic representation of each token; S3: Based on semantic compression regularization technology, calculate the uncertainty contribution of semantic representation to the task and obtain the uncertainty weight of each token; S4: Binarize the uncertainty weights according to the preset threshold, mark the high uncertainty tokens as the masked state, and keep the low uncertainty tokens as the available state to obtain the uncertainty monitoring results.

2. The fine-grained text uncertainty monitoring method based on semantic compression regularization as described in claim 1, characterized in that, In step S2, the pre-trained large language model adopts a multi-layer self-attention pre-trained language model.

3. The fine-grained text uncertainty monitoring method based on semantic compression regularization as described in claim 1, characterized in that, In step S2, the pre-trained large language model performs forward propagation on the test text to obtain the hidden states of each layer; the hidden representations of each layer are concatenated in a weighted sum manner to form a token-level context representation, and the semantic representation of each token is obtained.

4. The fine-grained text uncertainty monitoring method based on semantic compression regularization as described in claim 1, characterized in that, In step S3, uncertainty weights are calculated using a semantic compression regularization module, which includes an encoder and a decoder. In the encoder, the semantic representation is passed sequentially through the first fully connected layer, the first batch normalization, the first ReLU activation, random dropout, and two parallel fully connected branches, and then the latent variables are sampled using the reparameterization technique. In the decoder, the latent variables pass through the second fully connected layer, the second batch normalization, the second ReLU activation and random dropout, the third fully connected layer, and the Sigmoid function in sequence to obtain the uncertain weights.

5. The fine-grained text uncertainty monitoring method based on semantic compression regularization as described in claim 4, characterized in that, Two parallel fully connected branches are used to predict the mean and variance of the potential distribution, respectively.

6. A fine-grained text uncertainty monitoring method based on semantic compression regularization as described in claim 4 or 5, characterized in that, The closer the uncertainty weight is to 1, the higher the predictive uncertainty introduced by the token in the current context.

7. The fine-grained text uncertainty monitoring method based on semantic compression regularization as described in claim 1, characterized in that, In step S3, the semantic compression regularization module minimizes the weighted joint loss of classification loss and regularization term as the loss during the training phase.

8. The fine-grained text uncertainty monitoring method based on semantic compression regularization as described in claim 7, characterized in that, The loss function is: in, For loss function, For classification cross-entropy loss, To adjust the trade-off between semantic compression regularization and task discrimination, KL represents the Kullback-Leibler divergence. It follows a Gaussian posterior distribution. Let be a potential variable for the t-th Token. Let be the semantic representation vector of the t-th token. This is the prior distribution.

9. A fine-grained text uncertainty monitoring method based on semantic compression regularization as described in claim 1, characterized in that, In step S4, if the uncertainty weight is greater than a preset threshold, the corresponding Token is marked as blocked; otherwise, the corresponding Token is marked as available.

10. A fine-grained text uncertainty monitoring system based on semantic compression regularization, characterized in that, A fine-grained text uncertainty monitoring method based on semantic compression regularization as described in any one of claims 1 to 9, comprising: The text acquisition module is used to acquire the text to be tested. The context semantic fusion module is used to segment the text to be tested into several tokens, and use a pre-trained large language model to perform context fusion on each token to obtain the semantic representation of each token. The semantic compression regularization module is used to calculate the uncertainty contribution of semantic representation to the task based on semantic compression regularization technology, and obtain the uncertainty weight of each token. The monitoring result calculation module is used to binarize the uncertainty weights according to a preset threshold, mark high uncertainty tokens as masked, and keep low uncertainty tokens as available, thereby obtaining the uncertainty monitoring results.

Images