A method, device, and medium for filtering sensitive words in low-resource languages.

Abstract

This specification discloses a method, device, and medium for filtering sensitive words in low-resource languages, relating to the field of data processing technology. The method includes: acquiring original Lao text, determining a standardized Lao character sequence, performing multi-path word segmentation and disambiguation on the standardized Lao character sequence to generate multiple candidate word segments; extracting the contextual semantic vector of each candidate word segmentation sequence using a pre-trained Lao semantic coding model to determine the calibration semantic vector corresponding to the original Lao text; determining an initial verification result based on a pre-set lightweight classifier and the calibration semantic vector, the initial verification result including an initial verification confidence score; when the initial verification confidence score is less than a pre-set confidence threshold, performing multi-dimensional deep verification based on the calibration semantic vector and the standardized Lao character sequence to determine the deep verification result; and determining the final verification result using the initial verification result and the deep verification result for sensitive word filtering.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a method, device and medium for filtering sensitive words in low-resource languages. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, applications based on multilingual human-computer dialogue and social network content generation are becoming increasingly widespread. Lao, as an important official language in Southeast Asia, sees a continuous increase in demand for digital applications. However, Lao is a low-resource language, severely lacking in publicly available annotated corpora, pre-trained word vectors, and domain knowledge bases, posing a significant challenge to content security review in Lao-language scenarios.

[0003] Existing sensitive word filtering systems designed for mainstream Chinese and English languages, such as static dictionary-based string matching and supervised learning-based classification schemes, face adaptation challenges when transferred to the Lao language environment. Because Lao is an analytic language with continuous, non-spaced writing, it lacks explicit boundary markers between words in sentences. Furthermore, modifiers, adverbs, and complements are often placed after the headword, resulting in complex modifier orders and unique conjunction structures, further exacerbating the lack of explicit word boundaries. Static dictionary-based string matching schemes (such as the Aho-Corasick automaton) perform isolated matching at the lexical level, failing to perceive the overall semantic intent of the sentence. Consequently, they cannot reliably segment words, leading to a high rate of misclassifying normal words as sensitive words, posing a risk of false positives or false negatives, and severely impacting content flow efficiency and user experience. Moreover, supervised learning-based classification schemes train classifiers (such as SVM, CNN, RNN, or BERT pre-trained language models) on labeled corpora, directly classifying input text into binary or multi-class categories based on whether it violates regulations. However, in low-resource language environments such as Lao, it is difficult to obtain large-scale labeled corpora. Pre-trained language models lack sufficient unsupervised corpora for adequate training, resulting in representation collapse. The distance between different semantics in the vector space cannot be effectively distinguished, leading to the failure of cosine similarity retrieval. At the same time, the scarcity of labeled data leads to inflated confidence of the classifier.

[0004] Therefore, in the process of filtering sensitive words for low-resource languages ​​such as Lao, the linguistic characteristics of these languages ​​are not taken into account, resulting in a high rate of false negatives and a high rate of false positives. Summary of the Invention

[0005] This specification provides one or more embodiments of a method, device, and medium for filtering sensitive words in low-resource languages, which addresses the following technical problem: in the process of filtering sensitive words for low-resource languages ​​such as Lao, the language characteristics of low-resource languages ​​such as Lao are not considered, resulting in a high false negative rate and a high false positive rate.

[0006] One or more embodiments of this specification employ the following technical solutions: This specification provides one or more embodiments of a method for filtering sensitive words in a low-resource language. The method includes: acquiring original Lao text; preprocessing the original Lao text to determine a normalized Lao character sequence; performing multi-path segmentation and disambiguation on the normalized Lao character sequence to generate multiple candidate segmentation sequences, wherein the candidate segmentation sequences include character-level sequences and at least one sub-word-level sequence; extracting the contextual semantic vector of each candidate segmentation sequence using a preset Lao pre-trained semantic coding model to determine a calibration semantic vector corresponding to the original Lao text; determining an initial verification result based on a preset lightweight classifier and the calibration semantic vector, wherein the initial verification result includes an initial verification confidence score; when the initial verification confidence score is less than a preset confidence threshold, performing multi-dimensional deep verification based on the calibration semantic vector and the normalized Lao character sequence to determine a deep verification result; and determining a final verification result using the initial verification result and the deep verification result for sensitive word filtering.

[0007] This specification provides one or more embodiments of a low-resource language sensitive word filtering device, including: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the above-described method.

[0008] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0009] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By generating multiple candidate sequences at both the character and sub-word levels through multi-path segmentation and disambiguation, semantic distortion caused by errors in a single segmenter is avoided in Lao writing environments without spaces, reducing the risk of misjudgment and missed judgment due to blurred word boundaries. Contextual semantic vectors are extracted based on a pre-trained Lao semantic encoding model, effectively capturing Lao's unique syntactic structures such as post-modifiers and negative word associations under low-resource conditions. Whitening transformation is then used to calibrate the semantic vector distribution, solving the problem of representation collapse in the pre-trained model. A lightweight classifier combined with nearest neighbor consistency and segmentation divergence coefficients is used to correct confidence levels, reasonably lowering the confidence of samples at category boundaries or in semantically ambiguous regions, thus overcoming misjudgments caused by conventional classifiers. When confidence is insufficient, deep validation is triggered, ensuring high throughput while improving recall capabilities for complex and hidden violations. This effectively solves the technical problems of high misjudgment rates, lack of semantic understanding, insufficient coverage of novel variants, and the difficulty in balancing accuracy and efficiency in low-resource Lao environments. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating a method for filtering sensitive words in a low-resource language, provided in an embodiment of this specification. Figure 2 A schematic diagram of the system architecture of a low-resource language sensitive word filtering system provided for embodiments of this specification; Figure 3 A flowchart illustrating another method for filtering sensitive words in a low-resource language, provided as an embodiment of this specification. Figure 4 This is a schematic diagram of the structure of a low-resource language sensitive word filtering device provided in the embodiments of this specification. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0012] This specification provides a method for filtering sensitive words in low-resource languages. It should be noted that the executing entity in this specification can be a server or any device with data processing capabilities. Figure 1 A flowchart illustrating a low-resource language sensitive word filtering method provided in this specification is shown below. Figure 1 As shown, the main steps include the following: Step S101: Obtain the original Lao text, preprocess the original Lao text, determine the normalized Lao character sequence, and perform multi-path word segmentation and disambiguation on the normalized Lao character sequence to generate multiple candidate word segmentation sequences.

[0013] The candidate word segmentation sequence includes a character-level sequence and at least one sub-word-level sequence; In one embodiment of this specification, raw Lao text input by the user is obtained. This raw Lao text can be raw text information received via a network interface from a social media platform, instant messaging software, or chat platform. The raw Lao text is preprocessed to determine a normalized Lao character sequence. All characters in the raw Lao text are uniformly converted to the standard Lao Unicode encoding, eliminating inconsistencies in the representation of the same character due to different encoding methods. Furthermore, each character in the raw Lao text is traversed, removing noise symbols that do not belong to the basic Lao character set, numbers, punctuation marks, and common whitespace characters. The remaining whitespace characters are merged, consecutive whitespace characters are replaced with single spaces, and various punctuation marks are mapped to a unified standard form, resulting in a normalized Lao character sequence.

[0014] Lao is an analytical language written continuously without spaces and lacks high-quality word segmentation tools. Given these resource constraints, if the word segmenter incorrectly segments the boundaries of sensitive words, subsequent semantic analysis based on word units will result in either false positives or false negatives. Therefore, the embodiments in this specification perform multi-path word segmentation and disambiguation on the normalized Lao character sequence, generating multiple candidate word segments with different granularities, thus avoiding the error propagation risks that may arise from traditional methods relying on a single word segmenter.

[0015] Multi-path word segmentation and disambiguation are performed on the normalized Lao character sequence to generate multiple candidate word segments. Specifically, this includes: determining a character-level sequence for each character in the normalized Lao character sequence; obtaining a byte-pair encoding model pre-trained on unlabeled Lao corpus to perform sub-word segmentation on the normalized Lao character sequence to determine a first sub-word-level sequence; generating at least one candidate sub-word sequence using a pre-set unsupervised word segmentation model and the normalized Lao character sequence; removing sub-word-level sequences identical to the first sub-word-level sequence from the candidate sub-word sequences to determine a second sub-word-level sequence; and determining the multiple candidate word segments using the character-level sequence, the first sub-word-level sequence, and the second sub-word-level sequence.

[0016] In one embodiment of this specification, the specific implementation steps of multi-path word segmentation disambiguation are as follows: Each character in the normalized Lao character sequence is used as an independent minimum semantic unit. Without any merging operation, a character-level sequence is directly constructed. Each character in the character-level sequence retains its original order in the original sequence. The generation of the character-level sequence does not rely on a word segmentation dictionary or statistical model and completely preserves all information at the character level.

[0017] To obtain a byte-pair encoding model pre-trained on an unlabeled Lao corpus, in one example, the byte-pair encoding model in this embodiment can be a deterministic compression algorithm using statistical frequency. The corresponding training process is as follows: Collect a large-scale unlabeled Lao plain text corpus, for example, from publicly available Lao news websites, encyclopedias, or social media data. This corpus is stored in the form of multiple independent documents or sentences, requiring no preprocessing or feature extraction. When training the byte-pair encoding model, scan the unlabeled Lao plain text corpus, count the total frequency of all adjacent character pairs (i.e., two consecutive characters) in all documents, and obtain a complete character pair frequency table. Then, repeatedly perform a merging operation, determine the character pair with the highest frequency from the current frequency table, and merge them into a new sub-word unit; after merging, traverse the entire corpus again, updating the frequency of adjacent unit pairs affected by the merging. Iterate according to the above steps until the vocabulary reaches a preset vocabulary size or meets other set stopping conditions. After training, the order of the merging operations and the final byte-pair encoding vocabulary are saved for subsequent greedy longest match segmentation of any input sequence. The byte-pair encoding model's algorithm essentially employs a greedy merging strategy, using a pre-trained byte-pair encoding vocabulary to segment any input string into a sequence of known sub-word units.

[0018] The normalized Lao character sequence is input into the byte pair encoding model. According to the merging rules recorded during training, the model scans each character one by one, starting from the left side of the sequence. Each time, it prioritizes matching the longest sub-word unit in the vocabulary that starts from the current scanning position. If the match is successful, the sub-word unit is output as a segment and the scanning position is moved after the sub-word unit. This process is repeated until the end of the sequence, thereby dividing the entire character sequence into multiple sub-word units. The segmentation result is determined as the first sub-word level sequence.

[0019] Simultaneously, a pre-built unsupervised word segmentation model is invoked. This model is based on the principle of word segmentation without labeled data and can be implemented using generative methods based on character-level language models or compression methods based on minimum description length. For example, in one example, a character-level language model based on Long Short-Term Memory (LSTM) or Transformer can be used. This model learns the generation probability of each character given a preceding character sequence through autoregressive training on a large amount of unlabeled Lao corpus. Specifically, during the training phase, a large-scale unlabeled Lao plain text from the same source as the byte-pair encoded training corpus is collected. Each text is treated as a character sequence, and an LSTM or Transformer-based character-level language model is constructed. The input is the sequence of the first t-1 characters, and the output is the probability distribution of the t-th character. The model parameters are trained by maximizing the likelihood of all character sequences in the training corpus, learning the transition probabilities between characters. During the word segmentation phase, the negative log-likelihood of the sequence under different segmentation boundaries is calculated, and the segmentation path that maximizes the overall likelihood is selected as the output. Specifically, in the word segmentation stage, the Viterbi algorithm is used to search for the optimal segmentation boundary of the normalized Lao character sequence. Each possible word boundary position is considered as a decision point, and the language model provides the generation probability (i.e., the probability of the word appearing in the language model) for each candidate word (continuous character segment). The goal of word segmentation is to maximize the product of the character transition probabilities within each word of the entire sequence, and the optimal segmentation scheme is obtained through backtracking. To generate multiple candidate sub-word sequences, beam search can be used to retain the top K most probable segmentation paths as candidate outputs.

[0020] In another example, unsupervised statistical word segmentation algorithms, such as those utilizing point mutual information or normalized segmentation criteria, identify closely bound adjacent character pairs within a character sequence as word boundaries. A normalized Lao character sequence is input into the unsupervised word segmentation model, which calculates and outputs at least one candidate subword sequence through its internal algorithm.

[0021] It should be noted that unsupervised word segmentation models may generate multiple different segmentation assumptions, such as retaining multiple high-probability paths through beam search. This can lead to situations where the segmentation result of a candidate word sequence is completely identical in text content to the first word-level sequence generated by the byte-pair encoding model. To avoid generating duplicate candidate sequences in subsequent processing, a deduplication operation is performed on each candidate word sequence output by the unsupervised word segmentation model.

[0022] All candidate sub-word sequences output by the unsupervised word segmentation model are obtained. The text representation of each candidate sub-word sequence (i.e., the complete string formed by concatenating all sub-word units in sequence according to the segmentation boundaries) is compared character by character with the text representation of the first sub-word level sequence. If they are completely identical, the candidate sub-word sequence is discarded; if they are different, i.e., there are any character differences, the candidate sub-word sequence is retained. After the above elimination operation, a second sub-word level sequence that is different from the first sub-word level sequence is determined. It should be noted that after the elimination operation, the second sub-word level sequence may not exist, i.e., each candidate sub-word sequence has the same text representation as the first sub-word level sequence. Finally, the character level sequence, the first sub-word level sequence, and the second sub-word level sequence are combined to determine multiple candidate word segmentation sequences. These multiple candidate word segmentation sequences provide the finest-grained character-level and two different sources of sub-word level segmentation sequences.

[0023] The aforementioned technical solution addresses the low-resource characteristics of Lao, which features continuous writing without spaces and lacks publicly available mature word segmentation tools, thus solving the reliability problem of traditional single word segmenters in low-resource environments. By simultaneously generating character-level sequences, byte-pair encoded sub-word sequences, and candidate sequences based on an unsupervised word segmentation model, a multi-granularity input representation is constructed. The character-level sequences ensure that even if the word segmentation tool fails, subsequent semantic encoding can still be understood based on character units, effectively eliminating the impact of word segmentation ambiguity on semantic analysis. The byte-pair encoding model and the unsupervised word segmentation model, based on different statistical principles, retain differentiated segmentation assumptions through deduplication operations, avoiding redundant calculations. This allows subsequent attention fusion to obtain complementary input perspectives, reducing the false positive and false negative rates caused by blurred word boundaries.

[0024] Step S102: Extract the context semantic vector of each candidate word segmentation sequence through the preset Lao pre-trained semantic coding model to determine the calibration semantic vector corresponding to the original Lao text. Based on the preset lightweight classifier and calibration semantic vector, determine the initial verification result.

[0025] The initial verification result includes the initial verification confidence score; Using a pre-trained Lao semantic encoding model, the contextual semantic vector of each candidate word segmentation sequence is extracted to determine the calibration semantic vector corresponding to the original Lao text. Specifically, this involves: determining an initial embedding vector for each unit in the candidate word segmentation sequence; inserting a classification label at the beginning of the candidate word segmentation sequence; obtaining the learnable embedding vector corresponding to the classification label as the first position embedding of the sequence; inputting the entire embedding sequence containing the classification label embedding vector into the Transformer encoder stack of the semantic encoding model; dynamically calculating the attention bias matrix according to pre-defined Lao grammar rules during the multi-head self-attention calculation process of each Transformer encoder layer; adding the attention bias matrix to the self-attention score and then performing Softmax normalization to determine the contextual semantic vector corresponding to the candidate word segmentation sequence; performing attention fusion on the contextual semantic vectors of multiple candidate word segmentation sequences to determine the fused semantic vector; and performing representation calibration on the fused semantic vector according to a pre-defined calibration strategy to determine the calibration semantic vector corresponding to the original Lao text.

[0026] In one embodiment of this specification, semantic encoding is performed on each candidate word segmentation sequence, and the specific implementation process is as follows: Obtain the current candidate word segmentation sequence, which consists of several sub-word units or character units arranged in order. For each unit in the current candidate word segmentation sequence, i.e., each character or each sub-word, perform embedding mapping. Convert each unit into an initial embedding vector of fixed dimension through a pre-trained word embedding matrix. At the same time, insert a special classification label, such as the [CLS] label, at the beginning of the current candidate word segmentation sequence, and obtain the learnable embedding vector corresponding to the classification label from the parameters of the semantic coding model. Embed the learnable embedding vector as the first position of the sequence, so that the length of the embedding vector sequence of the entire sequence is increased by one compared with the length of the original sequence.

[0027] The entire embedding sequence, including the classification label embedding vectors, is input into the Transformer encoder stack of the pre-trained Lao semantic encoding model. The semantic encoding model is a multi-layer Transformer encoder structure, with each layer containing a multi-head self-attention sublayer and a feedforward neural network sublayer. During the multi-head self-attention calculation in each Transformer encoder layer, the attention bias matrix is ​​dynamically calculated according to pre-defined Lao grammar rules. When the part of speech corresponding to position j in the sequence is an adjective or adverb, and the part of speech corresponding to position i is a noun or verb and i is less than j, the attention bias value from position i to position j is increased. When the word corresponding to position i in the sequence is a negation word, the attention bias value is increased from position i to each position within a predetermined range to its right. The detailed calculation process of the attention bias matrix is ​​as follows: The first step is to obtain the part-of-speech tagging information corresponding to each position in the current layer input sequence and the negation identifier of whether the word corresponding to each position belongs to the preset negation word set. The part-of-speech tagging information can be obtained by real-time annotation of the current candidate word segmentation sequence through a preset lightweight part-of-speech tagger, such as a conditional random field-based model or a neural network-based sequence labeling model. The negation identifier is determined by querying a pre-built Lao negation word list, which includes commonly used negation words.

[0028] The second step involves scanning each adjective or adverb position in the input sequence as a modifier position based on the postmodifier rules in the Lao grammar rule base. The nearest noun or verb position to the left of each modifier position is then searched as the headword position. This determines the association between the modifier position and the headword position, and a first attention bias value is generated for this association. Specifically, based on the postmodifier rules in the Lao grammar rule base, each adjective or adverb position in the input sequence is scanned as a modifier position. The nearest noun or verb position within a predetermined window to the left of each modifier position is searched as the headword position modified by that modifier, thus determining a one-to-one association between the modifier position and the headword position. This process may involve traversing leftwards until the beginning of the sequence or until a syntactic boundary is encountered. A first attention bias value, a positive scalar, is generated for this one-to-one association and is used to enhance the attention weight from the headword position to the modifier position in subsequent self-attention calculations.

[0029] Simultaneously, based on the negation rules in the Lao grammar rule base, each negation word position in the input sequence is scanned. Within a predetermined window to the right of each negation word position, all modified word positions are identified, determining the association between the negation word position and each modified word position, and generating a second attention bias value for this association. Specifically, based on the negation rules in the Lao grammar rule base, each negation word position in the input sequence is scanned. Within a predetermined window to the right of each negation word position, for example, traversing several character or sub-word positions to the right, all word positions modified by the negation word are identified, typically adjacent verbs, adjectives, or adverbs, thus determining a one-to-many association between the negation word position and each modified word position. A corresponding second attention bias value is generated for each pair of associations. This second attention bias value is also a positive scalar, used to enhance the attention weight from the negation word position to the modified word position.

[0030] It should be noted that the specific values ​​of the first and second attention biases can be set to a fixed positive scalar constant, such as a bias value of 1.0, representing an additional weight added to the self-attention score. The magnitude of this value can be adjusted according to the confidence level of the Lao grammatical rules; a larger value indicates that the grammatical relation needs more attention from the model. In a preferred embodiment, the bias value is set to 0.5 for postmodifier relations and 0.8 for negation relations. The above values ​​are only examples and can be adjusted according to specific language characteristics and training effects. The predetermined window range refers to limiting the search range to the left or right of the current word position. For example, for postmodifier rules, the left window range can be set to search for a maximum of 5 characters or subwords to the left; if no noun or verb is found within this range, the bias for that modifier is abandoned. For negation rules, the right window range can be set to search for a maximum of 3 words to the right. The specific length of the window can be preset according to the average length of Lao sentences and language habits, or it can be specified by the user during deployment as a configurable parameter.

[0031] The third step is to sum the first attention bias value and the second attention bias value to the corresponding positions in the attention score matrix to obtain the attention bias matrix. Specifically, all the first and second attention bias values ​​generated above are summed according to their corresponding coordinates from the source position to the target position to the corresponding positions in an initial all-zero matrix with the same dimension as the attention score matrix, thus obtaining the attention bias matrix. This completes the calculation of the attention bias matrix.

[0032] After the attention bias matrix is ​​calculated, in the current multi-head self-attention sub-layer, the standard self-attention score matrix is ​​calculated by the dot product of the query vector and the key vector. Then, the corresponding positions of the attention bias matrix and the self-attention score matrix are added together. The result of the addition is subjected to Softmax normalization to obtain the attention weight matrix. Finally, the weight matrix is ​​multiplied by the value vector to obtain the output representation of the current layer.

[0033] Repeat the calculations for each layer until all Transformer encoder layers are passed. Extract the vector corresponding to the classification label position output by the top Transformer encoder. This vector incorporates the contextual semantic information of the entire sequence, determining the contextual semantic vector of the candidate word segmentation sequence. Perform the above process for each candidate word segmentation sequence to obtain the contextual semantic vector corresponding to each candidate word segmentation sequence. Perform attention fusion on multiple contextual semantic vectors. Concatenate the contextual semantic vectors of multiple candidate word segmentation sequences into a matrix. Calculate the attention weight of each vector through a small attention network (e.g., calculate a weighted sum based on the learnable weights of each vector). Then, perform a weighted sum on all vectors to obtain a fused semantic vector. The resulting fused vector integrates the semantic information from different word segmentation paths.

[0034] The aforementioned technical solution addresses the unique syntactic features of Lao, such as the postposition of attributives, adverbs, and complements, as well as the specific positions of negations. During the self-attention computation process, it enhances the semantic association weights between the head word and post-modifiers, and between the negation word and the modified component. It directly injects attention biases using a small number of deterministic grammatical rules, enabling the model to focus on Lao's unique syntactic structures even with low-quality pre-training or insufficient fine-tuning data. This significantly improves the semantic encoding's ability to identify sensitive post-modifier words and negative transitional sentences. Simultaneously, attention fusion of the semantic vectors from multiple candidate word segmentation sequences automatically evaluates the reliability of semantic representations under different segmentation paths, reducing the impact of single segmentation errors on the final semantic vector. This effectively solves the problems of lacking semantic contextual understanding and unreliable semantic representation quality in low-resource environments, common in existing low-resource languages.

[0035] According to the preset calibration strategy, the fused semantic vector is calibrated to determine the calibration semantic vector corresponding to the original Lao text. Specifically, this includes: obtaining an unlabeled Lao general text set; extracting semantic vectors from the Lao general text set to determine the Lao general text vector set corresponding to the Lao general text set; performing statistical analysis on the distribution of text vectors in the Lao general text vector set to determine the mean vector and covariance matrix of the Lao general text vector distribution; calculating the difference vector between the fused semantic vector and the mean vector; and determining the calibration semantic vector by raising the difference vector and the covariance matrix to a preset power.

[0036] In one embodiment of this specification, an unlabeled general Lao text set is obtained. The texts in this general Lao text set can be collected from publicly available Lao news websites, encyclopedias, social media, or authoritative announcements, and these texts do not carry any sensitive annotation information. It should be noted that this general Lao text set can be a portion of an unlabeled Lao plain text corpus, or it can be collected separately. For each text in the general Lao text set, the exact same series of operations as in the aforementioned steps from preprocessing to generating the fused semantic vector are sequentially performed: preprocessing, multi-path word segmentation and disambiguation, semantic encoding, and attention fusion, thereby obtaining the fused semantic vector corresponding to each general text, to construct a general Lao text vector set.

[0037] Statistical analysis is performed on the vector distribution in the Lao general text vector set. The arithmetic mean of all vectors in the Lao general text vector set is calculated to determine the mean vector, which represents the center position of the general Lao text in the semantic vector space. Simultaneously, the covariance matrix of all vectors in the Lao general text vector set is calculated. This covariance matrix reflects the dispersion of vectors in different dimensions and the correlation between dimensions. The fused semantic vector to be calibrated is obtained, and the difference vector between the fused semantic vector and the mean vector is calculated to represent the offset of the current semantic vector relative to the center of the general text. Then, the covariance matrix is ​​mathematically transformed to calculate its preset negative fractional power (e.g., negative 1 / 2 power). A matrix multiplication operation is performed between this preset negative fractional power of the covariance matrix and the difference vector, i.e., the negative fractional power of the covariance matrix is ​​left-multiplied by the difference vector to determine the calibration semantic vector.

[0038] It should be noted that when the number of samples in the unlabeled Lao general text set used for statistical analysis is less than the dimension of the semantic vector, the covariance matrix may be a singular matrix, and its negative fractional powers cannot be directly calculated. To address this, the covariance matrix is ​​regularized by adding a small positive constant, such as 1, to the diagonal of the covariance matrix. e-6 The product of the matrix and the identity matrix is ​​then used, and the regularized covariance matrix is ​​used for subsequent negative fractional power calculations. This regularization operation ensures the positive definiteness of the matrix, allowing the whitening transformation to be performed stably under all conditions.

[0039] To address the representation collapse problem that easily occurs in low-resource language pre-trained language models under insufficient corpus conditions, this paper avoids directly using the raw vectors output by the model for similarity calculation or classification in low-resource environments. Instead, it pre-collects unlabeled general Lao text, calculates the mean vector and covariance matrix of its vector distribution, and then performs a whitening transformation on the fused semantic vector of each input sample, stretching the vector distribution from anisotropic narrow cone to isotropic spherical. This process does not rely on sensitive labeled data and can be completed using only unlabeled general text, thus meeting the language characteristic of Lao's scarcity of labeled resources. After calibration, the Euclidean distance and cosine similarity between the semantic vectors can accurately reflect the semantic proximity, significantly improving the classification accuracy of the subsequent lightweight classifier and the reliability of similarity matching in the semantic vector retrieval module.

[0040] The initial verification result is determined using a preset lightweight classifier and the calibration semantic vector. Specifically, this includes: inputting the calibration semantic vector into the lightweight classifier to output an initial risk category and an initial confidence score; correcting the initial confidence score based on the calibration semantic vector and the multiple candidate word segments to determine an initial verification confidence score; and using the initial verification confidence score and the initial risk category to determine the initial verification result.

[0041] In one embodiment of this specification, a calibration semantic vector is input into a lightweight classifier to output an initial risk category and an initial confidence score. In one example, the lightweight classifier typically employs a multilayer perceptron structure, with the input layer dimension being the same as the dimension of the calibration semantic vector, and the number of output layer nodes corresponding to a preset number of risk categories, such as normal, moderate risk, high risk, and severe risk. After forward propagation, the lightweight classifier outputs a probability distribution corresponding to multiple risk categories, where the probability value corresponding to each risk category is the prediction confidence for that risk category. The risk category with the highest probability is selected as the initial risk category, and its corresponding probability value is taken as the initial confidence score. However, the initial confidence score only reflects the degree of certainty of the lightweight classifier's prediction and does not consider the local consistency of samples in the feature space or the impact of word segmentation ambiguity; therefore, it needs to be corrected.

[0042] Based on the calibration semantic vector and the multiple candidate word segmentation sequences, the initial confidence score is corrected to determine the initial verification confidence score. Specifically, this includes: obtaining the training sample vector space corresponding to the lightweight classifier; calculating the vector distance between each training sample vector and the calibration semantic vector within the training sample vector space; determining a predetermined number of closest specified training sample vectors according to the vector distance from closest to furthest to obtain the class label of each specified training sample vector; counting the number of specified training samples in the predetermined number of specified training sample vectors whose class label is the same as the initial risk class; and comparing the number of specified training samples in the same class with the initial risk class. The ratio of the number of training sample vectors is specified to determine the nearest neighbor consistency ratio; the preliminary risk category corresponding to each candidate word segmentation sequence is obtained to determine the sequence pair formed by any two candidate word segmentation sequences among the multiple candidate word segmentation sequences, and the total number of sequence pairs corresponding to the multiple candidate word segmentation sequences is counted; the preliminary risk categories of the two candidate word segmentation sequences in each sequence pair are compared to see if they are the same, so as to count the divergent sequence pairs with different preliminary risk categories and determine the number of divergent sequence pairs; the classification divergence coefficient is determined according to the ratio of the number of divergent sequence pairs to the total number of sequence pairs; the initial confidence score is corrected by the classification divergence coefficient and the nearest neighbor consistency ratio to determine the initial validation confidence score.

[0043] In one embodiment of this specification, the training sample vector space corresponding to the lightweight classifier is obtained. When training the lightweight classifier, the training samples consist of original Lao text and manually labeled risk categories. Each training sample is processed according to the aforementioned complete process from preprocessing to obtaining a calibration semantic vector. The mean and covariance matrices used for the whitening transformation corresponding to the training sample are pre-obtained statistically from the fused semantic vectors of the unlabeled Lao general text vector set. A calibration semantic vector corresponding to the training sample is generated, and the calibration semantic vector and the labeled category of the sample are stored, thus determining the training sample vector space corresponding to the lightweight classifier. In the training sample vector space, the vector distance between each training sample vector and the calibration semantic vector is calculated. This vector distance can be Euclidean distance or cosine distance. Then, all training samples are sorted in ascending order of vector distance, and a predetermined number of training sample vectors with the closest distance are selected, i.e., designated training sample vectors. Obtain the class label associated with each specified training sample vector, and count the number of samples whose class labels are the same as the initial risk class output by the current lightweight classifier. The ratio of the number of such samples to the total number of specified training sample vectors is determined as the nearest neighbor consistency ratio. The obtained nearest neighbor consistency ratio reflects the class distribution of the training samples around the current sample. If the class of most of the nearest neighbor samples is consistent with the classifier's prediction, it means that the current sample is located inside the class region, thus verifying that the lightweight classifier's prediction is relatively reliable. Conversely, if most of the nearest neighbor samples belong to other classes, it means that the current sample is located near the class boundary or there is labeling noise, and the reliability of the classifier's prediction should be reduced.

[0044] It's important to note that in conventional sensitive word classification methods, the confidence score output by the classifier only reflects the model's certainty about its own predictions. This confidence score is typically calculated solely based on the probability distribution output by the model. However, for low-resource languages ​​like Lao, which are written continuously without spaces, the same character sequence can produce completely different semantic interpretations at different segmentation boundaries. The resulting segmentation ambiguity is an inherent source of semantic ambiguity in the input text, independent of the model's own uncertainty. If we rely solely on the classifier's confidence score and ignore the impact of segmentation ambiguity, for texts with unclear segmentation boundaries and differing risk category judgments under different segmentation methods, unreliable predictions with high confidence scores may be output, leading to missed or false positives. Therefore, we introduce the degree of divergence in the classification results of multiple candidate segmentation sequences as a quantitative measure of the input semantic ambiguity and use it to correct the original confidence score. This ensures that the confidence score simultaneously reflects the certainty of the model's predictions and the inherent ambiguity of the input text, providing a more reliable decision-making basis for subsequent tiered processing.

[0045] In the aforementioned steps, multiple candidate word segmentation sequences were determined through a multi-path word segmentation disambiguation process, and the contextual semantic vector of each candidate word segmentation sequence was extracted. After extracting the contextual semantic vector of each candidate word segmentation sequence, in addition to generating a fused semantic vector and an initial risk category, it is also used to calculate the classification divergence coefficient, with no order dependency in execution. The contextual semantic vector of each candidate word segmentation sequence is input into a pre-trained lightweight classifier. In this embodiment, the classifier is a multilayer perceptron structure, with the input layer dimension being the same as the contextual semantic vector dimension, and the output layer corresponding to a preset risk category, such as normal, moderate risk, high risk, and severe risk. The classifier outputs the probability distribution of each candidate word segmentation sequence through forward propagation, and takes the category with the highest probability as the initial risk category of each candidate word segmentation sequence. Since different candidate word segmentation sequences come from different segmentation methods, the corresponding initial risk categories may be the same or different. After obtaining the preliminary risk categories of all candidate sequences, any two candidate sequences are paired to determine the total number of sequence pairs. The number of sequence pairs with different preliminary risk categories is then counted, representing the number of divergent sequence pairs. The ratio of the number of divergent sequence pairs to the total number of sequence pairs is calculated to obtain the classification divergence coefficient. The classification divergence coefficient ranges from 0 to 1. If the preliminary risk categories of all candidate word segmentation sequences are completely identical, the divergence coefficient is 0. It should be noted that the classification divergence coefficient reflects the degree of difference in risk judgment caused by word segmentation ambiguity. The greater the divergence, the higher the uncertainty of the word segmentation boundary of the current text, and the subsequent confidence level should be appropriately reduced.

[0046] The initial confidence score is obtained by multiplying the nearest neighbor consistency ratio and 1 minus the classification divergence coefficient sequentially. The correction formula uses 1 minus the classification divergence coefficient as the multiplicative factor, reflecting the degree of inconsistency in risk category judgments under different segmentation boundaries. When the divergence coefficient is 0, all candidate sequences have identical initial risk categories, indicating that the segmentation boundary has no impact on semantic understanding; in this case, the correction factor is 1, and there is no need to reduce the confidence. When the divergence coefficient approaches 1, meaning the prediction results of each candidate sequence are extremely dispersed, it indicates semantic ambiguity in the text itself, and arbitrarily choosing a segmentation method is unreliable; in this case, the correction factor approaches 0, and the confidence should be almost zeroed, forcibly triggering deep validation. The initial validation confidence score obtained in this way considers both the local class consistency of the current sample in the training sample space and the degree of prediction divergence caused by segmentation ambiguity. The initial validation confidence score and the previously determined initial risk category are used as the initial validation result for subsequent confidence threshold comparison and deep validation trigger judgment.

[0047] The original confidence level is corrected by using the nearest neighbor consistency ratio and the classification divergence coefficient. The nearest neighbor consistency ratio utilizes local geometric information in the training sample vector space to reasonably suppress the confidence level of isolated or boundary samples that deviate from the mainstream class distribution, thereby reducing false positives. The classification divergence coefficient utilizes candidate sequences generated by multi-path segmentation disambiguation. By comparing the consistency of classification results under different segmentation methods, it quantifies the impact of segmentation uncertainty on semantic understanding. When different segmentation paths yield different risk categories, it indicates that the current text itself has semantic ambiguity. At this time, reducing the confidence level and triggering deep validation can identify uncertain samples of the model with extremely low computational cost and transfer them to the deep validation process, while high-confidence samples can be directly output as results, thus maintaining high processing throughput without sacrificing accuracy.

[0048] Step S103: When the initial verification confidence score is less than the preset confidence threshold, multi-dimensional deep verification is performed based on the calibration semantic vector and the normalized Lao character sequence to determine the deep verification result. The final verification result is determined based on the initial verification result and the deep verification result for sensitive word filtering.

[0049] If the initial verification confidence score is not less than the preset confidence threshold, the final result is determined based on the initial verification result. Sensitive word filtering is performed using the risk categories in the initial verification result to determine the final review conclusion (pass or block), and the result is returned to the caller via the review result output interface. If the initial verification confidence score is less than the preset confidence threshold, multi-dimensional deep verification is performed to determine the deep verification result. The final verification result is determined based on the initial verification result and the deep verification result, and then sensitive word filtering is performed. Sensitive word filtering is performed using the risk categories in the final verification result to determine the final review conclusion (pass or block), and the result is returned to the caller via the review result output interface.

[0050] Multi-dimensional deep verification is performed based on the calibrated semantic vector and the normalized Lao character sequence to determine the deep verification result. Specifically, this includes: acquiring a pre-constructed sensitive semantic vector library and a pre-set Lao grammar rule library, wherein the Lao grammar rule library includes rules for sensitive words with post-modifiers, rules for detecting conjunction structures, rules for collocation of function words and sensitive words, rules for mapping dialect vocabulary to standard language vocabulary, and rules for normalizing Lao character variants; and calculating the cosine similarity between the calibrated semantic vector and each sensitive semantic vector in the sensitive semantic vector library through a pre-defined approximate nearest neighbor search, determining the specified sensitive semantic vector with the highest cosine similarity. Semantic vector; based on the cosine similarity and risk category corresponding to the specified sensitive semantic vector, determine the semantic retrieval similarity score and the first candidate risk category; based on the Lao grammar rule base, perform rule matching on the normalized Lao character sequence to determine the rule risk score; through the rule risk score and the preset risk category threshold range, determine the second candidate risk category corresponding to the rule risk score; perform weighted fusion of the semantic retrieval similarity score and the rule risk score to determine the deep verification confidence score, and determine the deep verification risk category based on the higher-level risk category among the first candidate risk category and the second candidate risk category.

[0051] In one embodiment of this specification, during the multi-dimensional deep verification process, a pre-constructed sensitive semantic vector library and a pre-set Lao grammar rule library are acquired. The sensitive semantic vector library includes Lao sensitive text samples that have been manually identified as sensitive, and each Lao sensitive text sample is labeled with a sensitive risk type, such as general risk, high risk, and severe risk. Each Lao sensitive text sample is preprocessed, multi-path word segmentation and disambiguation, semantic encoding, and attention fusion according to the aforementioned steps, and whitening transformation is performed using the same whitening parameters to generate a calibration semantic vector. The sensitive semantic vectors corresponding to the Lao sensitive text samples and the sensitive risk types are associated and stored to construct the sensitive semantic vector library.

[0052] An approximate nearest neighbor search is performed on the calibration semantic vector of the current input text. An approximate nearest neighbor algorithm, such as an index structure based on local sensitive hashing or product quantization, is used to quickly find the vector most similar to the calibration semantic vector in the sensitive semantic vector library. By pre-indexing the sensitive semantic vector library, the retrieval complexity is reduced from linear scanning to logarithmic or constant level. During the retrieval process, the cosine similarity between the calibration semantic vector and each sensitive semantic vector in the library is calculated. The closer the cosine similarity is to 1, the more consistent the directions of the two vectors are in the semantic space. The vector with the highest cosine similarity among all the calculated results is selected as the designated sensitive semantic vector. The cosine similarity value corresponding to the designated sensitive semantic vector is determined as the semantic retrieval similarity score, reflecting the semantic closeness between the current input text and known sensitive samples. Simultaneously, based on the risk category associated with the designated sensitive semantic vector, it is determined as the first candidate risk category, indicating the risk level to which the current text might belong based solely on semantic similarity.

[0053] Based on the Lao grammar rule base, rule matching is performed on the standardized Lao character sequence to determine the rule risk score. Specifically, this includes: obtaining the basic risk weight corresponding to each grammar rule in the pre-set Lao grammar rule base, and determining the total basic risk weight corresponding to the Lao grammar rule base; performing rule matching on the standardized Lao character sequence according to the Lao grammar rule base, and determining the matching status corresponding to each grammar rule; when the matching status of the grammar rule is a successful match, accumulating the basic risk weight corresponding to the grammar rule to determine the cumulative weight for successful matching; and determining the rule risk score by the ratio of the cumulative weight for successful matching to the total basic risk weight.

[0054] In one embodiment of this specification, the basic risk weight corresponding to each grammatical rule in the pre-set Lao grammar rule base is obtained. Specifically, this includes a first basic risk weight corresponding to the post-modifier sensitive word rule, a second basic risk weight corresponding to the conjunction structure detection rule, a third basic risk weight corresponding to the function word and sensitive word collocation rule, a fourth basic risk weight corresponding to the dialect vocabulary to standard language vocabulary mapping rule, and a fifth basic risk weight corresponding to the Lao character variant normalization rule. The above basic risk weights are summed to obtain the total basic risk weights corresponding to the Lao grammar rule base. According to the Lao grammar rule base, rule matching is performed on the normalized Lao character sequence. The specific rule matching process is as follows: In Lao, adjectives and adverbs typically follow the nouns or verbs they modify, which is the opposite of the word order of pre-modifiers in mainstream languages ​​such as Chinese and English. Conventional sensitive word detection methods only match the words themselves and cannot identify whether a modifier is sensitive in a structure where the head word follows the modifier. In this embodiment, a sensitive modifier lexicon is obtained for matching the sensitive word rule of the post-modifier. The part-of-speech tagging results of the normalized Lao character sequence are analyzed, and all adjectives and adverbs are identified as candidate modifiers. For each candidate modifier, the nearest noun or verb is searched within a predetermined window to its left as the head word. It is then determined whether the candidate modifier belongs to the sensitive modifier lexicon. If it does, and the head word does not belong to the sensitive word lexicon, the rule is considered to have matched successfully, and the first basic risk weight is accumulated. Specifically, the normalized Lao character sequence is annotated using a part-of-speech tagger to identify the positions of all adjectives and adverbs, and these words are used as candidate modifiers. Within a predetermined window to the left (i.e., in front) of each candidate modifier (e.g., searching to the left within a distance of no more than a certain number of characters or subwords), the nearest noun or verb is searched, and this noun or verb is taken as the head word to be modified. It is then determined whether the candidate modifier belongs to a pre-built sensitive modifier lexicon, which collects common Lao adjectives and adverbs with negative connotations such as insult, discrimination, and attack. If it does, and the head word is not in the sensitive word lexicon (i.e., the head word itself is not a sensitive word), then the rule is considered a successful match. This avoids misjudging ordinary modifiers in neutral contexts as sensitive; for example, the adjective "good" is not sensitive when modifying a person, but may constitute a violation if it modifies a specific sensitive noun. Upon successful matching, the corresponding first basic risk weight is accumulated and used to calculate the rule risk score.

[0055] The Lao language contains the word " ... Special conjunction structures such as "(both...and..." are used to connect two parallel components, emphasizing the simultaneous existence of two properties or states. Some sensitive intentions may be hidden in the combined semantics of the parallel components; individual components may not be sensitive, but the combination expresses sensitive meanings such as irony, belittling, or attack. In the embodiments of this specification, when performing rule matching according to the conjunction structure detection rules, the normalized Lao character sequence is scanned to detect the existence of Lao conjunction structures. If they exist, the first text segment of the first parallel component and the second text segment of the second parallel component are extracted. Sensitive word detection is performed on both the first and second text segments. If no sensitive words are found, they are combined into a joint text segment, and semantic risk judgment is performed. If the semantic risk judgment is sensitive, the rule matching is considered successful, and the second basic risk weight is accumulated. Specifically, the normalized Lao character sequence is scanned, and the existence of predefined conjunction structure markers, such as the first conjunction "...", is detected through regular expression matching or string search. "and the second conjunction" If a conjunction exists, the text fragments of the first parallel component (located after the first conjunction and before the second conjunction) and the second parallel component (located after the second conjunction) are extracted based on their positions. Sensitive word detection is then performed on these two text fragments separately, for example, by calling a lightweight sensitive word library or regular expression matching. Only when neither fragment individually matches any sensitive words are the two fragments combined into a joint text fragment, for example, by connecting them with a space or a conjunction. Semantic risk assessment is then performed on this joint fragment, using a simple semantic classification model or rule template to determine whether the combined expression has a sensitive intent. If the semantic risk assessment result is sensitive, the rule is considered a successful match. This logic can capture expressions in the "both A and B" structure where A and B are harmless individually but become offensive or ironic when combined, compensating for the blind spots of single-fragment detection.

[0056] As an isolating language, Lao's grammatical meaning relies heavily on function words, such as negations, modal particles, and prepositions. Certain sensitive sentence structures require the combination of function words with specific sensitive words to trigger complete sensitive semantics. For example, negations and sensitive verbs may express ironic affirmation, while suggestive modal verbs and sensitive action words may express instigation or sedition. Based on this, in the embodiments of this specification, a pre-constructed table of function word and sensitive word collocation rules is obtained. The function word identifier in the collocation rule table is retrieved from the normalized Lao character sequence. Then, a search is conducted within a predetermined window to the right of the function word position to determine if a corresponding sensitive word exists in the collocation rule table. If it exists, the rule is considered a successful match, and the third basic risk weight is accumulated. Specifically, this is achieved by pre-constructing a table of function word and sensitive word collocation rules, where each rule contains a function word identifier (e.g., "..."). " indicates "no" The system uses a combination of methods, including: identifying the function word ("should"), sensitive word category identifiers (e.g., insulting verbs, inflammatory nouns), and corresponding risk weights. It retrieves the position of the function word identifier within the normalized Lao character sequence, and then searches within a predetermined window to the right of that position (e.g., looking up no more than a certain number of words to the right) to check for words belonging to the sensitive word category specified by the rule. If found, the rule is considered a successful match, and the corresponding third basic risk weight is accumulated. This effectively identifies sensitive sentence structures that rely on function word modulation, rather than solely on the appearance of the sensitive words themselves.

[0057] Lao has three major dialect regions: Vientiane, Luang Prabang, and Champasak. Different dialect regions may use different vocabulary to express the same thing. Standard Lao is based on the Vientiane dialect, while pre-trained semantic models and sensitive word libraries are usually based on Standard Lao. Dialect words may be identified as out-of-vocabulary words or generate incorrect semantic vectors in the Standard Lao model, leading to missed detections. Therefore, in this embodiment, a pre-constructed Lao dialect-Standard Lao mapping library is obtained, corresponding to the mapping rules from dialect words to Standard Lao words. The normalized Lao character sequence is segmented to obtain a sequence of word units. Each word unit is then matched against the dialect words in the mapping library. If a match is found, the word unit is replaced with the corresponding Standard Lao word, resulting in a Standard Lao text sequence. This Standard Lao text sequence is then input into the sensitive word library for matching. If a match is successful, the rule is considered matched successfully, and the fourth basic risk weight is accumulated. Specifically, a pre-constructed Lao dialect-Standard Lao mapping library is obtained. This mapping library is constructed by linguists or through parallel corpus mining, and each entry records a dialect word and its corresponding Standard Lao word. The standardized Lao character sequence is segmented using the same segmenter as described in the multi-path segmentation and disambiguation section, or a lightweight segmentation tool, to obtain a sequence of lexical units. Each lexical unit is then matched against dialect words in a mapping library, using either exact or fuzzy matching. If a match is found, the lexical unit is replaced with the corresponding standard Lao word. After all replacements are complete, a standard Lao text sequence is obtained. This standard Lao text sequence is then input into a sensitive word library (such as a general sensitive word library) for matching. If a match is successful, the rule is considered to have been successfully matched. Converting dialect expressions to standard Lao before detection significantly improves the recognition coverage of sensitive content in dialect areas.

[0058] In real-world scenarios, users often employ character variation techniques to circumvent sensitive word detection. For example, they might replace a letter in a sensitive word with another letter that has a similar shape (such as a Lao letter). and (Similar shapes), replace with numbers (such as 5 and Lao letters) (Similar shapes), or deletion / replacement of vowel symbols. Based on this, the embodiments of this specification obtain a pre-constructed character confusion pair mapping table corresponding to the Lao character variant normalization rule. For each character in the normalized Lao character sequence, if the character exists in the character confusion pair mapping table, it is replaced with the standard character in the mapping table to obtain a normalized variant text sequence. The normalized variant text sequence is input into a sensitive word library for matching. If the match is successful, the rule is determined to be matched successfully and the fifth basic risk weight is accumulated. Specifically, each character in the normalized Lao character sequence is traversed. If the character exists in the mapping table, that is, it belongs to the known confusion character, it is replaced with the corresponding standard character in the mapping table; if it does not exist, it remains unchanged. After processing all characters, a normalized variant text sequence is obtained. The sequence is input into a sensitive word library for matching. If the match is successful, the rule is determined to be matched successfully. Without relying on any machine learning model, most common variant forms can be restored by deterministic character replacement alone. The computational overhead is extremely low and the interpretability is strong, making it particularly suitable for real-time detection in low-resource environments.

[0059] Rule matching is performed according to the above rules. When the matching status of a grammar rule is "match successful", the basic risk weight corresponding to the grammar rule is accumulated to determine the cumulative weight of successful matching. The rule risk score is determined by the ratio of the cumulative weight of successful matching to the sum of the basic risk weights.

[0060] The semantic retrieval similarity score and the rule risk score are weighted and fused. The semantic retrieval similarity score is multiplied by a first weight coefficient, and the rule risk score is multiplied by a second weight coefficient. The sum of the two yields the deep validation confidence score. The values ​​of the two weight coefficients can be pre-set according to the application scenario's level of trust in semantic retrieval and the rule engine. For example, the weight of semantic retrieval is increased when it is necessary to prioritize capturing novel semantic variations, and the weight of the rule engine is increased when it is necessary to strictly follow known rules. Simultaneously, the risk levels of the first and second candidate risk categories are compared. A predefined partial order relationship exists between risk levels: severe risk is higher than higher risk, higher risk is higher than general risk, and general risk is higher than normal. The higher level is taken as the deep validation risk category. The deep validation confidence score and the deep validation risk category are determined as the deep validation result.

[0061] The final verification result is determined based on the initial verification result and the deep verification result for sensitive word filtering. Specifically, the risk levels of the initial risk category and the deep verification risk category are compared, and the higher risk level is selected as the final risk category. The confidence scores of the initial verification and the deep verification are compared, and the higher confidence score is selected as the final confidence score. When the deep verification obtains a higher confidence score due to rule matching or semantic retrieval, its result can override the initial verification result. If the two risk categories are inconsistent but the confidence scores are similar, the higher risk level is used to ensure security priority. Sensitive word filtering is performed according to the final verification result. The final review conclusion (pass or block) is determined by the final confidence score in the final verification result, and the corresponding risk level (general risk / higher risk / serious risk) is marked according to the final risk category.

[0062] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By generating multiple candidate sequences at both the character and sub-word levels through multi-path segmentation and disambiguation, semantic distortion caused by errors in a single segmenter is avoided in Lao writing environments without spaces, reducing the risk of misjudgment and missed judgment due to blurred word boundaries. Contextual semantic vectors are extracted based on a pre-trained Lao semantic encoding model, effectively capturing Lao's unique syntactic structures such as post-modifiers and negative word associations under low-resource conditions. Whitening transformation is then used to calibrate the semantic vector distribution, solving the problem of representation collapse in the pre-trained model. A lightweight classifier combined with nearest neighbor consistency and segmentation divergence coefficients is used to correct confidence levels, reasonably lowering the confidence of samples at category boundaries or in semantically ambiguous regions, thus overcoming misjudgments caused by conventional classifiers. When confidence is insufficient, deep validation is triggered, ensuring high throughput while improving recall capabilities for complex and hidden violations. This effectively solves the technical problems of high misjudgment rates, lack of semantic understanding, insufficient coverage of novel variants, and the difficulty in balancing accuracy and efficiency in low-resource Lao environments.

[0063] Figure 2 This specification provides a schematic diagram of the system architecture for a low-resource language sensitive word filtering system, as illustrated in the embodiments provided. Figure 2 As shown, the system consists of three layers: the interface layer, the core processing layer, and the data resource layer. The components and their connections are described below: The interface layer consists of a text input interface and an audit result output interface. The text input interface is responsible for receiving raw Lao text from upper-layer applications (such as dialogue systems and social platforms) and passing it to the input preprocessing module; the audit result output interface is responsible for returning the final audit conclusion and risk level from the decision output module to the caller.

[0064] The core processing layer consists of an input preprocessing module, a semantic encoding module, a lightweight classifier module, a semantic retrieval and rule validation module, and a decision output module connected in series. The input preprocessing module receives raw text from the text input interface, performs Unicode encoding normalization, noise character filtering, and punctuation standardization, and outputs a normalized text sequence to the semantic encoding module. The semantic encoding module integrates a pre-trained Lao language model using a Transformer encoder architecture. Its input is connected to the input preprocessing module, receiving the normalized text sequence; its output is a dense vector v containing deep semantic information, which is simultaneously connected to the input of the lightweight classifier module. The lightweight classifier module uses a Multi-Layer Perceptron (MLP) structure. Its input receives the semantic vector v from the semantic encoding module; its output is split into two paths based on a confidence threshold θ. When the confidence is higher than θ, the classification result is directly connected to the decision output module; when the confidence is lower than θ, the semantic vector v is connected to the semantic retrieval and rule validation module. The semantic retrieval and rule validation module consists of a semantic vector retrieval submodule and a rule engine submodule connected in parallel. The semantic vector retrieval submodule receives a semantic vector v and uses an Approximate Nearest Neighbor (ANN) retrieval algorithm to find similar vectors in a sensitive semantic vector library. The rule engine submodule receives normalized text and performs structured text analysis based on a pre-built Lao grammar rule library. The outputs of the two submodules are combined and then passed to the decision output module. The decision output module receives intermediate conclusions from the lightweight classifier module or the semantic retrieval and rule validation module, performs confidence-weighted fusion, outputs the final review conclusion (pass / block) and risk level, and returns it to the caller via the review result output interface.

[0065] The data resource layer is used to store three types of data required for system operation, mainly including Lao pre-trained language model weights (for the semantic encoding module to load and call), sensitive semantic vector library (for the semantic vector retrieval submodule to perform similarity retrieval), and Lao grammar rule library (for the rule engine submodule to perform rule matching).

[0066] Figure 3 A flowchart illustrating another method for filtering sensitive words in low-resource languages ​​provided in this specification is shown below. Figure 3 As shown, the main implementation steps are as follows: Step S1 (Input Preprocessing): Receive Lao text input from the user, perform Unicode encoding format conversion, remove noise characters outside the Lao character set, normalize punctuation marks and whitespace characters, and output a clean, normalized text sequence.

[0067] Step S2 (Semantic Encoding): Input the normalized text into the Lao pre-trained language model, perform multi-head self-attention calculation on the input sequence through the Transformer encoder, extract the [CLS] position vector of the final hidden layer as the semantic representation of the whole sentence, and output a dense semantic vector v with fixed dimensions.

[0068] Step S3 (Lightweight Classification): Input the semantic vector v into the lightweight classifier module, perform linear transformation and Softmax normalization, and output the confidence probability distribution of each classification label; if the highest confidence score is higher than the preset confidence threshold θ, the corresponding classification label is directly passed to the decision output module; if the highest confidence score is lower than θ, the semantic vector v is passed to the semantic retrieval and rule verification module for deep verification.

[0069] Step S4 (Deep Verification): The semantic retrieval submodule calculates the cosine similarity between the semantic vector v and each stored vector in the pre-built sensitive semantic vector library, retrieves the candidate options with the highest similarity, and determines whether the similarity threshold is exceeded; the rule engine submodule performs Lao grammar rule matching on the normalized text, identifies negative structures, special sentence templates, and known avoidance expression patterns; the two submodules combine to output the verification conclusion.

[0070] Step S5 (Decision Fusion Output): The decision output module performs weighted fusion of the classifier confidence score and the deep verification conclusion, outputs the final review conclusion (pass or block) according to the preset decision strategy, and marks the corresponding risk level (general risk / higher risk / serious risk).

[0071] In the aforementioned technical solution, the Lao pre-trained language model in the semantic encoding module, through the Transformer multi-head self-attention mechanism, considers the complete context sequence when encoding each word. The generated semantic vector v carries sentence-level overall intent information, rather than isolated word-level features. Compared to traditional dictionary matching that relies solely on word strings, the semantic vector-based classifier module can distinguish between benign and malicious uses of the same word in different contexts, thus significantly reducing misjudgments caused by polysemous words. Furthermore, the sensitive semantic vector library in the data resource layer stores the semantic vector prototypes of known sensitive content. The semantic retrieval and rule verification module can identify novel variant expressions semantically similar to known sensitive content through cosine similarity calculation, overcoming the limitation of the dictionary library not being able to cover unseen word variants and enhancing the system's ability to detect evasive expressions. The lightweight classifier module employs a multilayer perceptron (MLP) structure with a small parameter size. It can perform rapid initial screening with extremely low latency for the vast majority of clearly defined sensitive or normal text. Only boundary samples with confidence levels below a threshold θ are processed by the computationally expensive semantic retrieval and rule validation modules. This achieves on-demand allocation of computing resources, effectively balancing overall throughput and detection accuracy, and meeting the high-concurrency processing requirements of real-time dialogue scenarios. The system's core semantic capabilities originate from the general semantic representation obtained through large-scale unsupervised pre-training of the Lao pre-trained language model. The lightweight classifier only needs to perform supervised fine-tuning using a small number of labeled samples, significantly reducing dependence on large-scale Lao labeled corpora and effectively solving the problem of insufficient model generalization ability caused by the scarcity of training data in low-resource languages.

[0072] This specification also provides an embodiment of a low-resource language sensitive word filtering device, such as... Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0073] This specification also provides a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0074] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0075] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0076] The devices, media, and methods provided in the embodiments of this specification are one-to-one correspondences. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0077] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] The above are merely one or more embodiments of this specification and are not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for filtering sensitive words in low-resource languages, characterized in that, The method includes: The original Lao text is obtained, and the original Lao text is preprocessed to determine the normalized Lao character sequence. Multi-path word segmentation and disambiguation are then performed on the normalized Lao character sequence to generate multiple candidate word segments, wherein the candidate word segments include character-level sequences and at least one sub-word-level sequence. Using a pre-trained semantic coding model in Lao, the contextual semantic vector of each candidate word segmentation sequence is extracted to determine the calibration semantic vector corresponding to the original Lao text. Based on a pre-trained lightweight classifier and the calibration semantic vector, an initial verification result is determined, wherein the initial verification result includes an initial verification confidence score. When the initial verification confidence score is less than the preset confidence threshold, multi-dimensional deep verification is performed based on the calibration semantic vector and the normalized Lao character sequence to determine the deep verification result. The final verification result is determined based on the initial verification result and the deep verification result for sensitive word filtering. Multi-dimensional deep verification is performed based on the calibration semantic vector and the normalized Lao character sequence to determine the deep verification result, specifically including: Obtain a pre-built sensitive semantic vector library and a pre-set Lao grammar rule library, wherein the Lao grammar rule library includes rules for sensitive words of post-modifiers, rules for detecting conjunction structures, rules for collocation of function words and sensitive words, rules for mapping dialect words to standard words, and rules for normalizing Lao character variants; By using a preset approximate nearest neighbor retrieval, the cosine similarity between the calibrated semantic vector and each sensitive semantic vector in the sensitive semantic vector library is calculated, and the specified sensitive semantic vector with the highest cosine similarity is determined. Based on the cosine similarity and risk category corresponding to the specified sensitive semantic vector, determine the semantic retrieval similarity score and the first candidate risk category; Based on the Lao grammar rule base, rule matching is performed on the normalized Lao character sequence to determine the rule risk score; The second candidate risk category corresponding to the rule risk score is determined by using the rule risk score and the preset risk category threshold range; The semantic retrieval similarity score and the rule risk score are weighted and fused to determine the deep verification confidence score. The deep verification risk category is determined based on the higher-level risk category among the first candidate risk category and the second candidate risk category.

2. The method for filtering sensitive words in low-resource languages ​​according to claim 1, characterized in that, The normalized Lao character sequence is subjected to multi-path segmentation and disambiguation to generate multiple candidate segmentation sequences, specifically including: For each character in the normalized Lao character sequence, determine the character-level sequence; A byte-pair encoding model pre-trained on an unlabeled Lao corpus is obtained to perform sub-word segmentation on the normalized Lao character sequence and determine the first sub-word level sequence; At least one candidate sub-word sequence is generated using a pre-defined unsupervised word segmentation model and the standardized Lao character sequence. In the candidate sub-word sequence, sub-word sequences that are identical to the first sub-word sequence text are removed to determine the second sub-word sequence; The plurality of candidate word segments are determined using the character-level sequence, the first sub-word-level sequence, and the second sub-word-level sequence.

3. The method for filtering sensitive words in low-resource languages ​​according to claim 1, characterized in that, Using a pre-trained Lao semantic encoding model, the contextual semantic vector of each candidate word segmentation sequence is extracted to determine the calibration semantic vector corresponding to the original Lao text, specifically including: An initial embedding vector is determined for each unit in the candidate word segmentation sequence, and a classification label is inserted at the beginning of the candidate word segmentation sequence to obtain the learnable embedding vector corresponding to the classification label, which is then used as the first position embedding of the sequence. The entire embedding sequence containing the classification label embedding vector is input into the Transformer encoder stack of the semantic encoding model. During the multi-head self-attention calculation process of each Transformer encoder layer, the attention bias matrix is ​​dynamically calculated according to the preset Lao grammar rules. The attention bias matrix is ​​added to the self-attention score and then normalized using Softmax to determine the context semantic vector corresponding to the candidate word segmentation sequence. Attention fusion is performed on the context semantic vectors of the multiple candidate word segmentation sequences to determine the fused semantic vector; According to the preset calibration strategy, the fused semantic vector is calibrated to determine the calibrated semantic vector corresponding to the original Lao text.

4. The method for filtering sensitive words in low-resource languages ​​according to claim 3, characterized in that, According to a preset calibration strategy, the fused semantic vector is calibrated to determine the calibrated semantic vector corresponding to the original Lao text, specifically including: Obtain an unlabeled Lao general text set, extract semantic vectors from the Lao general text set, and determine the Lao general text vector set corresponding to the Lao general text set; Statistical analysis was performed on the distribution of text vectors in the Lao general text vector set to determine the mean vector and covariance matrix of the Lao general text vector distribution; The difference vector between the fused semantic vector and the mean vector is calculated, and the calibration semantic vector is determined by raising the difference vector and the covariance matrix to a preset power.

5. The method for filtering sensitive words in low-resource languages ​​according to claim 1, characterized in that, The initial verification result is determined using a preset lightweight classifier and the calibration semantic vector, specifically including: The calibration semantic vector is input into the lightweight classifier to output the initial risk category and initial confidence score; Based on the calibration semantic vector and the multiple candidate word segmentation sequences, the initial confidence score is corrected to determine the initial verification confidence score. The initial verification result is then determined using the initial verification confidence score and the initial risk category.

6. The method for filtering sensitive words in low-resource languages ​​according to claim 5, characterized in that, Based on the calibration semantic vector and the multiple candidate word segmentation sequences, the initial confidence score is corrected to determine the initial verification confidence score, specifically including: Obtain the training sample vector space corresponding to the lightweight classifier, and calculate the vector distance between each training sample vector and the calibration semantic vector in the training sample vector space; Based on the order of the vector distance from closest to furthest, determine a preset number of specified training sample vectors that are closest to each other, so as to obtain the category label of each specified training sample vector; The number of specified training samples with the same category label as the initial risk category in the preset number of specified training sample vectors is counted, and the nearest neighbor consistency ratio index is determined by the ratio of the number of specified training samples with the same category to the number of specified training sample vectors. Obtain the preliminary risk category corresponding to each candidate word segmentation sequence, so as to determine the sequence pair formed by any two candidate word segmentation sequences among the multiple candidate word segmentation sequences, and count the total number of sequence pairs corresponding to the multiple candidate word segmentation sequences; Compare whether the preliminary risk categories of the two candidate word segmentation sequences in each sequence pair are the same, so as to count the divergent sequence pairs with different preliminary risk categories and determine the number of divergent sequence pairs; The classification divergence coefficient is determined based on the ratio of the number of divergent sequence pairs to the total number of sequence pairs; The initial confidence score is determined by correcting the initial confidence score using the classification divergence coefficient and the nearest neighbor consistency ratio.

7. The method for filtering sensitive words in low-resource languages ​​according to claim 1, characterized in that, Based on the Lao grammar rule base, rule matching is performed on the normalized Lao character sequence to determine the rule risk score, specifically including: Obtain the basic risk weight corresponding to each grammar rule in the pre-set Lao grammar rule base, and determine the sum of the basic risk weights corresponding to the Lao grammar rule base; According to the Lao grammar rule library, the normalized Lao character sequence is matched according to the rules to determine the matching status of each grammar rule; When the matching status of the syntax rule is a successful match, the basic risk weight corresponding to the syntax rule is accumulated to determine the cumulative weight for a successful match; The rule risk score is determined by the ratio of the cumulative weight of successful matches to the sum of the basic risk weights.

8. A sensitive word filtering device for low-resource languages, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the method as described in any one of claims 1-7.

Images