A semantic defect identification method for a Chinese generation large model

Abstract

This invention relates to the field of natural language processing technology and discloses a semantic defect identification method for large-scale Chinese generation models. The method includes: acquiring the text to be tested; obtaining semantic perturbation versions under different perturbation types using a Chinese perturbation operator library and calculating the semantic inconsistency probability between these versions and the text; extracting word-level, sentence-level, and discourse-level semantic representations using a multi-granularity encoder; dynamically generating a gate mask based on the perturbation type; concatenating the semantic inconsistency probability and the gate mask into the word-level representation sequence; obtaining the location and contribution of defective words, the probability and type of sentence / segment defects, and the probability and type of discourse coherence defects through a cascaded multi-task defect detection framework; and finally, integrating the detection results from each level to calculate a comprehensive defect score. This invention designs a perturbation-guided adaptive feature modulation and a cross-granularity feedback closed-loop mechanism to achieve multi-level accurate identification and quantitative evaluation of semantic defects in large-scale Chinese generation models.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for identifying semantic defects in large Chinese generation models. Background Technology

[0002] In recent years, AI-based large-scale language generation models have achieved remarkable results in natural language generation and processing tasks. However, due to the unique characteristics of the Chinese language, such as the lack of explicit word boundaries, numerous homophones, high confusion between similar-looking characters, and flexible syntactic structures, and the fact that most existing semantic defect identification methods are designed for Indo-European languages ​​such as English, their perturbation strategies and defect type systems are based on the linguistic characteristics of English, making it difficult to effectively cover the unique semantic error patterns of Chinese.

[0003] Existing text semantic defect identification methods are mainly divided into two categories. One category is text quality assessment based on rules or statistical methods, which relies on predefined grammatical rules or statistical indicators such as language model perplexity. It has a certain ability to identify explicit grammatical errors, but it is difficult to capture deep semantic logical defects and discourse coherence issues. The other category is end-to-end defect detection based on neural networks, which conducts quality assessment of text at the word, sentence, or discourse level by constructing a multi-task learning framework. However, this type of method is usually designed as a unidirectional pipeline architecture, with information flow between each level of detection task in a single direction. Word-level detection results cannot influence the sentence-level and discourse-level evaluation processes in reverse.

[0004] Furthermore, existing methods have significant shortcomings in the integration of perturbation discrimination and defect detection. Perturbation testing is widely used to evaluate the robustness of models, but its discrimination results are usually not used to guide feature extraction or defect detection processes. In the evaluation results of multi-source assessment, existing methods cannot adaptively adjust the distribution weights of the evaluation results of each dimension according to the defect distribution characteristics of the text itself, thereby limiting the accuracy and discriminativeness of the final evaluation results. Summary of the Invention

[0005] The purpose of this invention is to propose a semantic defect identification method for large Chinese generative models in order to solve the problems in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a semantic defect identification method for large Chinese generation models, comprising the following steps:

[0007] S1. Obtain the test text generated by the Chinese generation model, obtain the semantic perturbation version of the test text under different perturbation types through the perturbation operator library, and obtain the semantic inconsistency probability between the test text and the semantic perturbation version of each perturbation type through the pre-trained semantic consistency discrimination model.

[0008] S2, through a pre-trained multi-granular encoder, extract multi-granular semantic representations of the text to be tested, including word-level representation sequences, sentence-level representation sequences, and document-level representations. Among them, a gate mask for word-level features is dynamically generated according to the perturbation type corresponding to the semantic inconsistency probability, and the semantic inconsistency probability and the gate mask are concatenated as additional features to the word-level representation sequence to form an enhanced word-level representation sequence.

[0009] S3, set up a cascaded multi-task defect detection framework on the output of the multi-granularity encoder, and perform cascaded detection with the multi-granularity semantic representation as input, including: obtaining defect word location and defect contribution through word-level defect detection task, combining the results of word-level defect detection, obtaining sentence segment defect probability and sentence-level defect type through sentence-level defect detection task, and combining the results of sentence-level defect detection, obtaining discourse coherence defect probability and discourse-level defect type through discourse-level defect detection task;

[0010] S4. Based on the semantic inconsistency probability, the defect contribution, the results of the sentence / segment defect detection, and the results of the discourse coherence defect detection, a comprehensive defect score is calculated.

[0011] The beneficial effects of the technical solution provided by this invention include at least the following:

[0012] This invention provides a semantic defect identification method for large-scale Chinese character generation models. By constructing a perturbation operator library and a semantic consistency discrimination model for Chinese, it can perceive the direction of semantic vulnerability in the text under test, thereby effectively detecting semantic defects of large-scale Chinese character generation models in the dimension of Chinese language characteristics. It significantly improves the ability to identify Chinese-specific semantic errors such as misuse of homophones, confusion of similar-looking characters, and improper placement of adverbs, and deeply couples the defect detection method with Chinese language characteristics, filling the gap in the applicability of existing methods in the field of Chinese semantic defect identification.

[0013] This invention achieves adaptive feature enhancement for perturbation perception by converting the perturbation discrimination result into a gating mask and injecting it into the word-level feature extraction process. It can adaptively adjust the attention level of each feature dimension according to the semantic vulnerability direction of the text to be tested, providing more targeted feature representation for subsequent multi-granularity defect detection, and significantly improving the recall rate and localization accuracy of defect detection.

[0014] This invention constructs a cross-granularity feedback closed loop of word-sentence-discourse-comprehensive scoring, breaking through the information flow limitations of the unidirectional cascaded detection architecture in existing technologies. At the word level, the sequence labeling model outputs the location of defective words and their contribution. At the sentence level, the contribution of defects is used as a weight to weight and aggregate the enhanced word-level representation before being superimposed on the sentence-level representation, thereby enhancing sentence-level detection with word-level information. At the discourse level, a discourse-level interference coefficient is generated based on the proportion of problematic sentences to correct the initial coherence defect probability, thus providing feedback from sentence-level information to discourse-level evaluation. At the comprehensive scoring level, the contribution of global defective word units is used to adjust the weights of the three probability terms in the comprehensive defect score. Through the above multi-level feedback mechanism, local fine-grained defect information can dynamically influence the global evaluation decision, resulting in higher accuracy and robustness compared to existing technologies. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a semantic defect identification method for large Chinese generation models proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0020] The following description, in conjunction with the accompanying drawings, details the specific scheme of the semantic defect identification method for large Chinese generative models provided by this invention.

[0021] Please see Figure 1The diagram illustrates a flowchart of a semantic defect identification method for large Chinese generation models according to an embodiment of the present invention. The method includes the following steps:

[0022] S1. Obtain the test text generated by the Chinese generation model. Obtain the semantic perturbation version of the test text under different perturbation types through the perturbation operator library. Obtain the semantic inconsistency probability between the test text and the semantic perturbation version of each perturbation type through the pre-trained semantic consistency discrimination model.

[0023] S2 extracts multi-granular semantic representations of the text to be tested through a pre-trained multi-granular encoder, including word-level representation sequences, sentence-level representation sequences, and document-level representations. Among them, a gate mask for word-level features is dynamically generated according to the perturbation type corresponding to the semantic inconsistency probability. The semantic inconsistency probability and the gate mask are concatenated as additional features to the word-level representation sequence to form an enhanced word-level representation sequence.

[0024] S3, set up a cascaded multi-task defect detection framework on the output of the multi-granular encoder, and perform cascaded detection with multi-granular semantic representation as input, including: obtaining defect word location and defect contribution through word-level defect detection task, combining the results of word-level defect detection, obtaining sentence segment defect probability and sentence-level defect type through sentence-level defect detection task, and combining the results of sentence-level defect detection, obtaining discourse coherence defect probability and discourse-level defect type through discourse-level defect detection task;

[0025] S4, based on the semantic inconsistency probability, defect contribution, the results of sentence and paragraph defect detection, and the results of discourse coherence defect detection, calculates the comprehensive defect score.

[0026] As one embodiment of the present invention, the perturbation operator library is a perturbation operator library for Chinese, and the types of perturbations that can be applied include homophonic substitution perturbation based on Chinese character pinyin, perturbation of similar-looking characters based on Chinese character structure, and word order rearrangement perturbation based on Chinese dependency syntax relations.

[0027] It should be noted that, in this embodiment, the perturbation operator library is a comprehensive perturbation operator library for Chinese. Those skilled in the art should understand that, in addition to including perturbation types specifically designed for the characteristics of the Chinese language, it also includes general perturbation types, including but not limited to:

[0028] Synonym substitution perturbation involves replacing keywords in the text with semantically similar synonyms.

[0029] Random deletion perturbation, that is, randomly deleting some words from the text with a preset probability;

[0030] Random perturbation involves inserting pre-defined distractor words or filler words at random positions in the text.

[0031] Word order disruption involves randomly swapping the positions of adjacent words in a text.

[0032] Back-translation perturbation involves translating the text into an intermediate language and then back-translating it back into the original language (i.e., Chinese), using the semantic drift generated during the back-translation process to construct a perturbed version.

[0033] Among the unique perturbation types designed for the characteristics of the Chinese language, the following are included:

[0034] The specific method of homophonic substitution perturbation based on Chinese Pinyin is as follows: identify the Pinyin sequence of each word element in the text to be tested, retrieve candidate Chinese characters with the same or similar Pinyin from the preset homophonic character mapping table, and replace the original Chinese characters with candidate Chinese characters with a preset substitution probability.

[0035] The perturbation based on the similarity of Chinese characters is as follows: the character structure of each Chinese character in the test text is analyzed, its radicals and strokes are extracted, and candidate Chinese characters with the same or similar radicals and stroke counts within a preset range are retrieved from the preset similarity mapping table. The original Chinese character is replaced with the candidate Chinese character with a preset replacement probability.

[0036] The word order reordering perturbation based on Chinese dependency syntax relations is as follows: Chinese dependency syntax analysis is performed on the test text to obtain the dependency relation tree between each word, the core predicate and its subordinate components in the dependency relation tree are identified, and the subordinate components of the core predicate are moved as a whole to the specific position of the core predicate while maintaining their internal order, so as to generate a word order reordering version that conforms to Chinese grammar but changes the focus structure of the original sentence.

[0037] Each perturbation type in the perturbation operator library acts independently on the text to be tested, and the replacement probability or rearrangement probability of each perturbation type is a configurable value between 0.1 and 0.3, so as to introduce semantic offsets while maintaining the basic readability of the text, for the semantic consistency discrimination model to judge.

[0038] In one embodiment of the present invention, the semantic consistency discrimination model is obtained through pre-training via the following steps:

[0039] Obtain the original text generated by the large Chinese generation model, and apply various types of semantic perturbations to the original text using a perturbation operator library for Chinese.

[0040] Positive sample pairs are constructed by pairing the original text with itself and labeling them with semantically consistent tags. Negative sample pairs are constructed by pairing the original text with the semantic perturbation versions corresponding to each perturbation type and labeling them with the corresponding perturbation type tags.

[0041] The positive and negative sample pairs are input into a Siamese network based on a Chinese text encoder, and the semantic vectors of the two paired text segments in each sample pair are extracted respectively.

[0042] The Siamese network is trained by comparing loss functions to calculate the semantic inconsistency probability between the two text segments in each sample pair. The Siamese network is trained by multi-classification loss functions to identify the perturbation type corresponding to the negative sample pair.

[0043] By jointly optimizing the comparative loss function and the multi-classification loss function until the Siamese network converges, a semantic consistency discrimination model is obtained.

[0044] It should be noted that in the aforementioned steps, during the pre-training process of the semantic consistency discrimination model, each perturbation type in the perturbation operator library is applied independently to the original text. Each perturbation type generates a corresponding semantic perturbation version. The semantic inconsistency probability is a continuous value between 0 and 1, which is used to characterize the semantic similarity between paired texts in the same sample pair. The larger the value, the more significant the semantic shift.

[0045] In this embodiment, the Siamese network based on the Chinese text encoder has two sub-networks that share the same network structure and parameters, which are used to map the input text into a semantic vector of fixed dimension. Positive sample pairs and negative samples are divided into multiple training batches and input into the Siamese network. The ratio of the number of positive sample pairs to negative sample pairs in each training batch is preferably 1:2 to 1:4.

[0046] The preferred contrastive loss function is constructed using InfoNCE loss or cosine similarity contrastive loss. By narrowing the semantic vector distance between two text segments in a positive sample pair and widening the semantic vector distance between two text segments in a negative sample pair, the Siamese network learns the ability to discriminate semantic consistency. The preferred multi-class loss function is constructed using cross-entropy loss. Based on the perturbation type label corresponding to the negative sample, the output layer of the Siamese network simultaneously possesses the ability to identify perturbation types.

[0047] During joint optimization, the total loss function is the weighted sum of the contrastive loss function and the multi-class loss function, where the ratio of the contrastive loss weight to the multi-class loss weight is preferably between 1:1 and 3:1. During training, each training batch contains both positive sample pairs and negative sample pairs corresponding to various perturbation types, enabling the Siamese network to maintain its sensitivity to distinguish perturbation type categories while optimizing semantic consistency discrimination ability. The convergence condition for training is that, on the validation set, the value of the contrastive loss function decreases by less than a preset threshold for five consecutive training epochs, and the classification accuracy of the multi-class loss function increases sufficiently for five consecutive training epochs, or the total number of training epochs reaches the preset maximum number of training epochs, at which point training is terminated.

[0048] In one embodiment of the present invention, the multi-granularity encoder is a pre-trained language model based on the Transformer architecture, and the step of extracting the multi-granularity semantic representation of the text to be tested includes:

[0049] Input the text to be tested into the pre-trained language model, obtain the hidden state sequence of its last layer output, and use the vector corresponding to each position in the hidden state sequence as the word-level representation of that position to form a word-level representation sequence.

[0050] Each natural sentence in the text to be tested is used as a unit to segment the word-level representation sequence. All word-level representations in each natural sentence are averaged and pooled to obtain the sentence-level representation corresponding to each natural sentence, thus forming a sentence-level representation sequence.

[0051] Global average pooling is performed on the sentence-level representation sequence to obtain the document-level representation of the text under test.

[0052] It should be noted that in this embodiment, the pre-trained language model based on the Transformer architecture adopts a Chinese pre-trained language model. Its input is the sequence of the text to be tested after word segmentation or character segmentation, and its output is a hidden state sequence of the same length as the input sequence. Each position in the hidden state sequence corresponds to a word in the input sequence, and the hidden state vector at that position is the word-level representation of that word.

[0053] The specific method for segmenting the word-level representation sequence by natural sentence is as follows: Natural sentence boundaries are identified based on punctuation marks in the test text. Using these natural sentence boundaries as dividing points, the word-level representation sequence is divided into subsequences corresponding to each natural sentence. For words that cross natural sentence boundaries due to model word segmentation, they are reassigned to the correct natural sentence sequence based on their actual character positions in the original text. During the average pooling of all word-level representations within each natural sentence, for natural sentences exceeding the preset maximum sentence length, the word-level representation sequence is truncated, retaining only the word-level representations of the first N words of the natural sentence for average pooling, where N is the maximum input length limit of the pre-trained language model. For empty natural sentences with zero length, the zero vector is used as the sentence-level representation.

[0054] The sentence-level representations of all natural sentences in the test text are calculated as an arithmetic mean on the sentence dimension to obtain a fixed-dimensional document-level representation. If the number of natural sentences in the test text is 1, the document-level representation is the same as the sentence-level representation of that natural sentence.

[0055] As one embodiment of the present invention, the step of concatenating the semantic inconsistency probability as an additional feature with the word-level representation sequence in the multi-granularity semantic representation includes:

[0056] Extract the perturbation type with the highest semantic inconsistency probability in step S1 and use it as the target perturbation type;

[0057] Based on the target perturbation type, the corresponding gate parameter vector is retrieved from the preset gate parameter matrix. The gate parameter matrix records the modulation weights between each perturbation type and each dimension of the word-level feature.

[0058] The gating parameter vector is multiplied element-wise with the feature vector at each position in the word-level representation sequence to generate a dynamic gating mask;

[0059] The semantic inconsistency probabilities corresponding to each perturbation type are weighted and averaged based on preset weights to obtain a fusion value of semantic inconsistency probabilities. The fusion value is then expanded to the same length as the word-level representation sequence to obtain a word-level extended vector.

[0060] The joint additional features are obtained by concatenating the dynamic gating mask and the word-level expansion vector along the feature dimension. The joint additional features are then concatenated with the word-level representation sequence along the feature dimension to obtain the enhanced word-level representation sequence.

[0061] It should be noted that in this embodiment, the gating parameter matrix is ​​a two-dimensional parameter matrix pre-built and stored in the system database. The number of rows is the total number of perturbation types. Each row corresponds to the gating parameter vector under the corresponding perturbation type. The number of columns is the same as the dimension of the feature vector at a single position in the word-level representation sequence. Each element in the gating parameter vector is a configurable value between 0 and 1, which is used to represent the importance of that dimension of the word-level feature in defect detection under the corresponding perturbation type.

[0062] For each position in the word-level representation sequence, extract the feature vector for that position from the word-level representation sequence, and multiply the gating parameter vector element-wise with the feature vector for that position to generate the gating mask vector for that position. The gating mask vector has the same dimension as the feature vector for that position in the word-level representation sequence. Perform the above operation on all positions to obtain a gating mask sequence with the same length as the word-level representation sequence and with the vector dimensions of each position unchanged, which serves as the dynamic gating mask.

[0063] In this embodiment, the preset weights are the prior weights corresponding to each perturbation type, which are configurable values ​​between 0 and 1, and the sum of the prior weights of all perturbation types is 1. The fusion value of the semantic inconsistency probability is the sum of the products of the semantic inconsistency probability corresponding to each perturbation type and the corresponding prior weight.

[0064] For the i-th position in the sequence, the gating mask vector corresponding to that position and one component (whose value is the fusion value) of the word-level expansion vector corresponding to that position are concatenated end to end along the feature dimension to obtain the joint additional feature vector for that position. The dimension of the joint additional feature vector is equal to the dimension of the gating mask vector plus 1. The above operation is performed on all positions to obtain a joint additional feature sequence with the same length as the word-level representation sequence and the dimension of each position vector being the dimension of the word-level representation sequence plus 1.

[0065] The vector at position i in the joint additional feature sequence is concatenated with the feature vector at position i in the word-level representation sequence along the feature dimension to obtain the enhanced word-level representation vector at that position.

[0066] In one embodiment of the present invention, the step of outputting defect word location and defect contribution through a word-level defect detection task includes:

[0067] The enhanced word-level representation sequence is input into the word-level defect detection task. The word-level defect detection task is based on the sequence labeling model. For each position in the enhanced word-level representation sequence, the probability value of the corresponding position belonging to the defective word is output after mapping by the Softmax function. Positions with probability values ​​higher than the preset defect probability threshold are labeled as defective words.

[0068] The positions marked as defective words are selected from the word-level representation sequence, and multiple consecutive positions marked as defective words are merged into a defective word unit. The defective word unit is further marked as to whether each defective word in the defective word unit is located at the starting position. Each defective word is itself as a defective word unit.

[0069] Calculate the probability value of each position within each defective word unit belonging to a defective word, take the average value as the defect contribution of each defective word unit, and output the defective word location and defect contribution corresponding to each defective word unit.

[0070] It should be noted that, in this embodiment, the sequence labeling model preferably adopts a sequence labeling architecture based on Conditional Random Field (CRF) or Bidirectional Long Short-Term Memory Network (BiLSTM). After performing a linear transformation on the hidden state vector at each position in the enhanced word-level representation sequence, the probability value of that position belonging to the defective word category is calculated by the Softmax function.

[0071] The labeling of defective word categories preferably adopts the BIO labeling system, where B represents the starting position of the defective word, I represents the internal position of the defective word, and O represents a non-defective word. Specifically, it includes: a preset defect probability threshold preferably set to 0.5~0.8, labeling all positions with a probability value higher than the defect probability threshold as defective words, and merging consecutive and adjacent defective word positions in the labeling results into one defective word unit. When merging, the B-labeled position is used as the starting boundary of the defective word unit, and the ending position of consecutive I-labeled positions is used as the ending boundary of the defective word unit. If there is only one non-defective word position between two defective word units and the word corresponding to that position is a punctuation mark, then the interval position is ignored and the two defective word units are merged into one defective word unit.

[0072] In this embodiment, the probability values ​​at each position are first smoothed, and the probability values ​​below the preset lower threshold are raised to the lower threshold. Then, the mean of the probability values ​​of each position in each defective word unit belonging to the defective word is calculated. The lower threshold is preferably set to 0.3~0.5. The calculated defect contribution is a continuous value between 0 and 1, which is used to characterize the semantic abnormality severity of the defective word unit in the entire text to be tested.

[0073] In one embodiment of the present invention, the step of outputting the sentence segment defect probability and sentence-level defect type through a sentence-level defect detection task, based on the results of word-level defect detection, includes:

[0074] Based on the location of defective words and their contribution to the overall analysis, the natural sentences containing each defective word unit are determined, including:

[0075] For a natural sentence containing at least one defective word unit, based on the defect contribution of each defective word unit in the natural sentence, the defect contribution is multiplied by the enhanced word-level representation at the corresponding position and then superimposed onto the sentence-level representation to obtain the enhanced sentence-level representation. The enhanced sentence-level representation is then input into the sentence-level defect detection task, and through dimensionality reduction of the fully connected layer and mapping of the Sigmoid function, the sentence segment defect probability and the corresponding sentence-level defect type are output.

[0076] For natural sentences that do not contain defective word units, the corresponding sentence-level representation is directly input into the sentence-level defect detection task for classification, and the sentence segment defect probability and the corresponding sentence-level defect type are output.

[0077] It should be noted that, in this embodiment, the sentence-level defect detection task preferably adopts a classification network architecture based on multilayer perceptron (MLP) or multi-head self-attention pooling, whose input dimension is consistent with the dimension of sentence-level representation, and whose output dimension is the number of sentence-level defect type categories plus 1 (where 1 corresponds to the no-defect category).

[0078] For a natural sentence containing at least one defective word unit, for the i-th word position in the natural sentence, if the position belongs to the j-th defective word unit, the enhanced word-level representation vector of the position is multiplied by the defect contribution value of the defective word unit; if the position does not belong to any defective word unit, it is multiplied by a preset baseline weight (preferably 0~0.3). Then, the weighted enhanced word-level representations of all word positions are summed to obtain a defect-weighted aggregated vector. Finally, the defect-weighted aggregated vector is added to the sentence-level representation to obtain the enhanced sentence-level representation.

[0079] The obtained enhanced sentence-level representation is successively reduced to half the original dimension by the first fully connected layer, then by the ReLU activation function, and then by the second fully connected layer to reduce the dimensionality to the number of sentence-level defect type categories. Subsequently, it is mapped to the independent probability value of each sentence-level defect type by the Sigmoid function. The maximum value among the independent probability values ​​of each sentence-level defect type is taken as the sentence defect probability, and the corresponding defect type label is taken.

[0080] If all sentence-level defect type probability values ​​are lower than the preset sentence-level defect identification threshold, the natural sentence is determined to be a defect-free sentence. The maximum value among the independent probability values ​​of each sentence-level defect type is still taken as the sentence segment defect probability, and the sentence-level defect type is marked as defect-free. The sentence-level defect identification threshold is preferably set to 0.3~0.5.

[0081] For natural sentences that do not contain defective word units, the corresponding sentence-level representation is directly input into the classification network for classification, and the sentence segment defect probability and the corresponding sentence-level defect type are output. The corresponding sentence-level defect types include, but are not limited to, semantic repetition, logical contradiction, unclear reference, improper collocation, incomplete components, and disordered word order (as well as no defects).

[0082] In one embodiment of the present invention, the step of outputting the probability of a passage coherence defect and the type of a passage-level defect through a passage-level defect detection task, based on the results of sentence-level defect detection, includes:

[0083] Obtain the sentence defect probability of each natural sentence, mark the natural sentences with a sentence defect probability higher than the preset sentence-level defect threshold as problematic sentences, calculate the proportion of problematic sentences based on the number of all natural sentences and the number of problematic sentences in the text to be tested, and multiply the proportion of problematic sentences by the preset interference item weight to obtain the text-level interference coefficient;

[0084] Input the chapter-level representation into the chapter-level defect detection task based on global pooling, and output the initial coherence defect probability and the corresponding chapter-level defect type;

[0085] The initial coherence defect probability is multiplied by the document-level interference coefficient to obtain the corrected document coherence defect probability.

[0086] It should be noted that, in this embodiment, global pooling is preferably performed by max pooling and average pooling sequentially on the feature dimension of the document-level representation. The max pooling result and the average pooling result are concatenated as a vector to serve as the document-level global feature. Then, it is mapped to the dimension of the document-level defect type category number plus 1 (where 1 corresponds to the no-defect category) through a fully connected layer. Finally, the initial coherence defect probability and the corresponding document-level defect type are output through the Softmax function.

[0087] The preset sentence-level defect threshold is a configurable value between 0.5 and 0.7, used to distinguish between normal sentences and problem sentences. The proportion of problem sentences is the ratio of the number of problem sentences to the total number of all natural sentences in the text to be tested, and its value ranges from 0 to 1. The preset interference item weight is a positive number greater than 1, preferably a configurable value between 1.2 and 2.0. The document-level interference coefficient is the product of the proportion of problem sentences and the interference item weight, and its value ranges from 0 to 2.0.

[0088] If the passage-level interference coefficient is greater than 1, the initial coherence defect probability is multiplied by the passage-level interference coefficient, and then the product result is restricted to between 0 and 1 by the pruning function (i.e., if the input value is greater than 1, the output is 1; if the input value is less than 0, the output is 0; otherwise, the output is the input value itself) to obtain the corrected passage coherence defect probability.

[0089] If the passage-level interference coefficient is ≤1, then the initial coherence defect probability is directly multiplied by the passage-level interference coefficient to obtain the corrected passage coherence defect probability.

[0090] Chapter-level defects include, but are not limited to, topic drift, causal break, reference chain break, and logical jump.

[0091] In one embodiment of the present invention, the step of fusing the semantic inconsistency probability, the results of sentence segment defect detection, and the results of discourse coherence defect detection to calculate a comprehensive defect score includes:

[0092] Extract the perturbation type with the highest semantic inconsistency probability in step S1 as the target perturbation type, and obtain the sentence-level defect type and discourse coherence defect type of each natural sentence.

[0093] Query the preset semantic association matrix to obtain the first and second matching degrees between the target perturbation type and the sentence-level defect type and the document-level defect type, respectively;

[0094] The defect contribution of all defective word units in the test text is globally normalized to obtain the global defect severity coefficient γ. The comprehensive defect score of the test text is then calculated using the following formula:

[0095] Score = P sem ·(M1 + λ1γ) + Psent ·(αM2 + M1 + λ2γ) + P doc ·(βM1 + M2 + λ3γ);

[0096] Among them, P sem Let P be the probability of semantic inconsistency. sent P is the global pooling value of the probability of a sentence defect. doc M1 represents the probability of a coherence defect, M2 represents the first degree of matching, α and β represent the preset sentence-level and document-level basic weights, respectively, and λ1, λ2, and λ3 represent the preset adjustment coefficients of the global defect severity coefficient on the three probability terms.

[0097] It should be noted that in this embodiment, the semantic association matrix is ​​a pre-constructed two-dimensional lookup table, which is stored in the system database before the method is executed and is called in read-only mode during the method execution.

[0098] The first dimension of the semantic association matrix is ​​each perturbation type, and the second dimension is each sentence-level defect type and each chapter-level defect type. Each element in the semantic association matrix is ​​the semantic similarity between the corresponding perturbation type and the corresponding defect type. The semantic similarity is preferably calculated based on the context co-occurrence statistics of perturbation type keywords and defect type keywords in a large-scale Chinese corpus, or based on the synonym relationship and hyponymy relationship between words in a Chinese semantic dictionary.

[0099] The first and second matching degrees both range from 0 to 1. The larger the value, the closer the semantic association between the target perturbation type and the corresponding defect type. When querying the semantic association matrix, if the target perturbation type, sentence-level defect type, or document-level defect type does not exist in the index of the semantic association matrix, the preset default value is used as the corresponding matching degree. The default value is preferably between 0.3 and 0.5.

[0100] In this embodiment, the defect contribution of all defective word units in the text to be tested is obtained to form a contribution sequence. Preferably, the contribution sequence is linearly transformed by the MIN-MAX normalization method so that the value of the transformed global defect severity coefficient γ is between 0 and 1. If there are no defective word units in the text to be tested, the global defect severity coefficient γ is 0.

[0101] The global pooling value P of the probability of segment defects sent The specific calculation method is as follows: obtain the sentence defect probability of each natural sentence in the text to be tested, construct the corresponding sentence-level probability sequence, and perform one of the following operations on the sentence-level probability sequence to obtain P. sent: Take the maximum value in the sentence-level probability sequence, or take the arithmetic mean of the sentence-level probability sequence, or take the arithmetic mean of the first K values ​​after arranging the sentence-level probability sequence in descending order (where K is a preset positive integer and K does not exceed the length of the sentence-level probability sequence).

[0102] The preset sentence-level basic weight α and the preset discourse-level basic weight β are both configurable values ​​between 0 and 1, and satisfy α+β= 1. The preset adjustment coefficients λ1, λ2, and λ3 are also configurable values ​​between 0 and 1, which are used to control the adjustment intensity of the global defect severity coefficient γ on the semantic inconsistency probability term, the sentence / segment defect probability term, and the discourse coherence defect probability term, respectively.

[0103] The overall defect score ranges from 0 to ∞. The larger the value, the more serious the semantic defect of the text under test. Preferably, when the overall defect score is ≥2.0, the text under test can be considered to have significant semantic defects.

[0104] As one embodiment of the present invention, the step of querying a preset semantic association matrix to obtain a first matching degree and a second matching degree between the target perturbation type and the sentence-level defect type and the document-level defect type includes:

[0105] Obtain a preset semantic association matrix of defect types. The semantic association matrix of defect types records the first semantic similarity between each perturbation type and each sentence-level defect type, and the second semantic similarity between each perturbation type and each document-level defect type.

[0106] Based on the target perturbation type and sentence-level defect type, query the corresponding first semantic similarity in the semantic association matrix as the first matching degree;

[0107] Based on the target perturbation type and the document-level defect type, query the corresponding second semantic similarity in the semantic association matrix as the second matching degree.

[0108] It should be noted that, in this embodiment, the semantic association matrix of defect types is a two-dimensional lookup table pre-built and stored in the system database. Its row index is each perturbation type, and its column index is the union of all sentence-level defect types and all chapter-level defect types. The value of each element in the matrix is ​​the semantic similarity between the corresponding perturbation type and the corresponding defect type, and the value range is between 0 and 1, where 0 indicates no semantic association and 1 indicates that the semantics are completely the same.

[0109] The semantic association matrix is ​​accessed in read-only mode, and its content remains unchanged during the defect identification process of a single test text. Those skilled in the art should know that when the method of the present invention is applied to large Chinese generation models in different fields, the semantic association matrix is ​​reconstructed or incrementally updated according to the special corpus of that field to adapt to the semantic association distribution characteristics of different fields.

[0110] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A semantic defect identification method for large-scale Chinese generative models, characterized in that, The method includes: S1. Obtain the test text generated by the Chinese generation model, obtain the semantic perturbation version of the test text under different perturbation types through the perturbation operator library, and obtain the semantic inconsistency probability between the test text and the semantic perturbation version of each perturbation type through the pre-trained semantic consistency discrimination model. S2, through a pre-trained multi-granular encoder, extract multi-granular semantic representations of the text to be tested, including word-level representation sequences, sentence-level representation sequences, and document-level representations. Among them, a gate mask for word-level features is dynamically generated according to the perturbation type corresponding to the semantic inconsistency probability, and the semantic inconsistency probability and the gate mask are concatenated as additional features to the word-level representation sequence to form an enhanced word-level representation sequence. S3, set up a cascaded multi-task defect detection framework on the output of the multi-granularity encoder, and perform cascaded detection with the multi-granularity semantic representation as input, including: obtaining defect word location and defect contribution through word-level defect detection task, combining the results of word-level defect detection, obtaining sentence segment defect probability and sentence-level defect type through sentence-level defect detection task, and combining the results of sentence-level defect detection, obtaining discourse coherence defect probability and discourse-level defect type through discourse-level defect detection task; S4. Based on the semantic inconsistency probability, the defect contribution, the results of the sentence / segment defect detection, and the results of the discourse coherence defect detection, a comprehensive defect score is calculated.

2. The semantic defect identification method for large-scale Chinese generation models according to claim 1, characterized in that: The perturbation operator library mentioned above is a perturbation operator library for Chinese, and the types of perturbations that can be applied include homophonic substitution perturbation based on Chinese character pinyin, perturbation of similar-looking characters based on Chinese character structure, and word order rearrangement perturbation based on Chinese dependency syntax relations.

3. The semantic defect identification method for large-scale Chinese generation models according to claim 2, characterized in that: The semantic consistency discrimination model is obtained through pre-training via the following steps: Obtain the original text generated by the Chinese generation model, and apply various types of semantic perturbations to the original text using a perturbation operator library for Chinese. The original text is paired with itself to construct positive sample pairs and labeled with semantically consistent tags. The original text is paired with the semantic perturbation versions corresponding to each perturbation type to construct negative sample pairs and labeled with the corresponding perturbation type tags. The positive and negative sample pairs are input into a Siamese network based on a Chinese text encoder to extract the semantic vectors of the two paired text segments in each sample pair. The Siamese network is trained by comparing loss functions to calculate the semantic inconsistency probability between two text segments in each sample pair, and the Siamese network is trained by multi-classification loss functions to identify the perturbation type corresponding to the negative sample pair. The contrastive loss function and the multi-classification loss function are jointly optimized until the Siamese network converges, thus obtaining the semantic consistency discrimination model.

4. The semantic defect identification method for large-scale Chinese generation models according to claim 1, characterized in that: The multi-granularity encoder is a pre-trained language model based on the Transformer architecture, and the step of extracting the multi-granularity semantic representation of the text to be tested includes: The text to be tested is input into the pre-trained language model to obtain the hidden state sequence of its last layer output. The vector corresponding to each position in the hidden state sequence is used as the word-level representation of that position to form a word-level representation sequence. Taking each natural sentence in the text to be tested as a unit, the word-level representation sequence is segmented, and all word-level representations in each natural sentence are averaged and pooled to obtain the sentence-level representation corresponding to each natural sentence, thus forming a sentence-level representation sequence. Global average pooling is performed on the sentence-level representation sequence to obtain the document-level representation of the text to be tested.

5. The semantic defect identification method for large Chinese generation models according to claim 4, characterized in that: The step of concatenating the semantic inconsistency probability as an additional feature with the word-level representation sequence in the multi-granularity semantic representation includes: Extract the perturbation type with the highest semantic inconsistency probability in step S1 and use it as the target perturbation type; According to the target perturbation type, the corresponding gate parameter vector is retrieved from the preset gate parameter matrix, which records the modulation weights between each perturbation type and each dimension of the word-level feature; The gating parameter vector is multiplied element-wise with the feature vector at each position in the word-level representation sequence to generate a dynamic gating mask; The semantic inconsistency probabilities corresponding to each perturbation type are weighted and averaged based on preset weights to obtain a fusion value of semantic inconsistency probabilities. The fusion value is then expanded to the same length as the word-level representation sequence to obtain a word-level extended vector. The dynamic gating mask and the word-level expansion vector are concatenated along the feature dimension to obtain a joint additional feature. The joint additional feature is then concatenated along the feature dimension with the word-level representation sequence to obtain an enhanced word-level representation sequence.

6. The semantic defect identification method for large-scale Chinese generation models according to claim 1, characterized in that: The steps described above, which output defect word location and defect contribution through the word-level defect detection task, include: The enhanced word-level representation sequence is input into the word-level defect detection task. The word-level defect detection task is based on a sequence labeling model. For each position in the enhanced word-level representation sequence, the probability value of the corresponding position belonging to the defective word is output after mapping by the Softmax function. Positions with the probability value higher than the preset defect probability threshold are labeled as defective words. The positions marked as defective words are selected from the word-level representation sequence, and multiple consecutive positions marked as defective words are merged into a defective word unit. The defective word unit is further marked as to whether each defective word in the defective word unit is located at the starting position. Each defective word is itself as a defective word unit. Calculate the probability value of each position within each defective word unit belonging to a defective word, take the average value as the defect contribution of each defective word unit, and output the defective word location and defect contribution corresponding to each defective word unit.

7. The semantic defect identification method for large Chinese generation models according to claim 6, characterized in that: The steps of combining the results of word-level defect detection and outputting the sentence defect probability and sentence-level defect type through a sentence-level defect detection task include: Based on the location of the defective words and their contribution, the natural sentences containing each defective word unit are determined, wherein: For a natural sentence containing at least one defective word unit, based on the defect contribution of each defective word unit in the natural sentence, the defect contribution is multiplied by the enhanced word-level representation at the corresponding position and then superimposed onto the sentence-level representation to obtain the enhanced sentence-level representation. The enhanced sentence-level representation is then input into the sentence-level defect detection task, and through dimensionality reduction of the fully connected layer and mapping of the Sigmoid function, the sentence segment defect probability and the corresponding sentence-level defect type are output. For natural sentences that do not contain defective word units, the corresponding sentence-level representation is directly input into the sentence-level defect detection task for classification, and the sentence segment defect probability and the corresponding sentence-level defect type are output.

8. The semantic defect identification method for large Chinese generative models according to claim 7, characterized in that: The steps involved in combining sentence-level defect detection results to output the probability of document coherence defects and the type of document-level defects through a document-level defect detection task include: Obtain the sentence defect probability of each natural sentence, mark the natural sentences with a sentence defect probability higher than the preset sentence-level defect threshold as problematic sentences, calculate the proportion of problematic sentences based on the number of all natural sentences and the number of problematic sentences in the text to be tested, and multiply the proportion of problematic sentences by the preset interference item weight to obtain the text-level interference coefficient; The document-level representation is input into a document-level defect detection task based on global pooling, and the initial coherence defect probability and the corresponding document-level defect type are output. The initial coherence defect probability is multiplied by the document-level interference coefficient to obtain the corrected document coherence defect probability.

9. The semantic defect identification method for large-scale Chinese generation models according to claim 1, characterized in that: The step of fusing the semantic inconsistency probability, the results of the sentence segment defect detection, and the results of the discourse coherence defect detection to calculate the comprehensive defect score includes: Extract the perturbation type with the highest semantic inconsistency probability in step S1 as the target perturbation type, and obtain the sentence-level defect type and discourse coherence defect type of each natural sentence. Query the preset semantic association matrix to obtain the first and second matching degrees between the target perturbation type and the sentence-level defect type and the document-level defect type, respectively; The defect contribution of all defective word units in the test text is globally normalized to obtain the global defect severity coefficient γ. The comprehensive defect score of the test text is then calculated using the following formula: Score = P sem ·(M1 + λ1γ) + P sent ·(αM2 + M1 + λ2γ) + P doc ·(βM1 + M2 + λ3γ); Among them, P sem Let P be the semantic inconsistency probability. sent P is the global pooling value of the probability of the sentence segment defect. doc M1 is the probability of the coherence defect, M2 is the first matching degree, α and β are the preset sentence-level basic weight and the document-level basic weight, respectively, and λ1, λ2 and λ3 are the preset adjustment coefficients of the global defect severity coefficient on the three probability terms.

10. The semantic defect identification method for large-scale Chinese generative models according to claim 9, characterized in that: The steps of querying the preset semantic association matrix to obtain the first and second matching degrees between the target perturbation type and the sentence-level defect type and the document-level defect type respectively include: Obtain a preset semantic association matrix of defect types, wherein the semantic association matrix of defect types records the first semantic similarity between each perturbation type and each sentence-level defect type, and the second semantic similarity between each perturbation type and each chapter-level defect type; Based on the target perturbation type and the sentence-level defect type, query the corresponding first semantic similarity in the semantic association matrix, and use it as the first matching degree; Based on the target perturbation type and the chapter-level defect type, the corresponding second semantic similarity in the semantic association matrix is ​​queried and used as the second matching degree.

Images