Corpus data cleaning and quality inspection method and device, storage medium and program product

Abstract

The invention provides a cleaning and quality inspection method and device for corpus data, a storage medium and a program product. The method comprises the steps of obtaining original corpus data; performing data preprocessing on the original corpus data to obtain first corpus data; performing noise filtering on the first corpus data to obtain second corpus data; performing semantic consistency check on the second corpus data to obtain third corpus data; performing grammar error correction on the third corpus data to obtain fourth corpus data; and the fourth corpus data is audited by adopting a multi-level auditing mechanism, existing suspected problem corpora are labeled and fed back, and a feedback result is continuously supervised and evaluated, so that the accuracy of labeled data is improved. Through the systematic cleaning step, noise and errors in the corpus data are effectively reduced, the quality of the corpus data is remarkably improved, the frequency of manual intervention is greatly reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and program product for cleaning and quality inspection of corpus data. Background Art

[0002] In the training process of natural language processing and machine learning models, data quality is a key factor in determining model performance. High-quality training data can significantly improve the prediction accuracy and generalization ability of the model, while poor-quality data will lead to poor model performance and even produce wrong prediction results. However, the actual collected corpus is usually full of noise, errors and inconsistencies, which not only brings great challenges to model training, but also reduces the effectiveness of the model in practical applications. At present, corpus cleaning mainly relies on manual intervention, which is not only time-consuming and labor-intensive, but also difficult to ensure the consistency and accuracy of the cleaning results. Summary of the invention

[0003] The present invention provides a method, device, storage medium and program product for cleaning and quality inspection of corpus data. Through systematic cleaning steps, the noise and errors in the corpus data are effectively reduced, and the quality of the corpus data is significantly improved, thereby improving the training effect of the machine learning model and the performance of practical applications.

[0004] In order to achieve the above purpose and other advantages, the present invention adopts the following technical solutions:

[0005] In a first aspect, the present invention provides a method for cleaning and quality checking corpus data, comprising:

[0006] Get the original corpus data;

[0007] Performing data preprocessing on the original corpus data to obtain first corpus data;

[0008] Performing noise filtering on the first corpus data to obtain second corpus data;

[0009] Performing a semantic consistency check on the second corpus data to obtain third corpus data;

[0010] Correcting grammatical errors of the third corpus data to obtain fourth corpus data;

[0011] The fourth corpus data is audited using a multi-level audit mechanism, suspected problematic corpora are annotated and feedback is provided, and the feedback results are continuously monitored and evaluated to improve the accuracy of the annotated data.

[0012] According to a method for cleaning and quality checking corpus data provided by the present invention, the step of performing data preprocessing on the original corpus data to obtain first corpus data includes:

[0013] Redundant information in the original corpus data is removed, and after formatting through text standardization, word segmentation is performed to segment the text to obtain the first corpus data.

[0014] According to a method for cleaning and quality checking corpus data provided by the present invention, the step of performing noise filtering on the first corpus data to obtain the second corpus data comprises:

[0015] Acquire multimodal features, context relevance, information entropy, and temporal characteristics of the first corpus data;

[0016] Perform a comprehensive noise score calculation based on the multimodal features, context relevance, information entropy, and temporal characteristics of the first corpus data to obtain a noise value;

[0017] An adaptive dynamic threshold is set according to the noise value, and when the noise value is greater than the dynamic threshold, it is marked as a noise data point and filtered;

[0018] A confidence assessment is provided for a noise filtering decision based on the noise value and the dynamic threshold.

[0019] According to a method for cleaning and quality checking corpus data provided by the present invention, the step of performing semantic consistency check on the second corpus data to obtain third corpus data includes:

[0020] Obtaining semantic vectors of multiple texts in the second corpus data;

[0021] For two texts constituting a text pair, respectively calculating the similarity, consistency evaluation score and logical reasoning result between the two texts based on the semantic vector;

[0022] Based on the similarity, consistency evaluation score and logical reasoning result between the two texts, the degree of contradiction between the two texts is quantified, and the contradiction detection value obtained by quantification is compared with a preset detection threshold. When the contradiction detection value is greater than the detection threshold, it is determined that there is a contradiction between the two texts;

[0023] Based on the contradiction detection value, a graph optimization algorithm is used to optimize the global semantic consistency of a set of different text pairs.

[0024] According to a method for cleaning and quality checking corpus data provided by the present invention, the step of correcting grammatical errors in the third corpus data to obtain fourth corpus data includes:

[0025] Obtaining a context encoding vector of the third corpus data;

[0026] Based on the context encoding vector, a graph neural network is used to model the grammatical structure, and the grammatical structure is encoded into a graph structure;

[0027] Based on the graph structure, creating a reinforcement learning driven error correction strategy to automatically identify and correct grammatical errors in the text;

[0028] A multi-head attention mechanism is used to evaluate the relevance of the corrected words and the context in the corrected text;

[0029] Based on the error detection probability of the graph neural network, the error detection probability of the multi-head attention mechanism, and the error detection probability of the language model, the corrected text is detected using a comprehensive error detection mechanism to obtain an error detection score for each original word in the text;

[0030] For the original word whose error detection score exceeds a preset value, evaluation scores of multiple candidate correction words are generated and sorted, and the best candidate correction word is selected for grammatical correction.

[0031] According to a method for cleaning and quality inspection of corpus data provided by the present invention, the fourth corpus data is audited using a multi-level audit mechanism, suspected problematic corpus is annotated and fed back, and the feedback results are continuously supervised and evaluated, including:

[0032] In the primary review, the suspected problematic corpus initially marked by the automated quality inspection algorithm is quickly screened, and the accuracy of the primary review is quantitatively evaluated;

[0033] In advanced review, for complex problem corpora or those that are difficult for the automated quality inspection algorithm to identify, contextual prompts and analysis suggestions provided by natural language processing technology are used to annotate the problem corpora and quantitatively evaluate the matching degree of the advanced review;

[0034] The annotated question corpus is input as optimized training data into the automated quality inspection algorithm for retraining, parameters of the automated quality inspection algorithm are optimized, and the recognition accuracy of the updated automated quality inspection algorithm is quantitatively evaluated.

[0035] According to a method for cleaning and quality inspection of corpus data provided by the present invention, the fourth corpus data is audited using a multi-level audit mechanism, suspected problematic corpus is annotated and fed back, and the feedback results are continuously supervised and evaluated, and the steps also include:

[0036] Introducing distributed artificial intelligence technology, multiple reviewers conduct independent review of the same corpus, and using a weighted voting mechanism to aggregate the review results of each reviewer;

[0037] Use blockchain technology to record the process of each review and annotation, and measure the supervision effect of the review process through the statistics of consistency between reviewers;

[0038] Through continuous supervision and evaluation, the review standards and processes are dynamically adjusted to adapt to the quality inspection task requirements of different types of corpus data.

[0039] In a second aspect, the present invention provides an electronic device, the electronic device comprising:

[0040] One or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed by the processor, implement the steps of any of the above-mentioned methods for cleaning and quality checking corpus data.

[0041] In a third aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for cleaning and quality inspection of corpus data.

[0042] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-described methods for cleaning and quality checking corpus data.

[0043] The present invention provides a method, device, storage medium and program product for cleaning and quality inspection of corpus data, which obtains original corpus data; performs data preprocessing on the original corpus data to obtain first corpus data; performs noise filtering on the first corpus data to obtain second corpus data; performs semantic consistency check on the second corpus data to obtain third corpus data; performs grammatical error correction on the third corpus data to obtain fourth corpus data; and audits the fourth corpus data using a multi-level audit mechanism, annotates and provides feedback on the existing suspected problem corpus, and continuously supervises and evaluates the feedback results to improve the accuracy of the annotated data. The present invention effectively reduces the noise and errors in the corpus data through systematic cleaning steps, significantly improves the quality of the corpus data, and greatly reduces the frequency of manual intervention, thereby improving the efficiency of data processing. High-quality corpus data can significantly improve the performance of the model in various natural language processing tasks, thereby improving the training effect of the machine learning model and the performance of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 It is a flowchart of a method for cleaning and quality inspection of corpus data provided by an embodiment of the present invention;

[0046] Figure 2 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows.

[0048] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present invention should be the usual meanings understood by people with ordinary skills in the technical field to which the present invention belongs. The words "one", "a", "a", "the" and the like involved in the present invention do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in the present invention are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in the present invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0049] When it comes to method steps, the sequence illustrated in this document represents an exemplary solution, but does not mean a limitation on the sequence.

[0050] Reference Figure 1 As shown, the present invention provides a method for cleaning and quality inspection of corpus data, comprising:

[0051] Step S1: Obtain original corpus data.

[0052] In this embodiment, the original corpus data may be derived from the user's corpus, or the corresponding original corpus data may be collected through the network. The source of the original corpus data is not specifically limited in this application. The corpus may contain various types of text data, such as books, news reports, social media, customer reviews, conversation records, etc. These corpus data can be used to train machine learning models to help machines better understand the structure and usage of human language. However, the actual collected corpus is usually full of noise, errors, and inconsistencies, which not only brings great challenges to model training, but also reduces the effect of the model in practical applications. Therefore, before the machine learning model is trained, the corpus data needs to be deeply analyzed and processed, and the systematic cleaning steps are used to improve the accuracy, consistency, and readability of the corpus data, thereby improving the training effect of the machine learning model and the performance of practical applications.

[0053] Step S2: preprocessing the original corpus data to obtain first corpus data.

[0054] In this embodiment, step S2 specifically includes:

[0055] Redundant information in the original corpus data is removed, and after formatting through text standardization, word segmentation is used to segment the text to obtain the first corpus data.

[0056] Specifically, data preprocessing is the basic link of corpus data quality inspection, including operations such as removing redundant information, text standardization and word segmentation. Removing redundant information can remove information that is irrelevant or repeated to subsequent analysis, so as to reduce the amount of data and improve processing efficiency; text standardization ensures that text data from different sources have a consistent format, eliminates analysis bias caused by format differences, and enables the model to more easily identify and learn patterns in the text; word segmentation is the basic step for subsequent analysis and processing, which can finely segment the text and cut continuous text into independent vocabulary units, which is more suitable for model training and analysis.

[0057] Step S3: Perform noise filtering on the first corpus data to obtain second corpus data.

[0058] Common noise data includes repeated text and format errors. Detecting and removing noise data through specific algorithms or rules can significantly improve the purity of the data and make model training more efficient and reliable.

[0059] In this embodiment, step S3 specifically includes:

[0060] Step S301: Acquire multimodal features, context relevance, information entropy and temporal characteristics of first corpus data.

[0061] In order to achieve unified processing of text, numerical and time series data in the corpus data, a multimodal representation learning function is created:

[0062]

[0063] Among them, R is the final multimodal representation, x_text, x_numeric and x_temporal are the input text data, numerical data and time series data respectively, f_text(), f_numeric() and f_temporal() are the corresponding encoding functions respectively, Represents the feature fusion operation.

[0064] In order to flexibly adjust the importance of each mode in noise judgment, adaptive weight calculation is introduced:

[0065] W = softmax(MLP(R))

[0066] Among them, W is the weight vector of each mode, MLP() is the multi-layer perceptron function used to learn the weights, R is the multimodal representation, and the softmax function ensures that the sum of the weights is 1, which is convenient for subsequent calculations.

[0067] In order to consider a wider range of contextual information, especially when processing text data, in order to identify noise that appears normal locally but is inconsistent in the global context. The Transformer model is used to capture long-range dependencies in the data:

[0068] C=Transformer(R)

[0069] Among them, C is the contextual relevance score vector and R is the multimodal representation.

[0070] In order to evaluate the uncertainty of data from the perspective of information theory and provide a new perspective for noise detection, a noise evaluation mechanism based on Shannon entropy is introduced:

[0071] H(R)=-∑p(r_i)*log(p(r_i))

[0072] Where R is the multimodal representation, H(R) is the information entropy of R, and p(ri) is the probability of the i-th element in R.

[0073] In view of the noise in the time series data, an improved dynamic time warping (DTW) algorithm is applied to filter the time series alignment noise. By comparing with the reference sequence, unreasonable time patterns are identified:

[0074] D=ImprovedDTW(x_temporal,reference_sequence)

[0075] Among them, x_temporal is the time series data, reference_sequence is the reference sequence, and D is the distance score between the time series data and the reference sequence.

[0076] Step S302: Perform a comprehensive noise score calculation based on the multimodal features, context relevance, information entropy and temporal characteristics of the first corpus data to obtain a noise value.

[0077] Taking into account multimodal features, contextual relevance, information entropy, and temporal characteristics, a comprehensive noise evaluation is provided, and the final noise score is calculated by the following formula:

[0078] S=α*(1-W·C)+β*H(R)+γ*D

[0079] Where S is the noise value, R is the multimodal representation, H(R) is the information entropy of R, C is the context relevance score vector, W is the weight vector of each modality, and α, β, and γ are adjustable weight coefficients used to balance the contributions of different components.

[0080] Step S303: setting an adaptive dynamic threshold according to the noise value. When the noise value is greater than the dynamic threshold, it is marked as a noise data point and filtered.

[0081] In order to adapt to different data distributions, a dynamic threshold is used:

[0082] T=μ(S)+k*σ(S)

[0083] Where T is the dynamic threshold, S is the noise value, μ(S) and σ(S) are the mean and standard deviation of S, respectively, and k is an adjustable parameter. When S>T, the data point is marked as noise and filtered.

[0084] A reinforcement learning mechanism is introduced to dynamically adjust the α, β, γ and k parameters:

[0085] θ_t+1=θ_t+η*▽J(θ_t)

[0086] θ represents the set of parameters that need to be optimized, η is the learning rate, and J(θ) is the performance evaluation function. This allows the algorithm to continuously optimize itself as the data changes.

[0087] A federated learning framework is also integrated, allowing the use of distributed data sources while protecting privacy:

[0088] w_global=∑(n_i / n)*w_i

[0089] Among them, w_global is the global model parameter, n_i is the size of the i-th local dataset, n is the sum of the sizes of all local datasets, and w_i is the local model parameter.

[0090] Step S304: providing a confidence assessment for the noise filtering decision according to the noise value and the dynamic threshold.

[0091] Provides a confidence estimate for each filtering decision:

[0092] Confidence=1-(ST) / S

[0093] Among them, Confidence is the confidence, S is the noise value, and T is the dynamic threshold.

[0094] This indicator provides a measure of the reliability of noise filtering decisions and enhances the interpretability of the system. The adaptive multimodal noise filtering algorithm used in this step provides a comprehensive and effective solution for noise filtering in complex data environments. It can not only process multiple types of data, but also adaptively adjust strategies and continuously optimize performance, while also considering data processing methods such as privacy protection and uncertainty quantification.

[0095] Step S4: Perform a semantic consistency check on the second corpus data to obtain third corpus data.

[0096] In this embodiment, step S4 specifically includes:

[0097] Step S401: Obtain semantic vectors of multiple texts in the second corpus data.

[0098] A method that combines multiple advanced language models to obtain the deep semantics of text:

[0099] V=α*BERT(T)+β*GloVe(T)+γ*ELMo(T)

[0100] Among them, V is the comprehensive semantic vector of text T, BERT, GloVe and ELMo represent different language models, and α, β, γ are the corresponding weight coefficients. This fusion method makes full use of the advantages of each model and can capture richer semantic information.

[0101] Step S402: For the two texts constituting the text pair, based on the semantic vectors, the similarity, consistency evaluation score and logical reasoning result between the two texts are respectively calculated.

[0102] In order to more accurately measure the semantic distance between texts, the improved cosine similarity formula is used to calculate the semantic similarity:

[0103] Sim(V1,V2)=(V1·V2) / (||V1||*||V2||)*(1+λ*tanh(||V1-V2||))

[0104] Among them, V1 and V2 are the semantic vectors of the two texts, λ is an adjustable parameter, and the tanh function is used to adjust the vector difference. By introducing the influence of the vector difference, the semantic difference can be reflected more accurately.

[0105] In order to comprehensively consider various semantic relationships between texts, a three-level consistency evaluation mechanism is used to calculate the consistency evaluation scores between texts:

[0106]

[0107] Among them, WordNetSim is a similarity calculation function based on WordNet, ConceptNetRel function uses the network structure of ConceptNet to evaluate the conceptual relationship and interaction between two texts, LogicNetInf is a function based on logical network reasoning, L1, L2, L3 represent the consistency scores at the vocabulary, concept and logic levels respectively, w1, w2, w3 are the corresponding weights, and C is the consistency evaluation score.

[0108] In order to understand the complex logical relationship between texts, a reasoning module based on graph neural network is introduced:

[0109] R=GNN(ConstructGraph(T1,T2))

[0110] The ConstructGraph(T1,T2) function is used to construct a graph based on the text T1 and T2, and R represents the logical reasoning result output from the GNN. This module can effectively identify complex logical relationships such as cause and effect and condition by constructing a semantic graph and then using the graph neural network for reasoning.

[0111] Step S403: Based on the similarity, consistency evaluation score and logical reasoning results between the two texts, the degree of contradiction between the two texts is quantified, and the quantified contradiction detection value is compared with the preset detection threshold. When the contradiction detection value is greater than the detection threshold, it is determined that there is a contradiction between the two texts.

[0112] Create a comprehensive contradiction detection function based on the similarity between texts, consistency evaluation scores, and logical reasoning results:

[0113] Contradiction(T1,T2)=f(Sim(V1,V2),C,R)

[0114] Among them, Contradiction(T1, T2) represents the quantitative value of the degree of contradiction between the two texts, Sim(V1, V2) is the similarity calculation between the semantic vectors V1 and V2 of the two texts, C represents the multi-level consistency evaluation score, R is the logical reasoning result based on the graph neural network, and f is a composite function that can be adjusted according to the actual application scenario. When the contradiction detection value output by this composite function exceeds the preset detection threshold, the system determines that there is a contradiction between the two texts.

[0115] The detection threshold uses an adaptive threshold adjustment mechanism:

[0116] θ_t+1=θ_t+η*(ΔP / Δt)

[0117] Among them, θ_t is the threshold at time t, η is the learning rate, and ΔP / Δt is the performance change rate. This enables the system to dynamically adjust the judgment criteria according to actual conditions and maintain optimal performance.

[0118] Step S404: Based on the contradiction detection value, a graph optimization algorithm is used to perform global semantic consistency optimization on a set of different text pairs.

[0119] In order to ensure the global consistency of the entire corpus, a graph optimization algorithm is used for calculation, which can ensure local consistency while maximizing the logical coherence of the entire corpus:

[0120] G*=argmin_GΣ(i,j)∈E w_ij*Contradiction(T_i,T_j)

[0121] Among them, G* is the optimized corpus, E is the set of text pairs, w_ij is the weight of the edge, and the evaluation function Contradiction is used to quantify the degree of contradiction between two texts T_i and T_j.

[0122] Step S4 adopts this multi-level semantic consistency assessment framework, which greatly improves the accuracy and comprehensiveness of semantic consistency checking through in-depth semantic analysis and logical reasoning. In practical applications, it can effectively identify potential logical contradictions that are difficult to find with traditional methods.

[0123] Step S5: Correct grammatical errors in the third corpus data to obtain fourth corpus data.

[0124] In this embodiment, step S5 specifically includes:

[0125] Step S501: Obtain a context encoding vector of the third corpus data.

[0126] In order to accurately understand and process complex language features, we first perform multimodal context encoding in this step:

[0127] C=f(αT+βS+γV)

[0128] Among them, C is the encoding vector of comprehensive context, f is a nonlinear activation function (such as ReLU), T is the text information vector, S is the speech information vector, V is the visual information vector, α, β, γ are the weight coefficients of each modality, and α+β+γ=1 is satisfied.

[0129] This step fuses text, speech, and visual information to create an encoding vector of a comprehensive contextual representation that captures subtle linguistic features that may be overlooked by a single modality.

[0130] Step S502: Based on the context coding vector, a graph neural network is used to model the grammatical structure, and the grammatical structure is encoded into a graph structure.

[0131] The context encoding vector is used as one of the input features of the graph neural network model, and the grammatical structure of the sentence is modeled through the graph neural network:

[0132] h_v^(k+1)=σ(W1·MEAN({h_u^(k):u∈N(v)})+W2·h_v^(k)+b)

[0133] Among them, h_v^(k) represents the hidden state of node v after the kth iteration, σ represents the activation function (such as tanh), W1 and W2 represent learnable weight matrices, b represents the bias vector, N(v) represents the neighbor set of node v, and MEAN represents the average aggregation function.

[0134] This step can encode the grammatical structure of the sentence into a graph, which can effectively capture long-distance dependencies and complex grammatical relations.

[0135] Step S503: Based on the graph structure, a reinforcement learning-driven error correction strategy is created to automatically identify and correct grammatical errors in the text.

[0136] Based on the grammatical structure modeling of the graph neural network, the state of each grammatical structure or the text state after each possible correction action can be defined as the state in reinforcement learning. The policy value network is used for deep reinforcement learning to obtain the optimal error correction strategy:

[0137] Q(s_t,a_t)=(1-α)·Q(s_t,a_t)+α·[r_t+γ·max_a Q(s_{t+1},a)]

[0138] Among them, Q(s_t,a_t) represents the Q value of taking action a_t in state s_t, α represents the learning rate (0<α≤1), r_t represents the immediate reward, γ represents the discount factor (0≤γ<1), s_t, s_{t+1} represent the current state and the next state, and a represents a possible action.

[0139] Considering the long-term impact of the correction operation, the reinforcement learning algorithm not only considers the direct result of the current action, but also the impact of this action on the future state and the possible long-term rewards. Through continuous trial and error and adjustment, it learns to find the optimal error correction strategy to automatically identify and correct grammatical errors in the text.

[0140] Step S504: Use a multi-head attention mechanism to evaluate the relevance of the corrected words in the corrected text with the context.

[0141] The multi-head attention mechanism is used to enhance the contextual relevance evaluation:

[0142] Attention(Q,K,V)=softmax(QK T / d_k)V

[0143] MultiHead(Q,K,V)=Concat(head_1,...,head_h)W O

[0144] Where Q, K, V represent query, key, and value matrices, d_ represents the dimension of the key, head_i represents the i-th attention head, and W O Represents the output projection matrix.

[0145] This step can accurately assess the relevance between the word to be corrected and its context, which helps in processing long sentences and complex grammatical structures.

[0146] In order for the model to learn how to quickly adapt to new tasks, in the context of grammatical error correction, meta-learning is used to improve the adaptability of the model:

[0147] θ'=θ-α▽_θL_τ(f_θ)θ_τ*=θ'-β▽_θ'L_τ(f_θ')

[0148] Among them, θ represents the initial model parameters, θ' represents the parameters updated in the previous step of task τ, θ_τ* represents the parameters finally adapted on task τ, α and β represent the learning rates of the inner and outer loops, L_τ represents the loss function of task τ, and f_θ represents the parameterized model function.

[0149] Model adaptation based on meta-learning enables the model to quickly adapt to new grammatical rules or writing styles when faced with texts in different fields, styles or language characteristics without a lot of retraining, and quickly adjust its correction strategy to achieve better correction results.

[0150] Step S505: Based on the error detection probability of the graph neural network, the error detection probability of the multi-head attention mechanism and the error detection probability of the language model, a comprehensive error detection mechanism is used to detect the corrected text to obtain the error detection score of each original word in the text.

[0151] The corrected text is tested using a comprehensive error detection mechanism:

[0152] E(w_i)=λ1*P_GNN(w_i)+λ2*P_ATT(w_i)+λ3*P_LM(w_i)

[0153] Among them, E(w_i) represents the error detection score of word w_i, P_GNN(w_i) represents the error probability based on graph neural network, P_ATT(w_i) represents the error probability based on attention mechanism, P_LM(w_i) represents the error probability based on language model, λ1, λ2, λ3 represent the weights of each component respectively, satisfying λ1+λ2+λ3=1.

[0154] This step can fuse information from different model components to make more reliable error detection.

[0155] Step S506: for the original word whose error detection score exceeds a preset value, generate evaluation scores of multiple candidate correction words and sort them, and select the best candidate correction word for grammatical correction.

[0156] The calculation formula for the evaluation score of the candidate correction word is as follows:

[0157] Score(c_j)=ω1*(1 / ED(c_j,w_i))+ω2*log(P(c_j|context))+ω3*Coh(c_j,G)

[0158] Among them, c_j represents the j-th candidate correction word, Score(c_j) represents the evaluation score of the j-th candidate correction word, ED(c_j, w_i) represents the edit distance between c_j and the original word w_i, P(c_j|context) represents the candidate word probability based on the context, Coh(c_j, G) represents the consistency score between the candidate word and the graph structure G, ω1, ω2, ω3 represent the weights of each factor, and satisfy ω1+ω2+ω3=1.

[0159] The evaluation scores of the candidate correction words are ranked, and the best candidate correction word is selected by comprehensively considering multiple factors such as balancing edit distance, context probability, and graph structure consistency.

[0160] In order to efficiently identify and correct various grammatical errors, this step involves an innovative combination of advanced technologies such as multimodal information processing, graph neural networks, reinforcement learning, attention mechanisms and meta-learning, and proposes a multimodal context-aware grammatical error correction algorithm (MC-GECA). The MC-GECA algorithm can correct grammatical errors more comprehensively and accurately. It can not only effectively handle common grammatical errors, but also has the potential to adapt to language evolution, providing a new solution for improving the quality of corpus and model language comprehension capabilities.

[0161] Step S6: The fourth corpus data is audited using a multi-level audit mechanism, suspected problematic corpora are annotated and feedback is provided, and the feedback results are continuously monitored and evaluated to improve the accuracy of the annotated data.

[0162] In this embodiment, step S6 specifically includes:

[0163] Step S601: In the primary review, the suspected problematic corpus initially marked by the automated quality inspection algorithm is quickly screened, and the accuracy of the primary review is quantitatively evaluated;

[0164] Step S602: In advanced review, for complex or problem corpora that are difficult for automated quality inspection algorithms to identify, contextual prompts and analysis suggestions provided by natural language processing technology are used to annotate the problem corpora and quantitatively evaluate the matching degree of the advanced review;

[0165] Step S603: input the annotated question corpus as optimized training data into the automated quality inspection algorithm for retraining, optimize the parameters of the automated quality inspection algorithm, and quantitatively evaluate the recognition accuracy of the updated automated quality inspection algorithm.

[0166] Specifically, manual review can identify subtle problems that are difficult for automated quality inspection algorithms to capture, and provide the algorithm with high-quality annotated data to further optimize the algorithm's performance. Therefore, combining manual review with annotation and feedback on suspected problematic corpora is one of the key steps in corpus data quality inspection. Through a continuous feedback loop, the combination of manual review and automated quality inspection methods can continuously improve the accuracy and efficiency of corpus cleaning.

[0167] A multi-level review mechanism combining primary review and senior review is adopted. After the automated quality inspection algorithm initially marks the suspected problematic corpus, the system submits the data to the primary reviewer. The primary reviewer is responsible for quickly screening the suspected problematic corpus marked by the automated quality inspection algorithm. The review quality is evaluated through quantitative assessment. The accuracy of the primary review can be defined as:

[0168] Accuracy = number of correct annotations / total number of annotations

[0169] For complex or automated quality inspection algorithms that are difficult to identify, senior reviewers will conduct more in-depth analysis and annotation. Senior reviewers use intelligent auxiliary tools to provide contextual prompts and analysis suggestions through natural language processing (NLP) technology to help reviewers make accurate judgments quickly and further improve the quality of annotations. The quantitative assessment of review accuracy will be measured by the degree of match between the reviewer's annotations and the standard answers:

[0170] Matching degree = number of correct annotations / number of standard annotations

[0171] All manual review results will be used as feedback data to adjust and optimize the parameters and strategies of the cleaning algorithm. The system automatically optimizes the parameters of the algorithm through the reinforcement learning mechanism, gradually reducing the reliance on manual review, making the corpus cleaning process more efficient and intelligent. The effect of algorithm improvement can be measured by comparing the accuracy, recall rate and other indicators before and after the algorithm update.

[0172] In this embodiment, step S6 further includes:

[0173] Introducing distributed artificial intelligence technology, multiple reviewers conduct independent review of the same corpus, and using a weighted voting mechanism to aggregate the review results of each reviewer;

[0174] Use blockchain technology to record the process of each review and annotation, and measure the supervision effect of the review process through the statistics of consistency between reviewers;

[0175] Through continuous supervision and evaluation, the review standards and processes are dynamically adjusted to adapt to the quality inspection task requirements of different types of corpus data.

[0176] Specifically, in order to avoid subjectivity in manual review, the system has developed a set of detailed review standards and operating guidelines, and through regular training and assessment mechanisms to ensure that reviewers can accurately understand and implement the standards. The quantitative assessment indicators of review quality include the reviewer's learning curve and error rate:

[0177] Error rate = number of incorrect annotations / total number of annotations

[0178] The system also introduces distributed artificial intelligence (DAI) technology to achieve efficient collaboration among multiple reviewers through an intelligent collaboration mechanism. Specifically, by introducing multiple reviewers to independently review the same corpus and using a voting mechanism to aggregate the review results, the impact of personal bias on the review results is effectively reduced, ensuring the objectivity of the review results. The weight of the review result can be calculated based on the reviewer's historical performance using the following formula: i =Q i / ∑ n j＝1 Q j , where Wi is the weight of the i-th reviewer, Q i Score the quality of the reviewer's historical reviews. By dynamically adjusting the weights, the system can reduce the impact of personal subjective bias and improve the reliability of the review results.

[0179] At the same time, the system uses blockchain technology to record the process of each review and annotation, monitor the review process in real time, and ensure the transparency and immutability of all operations. Each review record is archived through blockchain technology to ensure that each link can be traced and reviewed when a dispute arises, providing traceable evidence for subsequent audits and dispute resolution, and ensuring the credibility of the entire review process.

[0180] The supervisory effect of the audit process can be measured by the statistic of consistency between auditors (Kappa coefficient): Kappa coefficient = (Po-Pe) / (1-Pe), where Po is the observed consistency and Pe is the random consistency.

[0181] Therefore, the system dynamically adjusts the review standards and processes through continuous supervision and evaluation to adapt to different corpus types and task requirements. By introducing a reinforcement learning mechanism, the system can self-optimize and improve the overall efficiency and accuracy of corpus cleaning.

[0182] In summary, the embodiment of the present invention provides a method for cleaning and quality inspection of corpus data, which obtains original corpus data; performs data preprocessing on the original corpus data to obtain first corpus data; performs noise filtering on the first corpus data to obtain second corpus data; performs semantic consistency check on the second corpus data to obtain third corpus data; performs grammatical error correction on the third corpus data to obtain fourth corpus data; and audits the fourth corpus data using a multi-level audit mechanism, annotates and provides feedback on the existing suspected problem corpus, and continuously monitors and evaluates the feedback results to improve the accuracy of the annotated data. The present invention effectively reduces the noise and errors in the corpus data through systematic cleaning steps, significantly improves the quality of the corpus data, and greatly reduces the frequency of manual intervention, thereby improving the efficiency of data processing. High-quality corpus data can significantly improve the performance of the model in various natural language processing tasks, thereby improving the training effect of the machine learning model and the performance of practical applications.

[0183] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0184] The electronic device includes: one or more processors; and a memory storing computer program instructions, and when the computer program instructions are executed, the processor executes the steps of the corpus data cleaning and quality inspection method provided in any one or more of the above embodiments. Figure 2 An exemplary structural diagram of the electronic device is disclosed. Figure 2As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0185] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0186] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0187] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0188] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the corpus data cleaning and quality inspection method provided in any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0189] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0190] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0191] It should be noted that more specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0192] Computer readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0193] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0194] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0195] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0196] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0197] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A method for cleaning and quality inspection of corpus data, It is characterized in that include: Get the original corpus data; Performing data preprocessing on the original corpus data to obtain first corpus data; Performing noise filtering on the first corpus data to obtain second corpus data; Performing a semantic consistency check on the second corpus data to obtain third corpus data; Correcting grammatical errors of the third corpus data to obtain fourth corpus data; The fourth corpus data is audited using a multi-level audit mechanism, suspected problematic corpora are annotated and feedback is provided, and the feedback results are continuously monitored and evaluated to improve the accuracy of the annotated data.

2. The method for cleaning and quality inspection of corpus data according to claim 1, It is characterized in that The step of performing data preprocessing on the original corpus data to obtain the first corpus data comprises: Redundant information in the original corpus data is removed, and after formatting through text standardization, word segmentation is performed to segment the text to obtain the first corpus data.

3. The method for cleaning and quality inspection of corpus data according to claim 1, It is characterized in that The step of performing noise filtering on the first corpus data to obtain second corpus data comprises: Acquire multimodal features, context relevance, information entropy, and temporal characteristics of the first corpus data; Perform a comprehensive noise score calculation based on the multimodal features, context relevance, information entropy, and temporal characteristics of the first corpus data to obtain a noise value; An adaptive dynamic threshold is set according to the noise value, and when the noise value is greater than the dynamic threshold, it is marked as a noise data point and filtered; A confidence assessment is provided for a noise filtering decision based on the noise value and the dynamic threshold.

4. The method for cleaning and quality inspection of corpus data according to claim 1, It is characterized in that The step of performing a semantic consistency check on the second corpus data to obtain the third corpus data comprises: Obtaining semantic vectors of multiple texts in the second corpus data; For two texts constituting a text pair, respectively calculating the similarity, consistency evaluation score and logical reasoning result between the two texts based on the semantic vector; Based on the similarity, consistency evaluation score and logical reasoning result between the two texts, the degree of contradiction between the two texts is quantified, and the contradiction detection value obtained by quantification is compared with a preset detection threshold. When the contradiction detection value is greater than the detection threshold, it is determined that there is a contradiction between the two texts; Based on the contradiction detection value, a graph optimization algorithm is used to perform global semantic consistency optimization on a set consisting of different text pairs.

5. The method for cleaning and quality inspection of corpus data according to claim 1, It is characterized in that The step of correcting grammatical errors in the third corpus data to obtain fourth corpus data comprises: Obtaining a context encoding vector of the third corpus data; Based on the context encoding vector, a graph neural network is used to model the grammatical structure, and the grammatical structure is encoded into a graph structure; Based on the graph structure, creating a reinforcement learning driven error correction strategy to automatically identify and correct grammatical errors in the text; A multi-head attention mechanism is used to evaluate the relevance of the corrected words and the context in the corrected text; Based on the error detection probability of the graph neural network, the error detection probability of the multi-head attention mechanism, and the error detection probability of the language model, the corrected text is detected using a comprehensive error detection mechanism to obtain an error detection score for each original word in the text; For the original word whose error detection score exceeds a preset value, evaluation scores of multiple candidate correction words are generated and sorted, and the best candidate correction word is selected for grammatical correction.

6. The method for cleaning and quality inspection of corpus data according to claim 1, It is characterized in that The steps of adopting a multi-level review mechanism to review the fourth corpus data, marking and providing feedback on the suspected problematic corpus, and subjecting the feedback results to continuous supervision and evaluation include: In the primary review, the suspected problematic corpus initially marked by the automated quality inspection algorithm is quickly screened, and the accuracy of the primary review is quantitatively evaluated; In advanced review, for complex problem corpora or those that are difficult for the automated quality inspection algorithm to identify, contextual prompts and analysis suggestions provided by natural language processing technology are used to annotate the problem corpora and quantitatively evaluate the matching degree of the advanced review; The annotated question corpus is input as optimized training data into the automated quality inspection algorithm for retraining, parameters of the automated quality inspection algorithm are optimized, and the recognition accuracy of the updated automated quality inspection algorithm is quantitatively evaluated.

7. The method for cleaning and quality checking corpus data according to claim 6, It is characterized in that The step of using a multi-level review mechanism to review the fourth corpus data, annotating and providing feedback on the suspected problematic corpus, and continuously monitoring and evaluating the feedback results also includes: Introducing distributed artificial intelligence technology, multiple reviewers conduct independent review of the same corpus, and using a weighted voting mechanism to aggregate the review results of each reviewer; Use blockchain technology to record the process of each review and annotation, and measure the supervision effect of the review process through the statistics of consistency between reviewers; Through continuous supervision and evaluation, the review standards and processes are dynamically adjusted to adapt to the quality inspection task requirements of different types of corpus data.

8. An electronic device, It is characterized in that The electronic device comprises: One or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed by the processor, implement the steps of the corpus data cleaning and quality inspection method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, It is characterized in that When the computer program / instructions are executed by a processor, the steps of the corpus data cleaning and quality inspection method as described in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instruction is executed by a processor, the steps of the corpus data cleaning and quality inspection method as described in any one of claims 1 to 7 are implemented.

Images