Data quality evaluation method and device fusing security attributes

Abstract

The invention discloses a security attribute fused data quality evaluation method and device, and the method comprises the following steps: obtaining corpus data, and carrying out the preprocessing of the corpus data, and obtaining a to-be-evaluated text; obtaining a preset reference text, and determining a text quality score according to the frequency that the to-be-evaluated text contains the keywords of the reference text; inputting the to-be-evaluated text into the trained target model, and outputting a text safety score through the target model based on the similarity between the text embedding vector of the to-be-evaluated text and the features in the built-in first quality feature space and the second quality feature space; according to the text quality score and the text safety score, performing weighted calculation to obtain a target score of the to-be-evaluated text, and evaluating the corpus data through the text quality score and the text safety score, so that the corpus data can be strictly screened, and potential safety hazard factors in the corpus data can be accurately identified; and the corpus data is ensured to have high quality and diversity.

Description

Technical Field

[0001] The present invention relates to the field of automated assessment technology, and in particular to a data quality assessment method and device integrating security attributes. Background Art

[0002] During the training process of large-scale data, the evaluation of the corpus is a complex and challenging task. As the field of natural language generation develops and the complexity of evaluation systems increases, evaluation methods must be both reliable and explainable and comprehensive. To reduce the need for manual evaluation, many automated evaluation metrics have been developed, such as word overlap evaluation metrics such as BLEU, ROUGE, and METEOR, which are based on heuristic criteria and rules. With the advancement of pre-training technology, metrics such as BERTScore, BARTScore, and ParaScore use pre-trained models to improve the evaluation process. In addition, the natural language generation community has paid more and more attention to evaluation methods based on large language models, which use the understanding and reasoning capabilities of large language models and are widely used.

[0003] However, existing automated evaluation indicators usually only focus on single factors such as fluency, coherence and relevance, and evaluate each sample independently, ignoring the complex relationships between samples. At present, there is a lack of a more comprehensive automated evaluation method that includes content security attributes and can scientifically and accurately evaluate the quality of text generated by large models to obtain high-quality text data. Summary of the invention

[0004] To solve the above problems, the purpose of the present invention is to provide a data quality assessment method, device and storage medium that integrate security attributes, which can strictly screen corpus data and accurately identify security risk factors in the corpus data, ensure that the corpus data is of high quality and diversity, and help improve the generalization ability and overall performance of large language models.

[0005] The technical solution adopted by the present invention to solve the problem is:

[0006] In a first aspect, an embodiment of the present application provides a data quality assessment method integrating security attributes, the method comprising: obtaining corpus data, and preprocessing the corpus data to obtain a text to be evaluated; obtaining a preset reference text, and determining a text quality score based on the frequency of the key words of the reference text contained in the text to be evaluated; inputting the text to be evaluated into a trained target model, and outputting a text security score through the target model based on the similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space respectively; wherein the target model is obtained by performing similarity analysis and learning of the first quality feature and the second quality feature based on different sample texts and the text security scores of the corresponding sample texts by a preset model; and performing weighted calculation based on the text quality score and the text security score to obtain a target score for the text to be evaluated.

[0007] In a second aspect, an embodiment of the present application provides a data quality assessment device integrating security attributes, comprising: an acquisition module, used to acquire corpus data and preprocess the corpus data to obtain a text to be evaluated; a determination module, used to acquire a preset reference text and determine a text quality score according to the frequency of the key words of the reference text contained in the text to be evaluated; an output module, used to input the text to be evaluated into a trained target model, and output a text security score through the target model based on the similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space respectively; wherein the target model is obtained by performing similarity analysis and learning of the first quality feature and the second quality feature based on different sample texts and the text security scores of the corresponding sample texts by a preset model; and an assessment module, used to perform weighted calculation according to the text quality score and the text security score to obtain a target score for the text to be evaluated.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data quality assessment method for integrating security attributes as described above is implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data quality assessment method for integrating security attributes as described above.

[0010] In an embodiment of the present application, corpus data is obtained and preprocessed to obtain a text to be evaluated; a preset reference text is obtained, and a text quality score is determined according to the frequency of the key words of the reference text contained in the text to be evaluated; the text to be evaluated is input into a trained target model, and a text safety score is output through the target model based on the similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space respectively; a weighted calculation is performed according to the text quality score and the text safety score to obtain a target score for the text to be evaluated, and the corpus data is evaluated according to the text quality score and the text safety score, so that the corpus data can be strictly screened and the safety hazard factors in the corpus data can be accurately identified, thereby ensuring that the corpus data has high quality and diversity, and helping to improve the generalization ability and overall performance of the large language model.

[0011] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a data quality assessment method integrating security attributes according to an embodiment of the present invention;

[0013] Figure 2 for Figure 1 Flowchart of step S1000;

[0014] Figure 3 for Figure 2 Flowchart of step S1300;

[0015] Figure 4 for Figure 1 Flow chart of step S2000;

[0016] Figure 5 for Figure 1 Flowchart of step S3000;

[0017] Figure 6 for Figure 1 A flowchart of step S3000 in another embodiment;

[0018] Figure 7 for Figure 6 Flowchart of step S3500 in FIG.

[0019] Figure 8 for Figure 1 Flowchart of step S4000;

[0020] Fig. 9 for Figure 5 The structural diagram of the target model in the figure;

[0021] Fig.10 This is a structural diagram of a data quality assessment device integrating security attributes according to an embodiment of the present invention;

[0022] Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0024] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0025] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0026] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0027] The data quality assessment method and device for integrating security attributes involved in the embodiment of the present invention are text assessment methods based on natural language processing (NLP) technology. Specifically, natural language processing technology is an important direction in the fields of computer science and artificial intelligence. The research on natural language processing technology can realize various theories and methods for effective communication between people and computers using natural language. Natural language processing refers to the technology of interactive communication with machines using natural language used by human communication. Through artificial processing of natural language, computers can read and understand it. Related research on natural language processing began with human exploration of machine translation. Although natural language processing involves multi-dimensional operations such as speech, grammar, semantics, and pragmatics, in simple terms, the basic task of natural language processing is to segment the processed corpus based on ontology dictionaries, word frequency statistics, contextual semantic analysis, etc., to form a word unit with the smallest part of speech as the unit and rich in semantics.

[0028] Corpus refers to language materials, including spoken materials and written materials. The original source of the corpus may be textbooks, newspapers, comprehensive journals, professional journals, books, etc. In practical applications, the evaluation of corpus is a complex and challenging task in the process of training large-scale data. With the development of the field of natural language generation and the increasing complexity of evaluation systems, evaluation methods must be reliable, explainable and comprehensive. Existing automated evaluation indicators usually only focus on single factors such as fluency, coherence and relevance, and evaluate each sample independently, ignoring the complex relationships between samples. At present, there is a lack of a more comprehensive automated evaluation method that includes content security attributes, which can scientifically and accurately evaluate the quality of text generated by large models to obtain high-quality text data.

[0029] Based on the above, an embodiment of the present invention provides a data quality assessment method and device integrating security attributes, which obtains corpus data and preprocesses the corpus data to obtain a text to be evaluated; obtains a preset reference text, and determines a text quality score according to the frequency of the key words of the reference text contained in the text to be evaluated; inputs the text to be evaluated into a trained target model, and outputs a text security score based on the similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space respectively through the target model; performs weighted calculation according to the text quality score and the text security score to obtain a target score for the text to be evaluated, and evaluates the corpus data through the text quality score and the text security score, so as to strictly screen the corpus data and accurately identify the potential safety hazards in the corpus data, thereby ensuring that the corpus data has high quality and diversity, and helping to improve the generalization ability and overall performance of the large language model.

[0030] See also Figure 1 , Figure 1 FIG. 2 shows a flow chart of a data quality assessment method for integrating security attributes provided by an embodiment of the present invention. Figure 1 As shown, the data quality assessment method for integrating security attributes in an embodiment of the present invention includes the following steps:

[0031] Step S1000: Acquire corpus data, and preprocess the corpus data to obtain a text to be evaluated.

[0032] It is understandable that in order to accurately classify the corpus data, it is necessary to define the data quality level of the corpus data. For example, according to the data quality of the corpus data, the corpus data is divided into the second quality feature space, the general quality sample and the first quality feature space. The specific definitions are as follows: For the second quality feature space, if the corpus data contains illegal and non-compliant content, such as malicious advertising, verbal abuse, pornography and other unsafe content, and contains special and weird content, a large number of repeated sentences in the sample, the semantic content of the previous and next paragraphs in the sample is very different, the knowledge content of the sample is very low, the text content is not very meaningful, the sample contains a large number of html and other tags, contains character encoding errors, garbled characters, invalid web links, contains a large number of advertisements, science and engineering formula format errors, etc., it is considered to be the second quality feature space; for general quality samples, the content of the corpus data is fluent, there are no semantic problems, the expression is clear and coherent, there are no problems with the security attributes of the content, although there are a small number of advertisements mixed in, it does not affect the reading and understanding of the main content, the theme is messy and there is no clear central idea, but the overall information still has certain value, etc., it is considered to be a general quality sample; for the first quality feature space, the semantics of the corpus data is fluent, the content is complete and rich, there is no other related content, and the readability and knowledge content are high, it is considered to be the first quality feature space.

[0033] Similarly, the corpus data sets are graded according to the quality level of the corpus data. For example, the corpus data sets are divided into low-quality data sets, general quality data sets and high-quality data sets. The specific definitions are as follows: Low-quality data sets, in the entire corpus data set, if the second quality feature space accounts for more than or equal to 1%, the data needs to be cleaned again in a targeted manner; if after debugging, the data quality problem cannot be further solved, then the corpus data set cannot be used for training. General quality data sets, in the entire corpus data set, if the second quality feature space accounts for less than 1% and greater than 0.1%, the data needs to be cleaned again in a targeted manner; if after debugging, the data quality problem cannot be further solved, then the data set is considered to be a general quality data set and can be used for training. High-quality data sets, in the entire corpus data set, if the second quality feature space accounts for less than 0.1%, then the corpus data set is considered to be a high-quality data set.

[0034] See also Figure 2 , Figure 2 FIG. 5 is a schematic diagram showing a specific implementation process of another embodiment of the above step S1000. Figure 2 As shown, step S1000 at least includes the following steps:

[0035] Step S1100: Perform format conversion processing on the corpus data to obtain corpus data in a unified format.

[0036] It is understandable that due to the different sources of corpus data, the format of corpus data will be inconsistent. In order to improve the processing speed and evaluation efficiency of corpus data, the format of corpus data needs to be processed uniformly. Specifically, format preprocessing converts corpus data in different formats (such as epub, txt, mobi, etc.) into JSON (JavaScript Object Notation) format for subsequent processing.

[0037] It is understandable that JSON is a lightweight data exchange format that uses a text format that is completely independent of programming languages ​​to store and represent data. The concise and clear hierarchical structure makes JSON an ideal data exchange language. In addition, JSON is easy for people to read and write, and is also easy for machines to parse and generate, and effectively improves network transmission efficiency. Specifically, JSON can convert a set of data represented in a JavaScript object into a string, and then the string can be easily transferred between networks or programs, and the string can be restored to a data format supported by each programming language when necessary. In practical applications, format conversion of corpus data is an important link in the field of language processing, which involves converting corpora of different formats into a unified or specific format for subsequent analysis and processing. Commonly used corpus format conversion tools include Olifant, Xbench, etc., which belong to the prior art and will not be described here.

[0038] Step S1200: Perform data cleaning on the corpus data in a unified format to obtain cleaned corpus data.

[0039] It can be understood that data cleaning is the process of re-examining and verifying data in order to remove duplicate information, correct existing errors, and provide data consistency. Data cleaning is the final step in discovering and correcting identifiable errors in data files, including checking data consistency, handling invalid values ​​and missing values, etc.

[0040] In this embodiment, data cleaning processing is performed on the unified format corpus data on a chapter basis, the number of occurrences of bad words and the proportion of the whole text are detected, and it is determined whether the text in the unified format corpus data is bad text. Different categories of bad word detection are supported, and different levels of strictness of detection are achieved by adjusting the threshold.

[0041] Step S1300: perform deduplication processing on the cleaned corpus data to obtain the text to be evaluated.

[0042] It is understood that deduplication refers to the removal of duplicate records in a data set to maintain the uniqueness of each record. This process is very important in the processing of corpus data, especially when performing data cleaning and preprocessing, because duplicate data may lead to inaccurate evaluation results of corpus data. Especially when processing large-scale data, data deduplication is a common problem. Methods for processing big data deduplication include distributed deduplication, memory optimization, hash-based deduplication methods, sorting-based deduplication methods, inverted index-based deduplication methods, Bloom filters, cardinality algorithms, etc.

[0043] See also Figure 3 , Figure 3 FIG. 5 is a schematic diagram showing a specific implementation process of another embodiment of the above step S1300. Figure 3 As shown, step S1300 at least includes the following steps:

[0044] Step S1310: Deduplication processing is performed on the cleaned corpus information of multiple samples to obtain deduplication corpus information.

[0045] It can be understood that the deduplication between samples is aimed at the repeated text after cleaning, and it is deduplicated at the chapter level, and a combination of local sensitive hashing (LSH) and Jaccard algorithm is used to achieve efficient deduplication. Specifically, the data is hashed, and the data with the same hash value is divided into the same group, and then each group of data is compared to remove duplicate data. This method has a low time complexity and can complete deduplication in a shorter time, but it is necessary to pay attention to the problem of hash conflicts. Use a hash function to hash each data, group the data with the same hash value into the same group, and then compare whether the data grouped into the same group is exactly the same to remove duplicate data. However, hash conflicts may lead to misjudgment and need to be used in combination with other methods, such as the Jaccard algorithm.

[0046] Step S1320: perform intra-sample deduplication processing on deduplication corpus information in the same sample to obtain a text to be evaluated.

[0047] It is understandable that the deduplication within the sample is to count the repetition of the corresponding paragraphs and phrases in the same sample through the N-Gram method. For example, a ratio threshold of 2-10 grams is set to ensure the diversity of data in the same sample. Among them, N-Gram is a language model commonly used in large-vocabulary continuous speech recognition. Specifically, the N-Gram model performs a sliding window operation of size N on the bytes of the text content, forming a sequence of byte segments of length N. Each byte segment is called a gram, and the frequency of occurrence of all grams is counted.

[0048] Step S2000: Obtain a preset reference text, and determine a text quality score according to the frequency of the key words of the reference text contained in the text to be evaluated.

[0049] It is understandable that before performing automated quality assessment on the text to be evaluated, it is necessary to screen the text to be evaluated using the PPL value through a language model. The PPL value (Perplexity) is an indicator used in NLP to measure the quality of a language model. Specifically, the PPL value mainly estimates the probability of a sentence appearing based on each word and normalizes it with the length of the sentence. In essence, the PPL value calculates the probability of a sentence. A target model that assigns a higher probability value to the text to be evaluated is considered a better model because it means that the target model can better fit the data in the text to be evaluated.

[0050] In natural language processing, the PPL value is widely used to evaluate the quality of language models. Compared with cross entropy, the PPL value is easier to calculate and more suitable for directly comparing the quality of two models. In addition, the smaller the PPL value, the higher the probability that the language model assigns to the sentences in the text to be evaluated, which means that the target model fits the test data better.

[0051] It can be understood that the target score of the text to be evaluated is determined by the text quality score of the text to be evaluated and the text safety score of the text to be evaluated. Among them, the text quality score is based on the pre-trained encoder-decoder model, which is used to evaluate the coherence, information content and fluency of the text in the text to be evaluated; in addition, the text safety score evaluates the content safety by calculating the cosine similarity between the embedding of the subspace corresponding to the text to be evaluated and various content attributes. Through this comprehensive evaluation method, including content safety attributes, the defects of existing automated methods are effectively solved, thereby obtaining high-quality text data.

[0052] See also Figure 4 , Figure 4 FIG. 2 shows a schematic diagram of a specific implementation process of another embodiment of the above step S2000. Figure 4 As shown, step S2000 at least includes the following steps:

[0053] Step S2100: Calculate the frequency of the key words of the reference text included in the text to be evaluated to obtain the text frequency value.

[0054] It can be understood that the text quality score is based on a pre-trained encoder-decoder model to evaluate the coherence, informativeness, and fluency of the text within the evaluated text. In addition, the text safety score evaluates content safety by calculating the cosine similarity between the embeddings of the evaluated text and the subspaces corresponding to various content attributes. Through this comprehensive evaluation method, including content safety attributes, the defects of existing automated methods are effectively addressed, thereby obtaining high-quality text data.

[0055] Specifically, given an end-to-end model represented by parameters, the text to be evaluated has a source sequence x containing n tokens = {x1,…,x n} and a reference text y = {y1,…,y m}. It is understandable that in NLP, a token refers to a basic unit in a text, which can usually be a word, a phrase, a punctuation mark, a character, etc., depending on the needs and methods of text processing. Dividing text into several tokens is the first step in text processing, and this process is called tokenization. In machine learning and deep learning, text usually needs to be converted into digital representations before it can be passed as input to the algorithm for processing.

[0056] In some text processing tasks, such as text classification, sentiment analysis, named entity recognition, etc., dividing the text into tokens and representing them numerically is a necessary preprocessing step. This can convert text data into a numerical form that the model can process, so as to perform subsequent feature extraction, model training, and prediction operations.

[0057] It can be understood that the source sequence x of the text to be evaluated obtained through the above steps is {x1,…,x n} and reference text y={y1,…,y m}, by constructing an end-to-end pre-training model and inputting the text to be evaluated into the pre-training model, a first probability is obtained. Specifically, the end-to-end pre-training model is particularly suitable for generation tasks, which includes both an encoder and a decoder, and the prediction process of the pre-training model is performed by autoregression.

[0058] It can be understood that the probability of generating y under the condition of x can be decomposed, and the text frequency value p(y|x,θ) is shown in the following formula:

[0059]

[0060] Among them, θ is the end-to-end model parameter for generating text frequency values.

[0061] Step S2200: Calculate the probability of generating a reference text based on the text to be evaluated, and obtain a generation probability value.

[0062] It can be understood that by setting the generation probability value T(y t ), which is used to measure the probability of generating a reference text based on the text to be evaluated.

[0063] Step S2300: Perform weighted calculation based on the text frequency value and the generation probability value to obtain a text quality score.

[0064] It can be understood that the text quality score is obtained by weighted calculation of the text frequency value and the generation probability value. Since the text quality score is based on the formulation of the generation task and the ability to use end-to-end pre-trained parameters, the text quality score can be flexibly used in different evaluation scenarios. Specifically, the text quality score TextScore is shown in the following formula:

[0065]

[0066] In this embodiment, by calculating the weighted logarithmic probability of the reference text relative to the source sequence, different emphasis is given to the key words specified in the reference text by adjusting the weights to achieve the effect of accurate evaluation. In practical applications, the text quality score can also be calculated by different weighting methods, such as uniform weighting ignoring stop words, term frequency (TF) weighting, inverse document frequency (IDF) weighting, and using the prior probability of each key word as a weighting factor. Among them, TF weighting can fully take into account the importance of words in the actual text, not just the frequency of their occurrence.

[0067] Step S3000: Input the text to be evaluated into the trained target model, and output the text safety score through the target model based on the similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space.

[0068] It is understandable that in order to calculate the similarity between the text to be evaluated and the first quality feature space and the second quality feature space, the text to be evaluated needs to be input into the preset attention network. Specifically, the preset attention network model structure includes three parts: input encoding layer, local enhancement module and global enhancement module, which are respectively used to obtain encoding information, enhance the output of the previous stage and enhance the global semantic information.

[0069] See also Figure 5 , Figure 5FIG. 5 is a schematic diagram showing a specific implementation process of another embodiment of the above step S3000. Figure 5 As shown, step S3000 at least includes the following steps:

[0070] Step S3100: input the text to be evaluated into the bidirectional encoder of the target model to obtain encoding information.

[0071] It is understandable that in order to input the text to be evaluated into the preset attention network, it is necessary to encode the text to be evaluated. In this embodiment, a Transformer-based bidirectional encoder (BidirectionalEncoder Representations from Transformers, BERT) is used to output the encoded information of the text to be evaluated. Specifically, BERT can pre-train deep bidirectional representations by jointly adjusting the left and right contexts in all layers. Therefore, only an additional output layer is needed to fine-tune the pre-trained BERT representation, thereby creating state-of-the-art models for a wide range of tasks, such as answering questions and language inference tasks, without the need for a large number of task-specific model structure modifications.

[0072] Specifically, the encoding information b is shown in the following formula:

[0073] b=BERT(i)

[0074] Wherein, i is an input word vector based on the text to be evaluated. In some embodiments, i is formed by connecting the character embedding in the text to be evaluated and the potential word boundary with the highest similarity.

[0075] Step S3200: input the encoded information into the attention layer of the target model for deep fusion to obtain the attention layer output information.

[0076] It is understandable that due to the different roles and status of each data in the encoded information, in order for the attention network to focus on the important information in the encoded information, it is necessary to obtain the attention weight by calculating the similarity between different positions in the encoded information. The self-attention mechanism can help the model better capture the dependencies within the sequence, thereby improving the expressiveness of the model. The multi-head attention mechanism applies the attention mechanism to multiple different representation subspaces so that the model can pay attention to different information at the same time. The core idea of ​​the attention mechanism is to weight the information at different positions in the input sequence so that the model can pay more attention to the important parts. In NLP tasks, the input is usually a sequence, such as a sentence or a document. When processing these sequences, traditional models treat the information at all positions equally, while the attention mechanism obtains the weight of each position through learning, so that the model can selectively focus on the key parts. In practical applications, deep fusion of encoded information through attention layers belongs to the prior art and will not be elaborated here.

[0077] Specifically, the attention layer output information a is shown in the following formula:

[0078] a=Attention(b)=Attention(BERT(i))

[0079] Step S3300: Input the attention layer output information and the word vectors of the preset local enhancement set into the target model to obtain local enhancement information, wherein the local enhancement set is composed of word vectors that maximize the mutual information in the text to be evaluated.

[0080] It is understandable that in text classification tasks, using global information enhancement and local feature extractors is an effective method to improve the accuracy and efficiency of text recognition and classification. Text classification is a basic task in natural language processing and is widely used in fields such as spam detection, sentiment analysis, and quality assessment. Although the traditional local feature-driven model has good interpretability and excellent performance, if the local extractor cannot obtain global information from the beginning, it requires a complex and expensive upper structure to help modify the inaccurate local representation and create new high-level features. To solve this problem, an effective method is to directly optimize the local extraction process. By designing a new text classification framework, the model has a global view in the local feature extraction stage, thereby avoiding complex upper structures and improving the efficiency and accuracy of classification.

[0081] In practical applications, the preset model includes a global enhancement module and a local enhancement module. The local enhancement module is responsible for extracting local features that fuse global information. In this way, the local extractor can take advantage of its natural advantages to notice more distant information, thereby capturing better local features related to the text to be evaluated and using them directly for evaluation. The global enhancement module is responsible for modeling global information and is any neural network model that can perform global semantic representation. This approach can to a certain extent solve the problems existing in traditional methods, that is, in the case of insufficient corpus, traditional methods are inefficient and difficult to train. It can directly optimize the local extraction process and combine global information to more effectively classify and recognize text, thereby improving the accuracy and efficiency of classification and recognition.

[0082] It can be understood that by calculating the correlation between the text to be evaluated and the local enhancement set, the mutual information between the text to be evaluated and the local enhancement set is obtained. Mutual information is a concept in information theory used to measure the correlation between two random variables, which represents the amount of information contained in one random variable about another random variable, or the uncertainty of a random variable reduced due to the knowledge of another random variable. By calculating the maximum mutual information between the text to be evaluated and the local enhancement set, the word vector with the maximum mutual information between the text to be evaluated and the local enhancement set is obtained.

[0083] See also Fig. 9 , Fig. 9 The structure diagram of the target model in the above steps is shown as Fig. 9 As shown in the figure, in order to improve the accuracy and scientificity of the evaluation of the attention layer output information a, the attention layer output information a needs to be input into the multilayer perceptron and gated recurrent unit in the local enhancement module of the target model. Specifically, the multilayer perceptron (MLP) is an artificial neural network model, which is a feedforward neural network consisting of an input layer, a hidden layer and an output layer. Each neuron receives the output of the neurons in the previous layer, and processes it through a nonlinear activation function after weighted summation. MLP can handle nonlinear problems and establish a mapping relationship between input and output by learning appropriate weights and biases. It is widely used in machine learning and deep learning tasks. The gated recurrent unit (GRU) is an improved recurrent neural network structure that aims to solve the gradient vanishing and explosion problems in traditional recurrent neural networks. GRU controls the flow of information by introducing a gating mechanism, including a reset gate and an update gate, so as to better capture long-term dependencies. This structure makes GRU more effective in processing sequence data, especially in processing long sequences and tasks that require memory of long-term dependencies.

[0084] It can be understood that the goal of the local enhancement module is to enhance the attention layer output information a obtained in the above steps. First, the word boundary with the maximum point-by-point mutual information is connected according to the output of a, and then the MLP layer and the GRU layer are input in sequence. Among them, mutual information is used to measure the concept of correlation between two random variables, which represents the amount of information contained in one random variable about another random variable, or the uncertainty of a random variable reduced due to the knowledge of another random variable. Word boundary refers to the separation between words, which can help computers accurately identify and understand the vocabulary in the text, so as to perform correct processing and analysis. Specifically, the local enhancement information g is shown in the following formula:

[0085] g = GRU(MLP(a⊕e1))

[0086] Among them, e1 is the word vector that maximizes the mutual information between the text to be evaluated and the local enhancement set.

[0087] Step S3400: embedding the local enhancement information and the output of the preset global enhancement set into the target model to obtain a text embedding vector, wherein the global enhancement set consists of text information that maximizes the mutual information in the text to be evaluated.

[0088] It can be understood that by calculating the correlation between the text to be evaluated and the global enhancement set, the mutual information between the text to be evaluated and the global enhancement set is obtained. By calculating the maximum mutual information between the text to be evaluated and the global enhancement set, the text information with the maximum mutual information between the text to be evaluated and the global enhancement set is obtained.

[0089] It can be understood that after obtaining the output information g of the local enhancement module, the global enhancement module needs to enhance the global semantic information. First, the output information g of the local enhancement module is normalized and input into the MLP layer of the attention network to obtain the text embedding vector. Among them, normalization refers to the standardization of data so that its mean is 0 and its variance is 1, thereby eliminating the influence of different feature dimensions and allowing different features to be compared and calculated. For time series data, the purpose of normalization is to reduce non-stationarity so that the model can better learn the laws of the data. Specifically, the text embedding vector o is shown in the following formula:

[0090] o=MLP(LN(g)⊕e2)

[0091] Among them, e2 is the embedding that maximizes the global mutual information in the text to be evaluated.

[0092] See also Figure 6 , Figure 6 FIG. 5 is a schematic diagram showing a specific implementation process of another embodiment of the above step S3000. Figure 6As shown, step S3000 at least further includes the following steps:

[0093] Step S3500: Calculate the cosine similarity between the text embedding vector and the features of the preset first quality feature space to obtain a first similarity.

[0094] It can be understood that in order to obtain the first similarity and the second similarity, it is necessary.

[0095] See also Figure 7 , Figure 7 FIG. 5 is a schematic diagram showing a specific implementation process of another embodiment of the above step S3500. Figure 7 As shown, step S3500 at least further includes the following steps:

[0096] Step S3510: Initialize the first quality feature space and the second quality feature space.

[0097] It is understandable that in order to improve the training effect of the preset model, it is necessary to initialize the first quality feature space and the second quality feature space manually. From the perspective of security and compliance, the first quality feature space is characterized by rich knowledge and information, fluent and authentic expression. The first quality feature space has no security risks when training large language models and significantly improves its reasoning ability. Specifically, n representative high-quality texts are selected from high-quality feature subspaces such as information volume, knowledge, and fluency subspaces, each subspace containing n / k samples, and the first quality feature space is then input into the model and encoded into n embedded word vectors.

[0098] In addition, the second quality feature space includes language attacks, malicious advertisements, and content involving politics or terrorism, which all pose security risks. Training the model with these texts will result in illegal output and significantly reduce model performance. Specifically, m representative low-quality texts are selected from low-quality feature subspaces such as malicious advertisements, language attacks, pornography, and political sensitivity subspaces. Each low-quality subspace contains m / k samples. The second quality feature space is then input into the model and encoded into m embedding word vectors.

[0099] Step S3520: Generate update information of the first quality feature space and the second quality feature space according to the text safety score of the sample text.

[0100] It is understandable that by obtaining the text safety score of the sample text, the similarity between the text data in the sample text and the first quality feature space and the second quality feature space can be determined, so as to judge whether the text data in the sample text can be updated to the first quality feature space and the second quality feature space. Therefore, by presetting the similarity threshold, the text safety score of the sample text is compared with the similarity threshold to judge whether the sample text can be updated to the first quality feature space or the second quality feature space, and then the update information of the first quality feature space and the second quality feature space is obtained.

[0101] Step S3530: Update the first quality feature space and the second quality feature space according to the update information to obtain a target model.

[0102] It can be understood that after obtaining the update information, the data in the sample text can be updated to the first quality feature space or the second quality feature space based on the relationship between the text safety score of the sample text and the similarity threshold. In this process, the relevant parameters of the preset model are further improved, the training effect of the preset model is improved, and the target model is obtained.

[0103] It is understandable that after updating the first quality feature space and the second quality feature space in the target model, the first similarity HS can be determined by calculating the cosine similarity between the text embedding vector o and the first quality feature space. In other embodiments, the gap between the text embedding vector and the first quality feature space can also be measured by calculating other similarities. Specifically, the first similarity HS is shown in the following formula:

[0104]

[0105] Among them, h1,…,h n are n high-quality feature word vectors.

[0106] Step S3600: Calculate the cosine similarity between the text embedding vector and the features of the preset second quality feature space to obtain a second similarity.

[0107] It is understandable that the cosine similarity between the text embedding vector and the second quality feature space is calculated to determine the second similarity. In other embodiments, the gap between the text embedding vector and the quality sample can also be measured by calculating other similarities. Specifically, the second similarity LS is shown in the following formula:

[0108]

[0109] Among them, l1,…,l m are m low-quality feature word vectors.

[0110] Step S3700: Perform weighted calculation based on the first similarity and the second similarity to obtain a text safety score.

[0111] It can be understood that after obtaining the first similarity and the second similarity, the text security score is obtained by. Specifically, the text security score SecuScore is shown in the following formula:

[0112] SecuScore=α1·HS-α2·LS

[0113] Among them, α1 and α2 are balance coefficients of the first similarity HS and the second similarity LS respectively.

[0114] Step S4000: Perform weighted calculation based on the text quality score and the text safety score to obtain a target score for the text to be evaluated.

[0115] It is understandable that in order to make the evaluation results of the corpus data more accurately reflect the security and accuracy of the corpus data, in the process of calculating the evaluation results of the corpus data, it is necessary to combine the text quality score TextScoreTotalScore obtained in the above step S2000 and the text security score SecuScore obtained in the above step S3000. Consistent with the process of obtaining the text security score SecuScore in the above steps, the target score of the text to be evaluated can be obtained by assigning a balance coefficient to the text quality score and the text security score.

[0116] See also Figure 8 , Figure 8 FIG. 4 is a schematic diagram showing a specific implementation process of another embodiment of the above step S4000. Figure 8 As shown, step S4000 at least includes the following steps:

[0117] Step S4100: Obtain a text quality parameter according to the text quality score and a preset first balance coefficient.

[0118] It can be understood that after obtaining the text quality score TextScore, the text quality parameter is obtained by setting the first balance coefficient β1, and its calculation formula is as follows:

[0119] β1·TextScore

[0120] Step S4200: Obtain a text safety parameter according to the text safety score and a preset second balance coefficient.

[0121] It can be understood that after obtaining the text security score SecuScore, the text security parameter is obtained by setting the second balance coefficient β2, and its calculation formula is as follows:

[0122] β2·SecuScore

[0123] Step S4300: sum the text quality parameter and the text safety parameter to obtain a target score for the text to be evaluated.

[0124] It can be understood that after obtaining the text quality parameter and the text security parameter, a sum operation is performed on them, and the target score TotalScore of the text to be evaluated is shown in the following formula:

[0125] TotalScore=β1·TextScore+β2·SecuScore

[0126] Among them, β1 and β2 are the balance coefficients of the text quality score TextScore and the text security score SecuScore, respectively. In this embodiment, the text quality score TextScore is used to evaluate the quality of text content expression, while the text security score SecuScore is used to evaluate the security risk of the text. Therefore, the target score TotalScore of the text to be evaluated can provide a more comprehensive evaluation of the text quality. Specifically, the balance coefficients β1 and β2 can be flexibly adjusted according to different evaluation tasks.

[0127] For example, if only the quality of text expression is considered, β1=1 and β2=0 can be set, and TotalScore=TextScore. If the security risk of the text is considered, β1=0 and β2=1 can be set, and TotalScore=SecuScore.

[0128] In other embodiments, the text to be evaluated is also subjected to manual quality evaluation to obtain manual quality evaluation results. Through this comprehensive evaluation method, content security attributes are included, which effectively solves the defects of existing automated methods, thereby obtaining high-quality text data. This embodiment can efficiently process and analyze a large amount of text data by using advanced natural language processing and machine learning technologies, thereby greatly reducing the reliance on manual annotation and evaluation. This not only reduces labor costs, but also improves the speed and consistency of evaluation and reduces the possibility of human errors. The process of automated quality evaluation can provide real-time feedback on evaluation results, help optimize the model training process, and improve the efficiency and reliability of the overall system. At the same time, it can also handle various languages ​​and text types, expand the application scenarios, and further enhance the comprehensiveness and accuracy of data evaluation.

[0129] It is understandable that manual quality assessment includes evaluation based on samples and data types, mainly evaluating the accuracy, consistency and completeness of the text, as well as its relevance and coverage to the model training objectives. Among them, the first quality feature space should accurately reflect the required information, maintain logical consistency and orderliness, and be closely related to the objectives of model training, covering various situations and contexts that the model may encounter, to ensure that the model has good generalization ability and performance in practical applications. In practical applications, manual quality assessment belongs to the existing technology and will not be elaborated here.

[0130] See also Fig.10 , Fig.10 It is a structural diagram of the data quality assessment device 500 for fusion security attributes provided in an embodiment of the present application. The entire process of the data quality assessment method for fusion security attributes provided in an embodiment of the present application involves the following modules in the data quality assessment device for fusion security attributes: an acquisition module 510, a determination module 520, an output module 530 and an evaluation module 540.

[0131] The acquisition module 510 is used to acquire corpus data and preprocess the corpus data to obtain the text to be evaluated;

[0132] The determination module 520 is used to obtain a preset reference text and determine the text quality score according to the frequency of the key words of the reference text contained in the text to be evaluated;

[0133] An output module 530 is used to input the text to be evaluated into the trained target model, and output the text safety score through the target model based on the similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space; wherein the target model is obtained by performing similarity analysis and learning of the first quality feature and the second quality feature based on different sample texts and the text safety scores of the corresponding sample texts by a preset model;

[0134] The evaluation module 540 is used to perform weighted calculation based on the text quality score and the text safety score to obtain a target score for the text to be evaluated.

[0135] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned device are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0136] Fig.11 The electronic device 600 provided in the embodiment of the present application is shown. The electronic device 600 includes but is not limited to:

[0137] Memory 601, used for storing programs;

[0138] The processor 602 is used to execute the program stored in the memory 601. When the processor 602 executes the program stored in the memory 601, the processor 602 is used to execute the above-mentioned data quality assessment method for integrating security attributes.

[0139] The processor 602 and the memory 601 may be connected via a bus or other means.

[0140] The memory 601 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs, such as the data quality assessment method for fusion security attributes described in any embodiment of the present application. The processor 602 implements the above-mentioned data quality assessment method for fusion security attributes by running the non-transitory software programs and instructions stored in the memory 601.

[0141] The memory 601 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data quality assessment methods for executing the above-mentioned fusion security attributes. In addition, the memory 601 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 601 may optionally include a memory remotely disposed relative to the processor 602, and these remote memories may be connected to the processor 602 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The non-transient software programs and instructions required to implement the above-mentioned data quality assessment method for fusion security attributes are stored in the memory 601. When executed by one or more processors 602, the data quality assessment method for fusion security attributes provided by any embodiment of the present application is executed.

[0143] The embodiment of the present application also provides a storage medium storing computer executable instructions, where the computer executable instructions are used to execute the above-mentioned data quality assessment method for integrating security attributes.

[0144] In one embodiment, the storage medium stores computer executable instructions, which are executed by one or more control processors 602, for example, by a processor 602 in the above-mentioned electronic device 600, so that the above-mentioned one or more processors 602 can execute the data quality assessment method of fusion security attributes provided in any embodiment of the present application.

[0145] The above described embodiments are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A data quality assessment method integrating security attributes, characterized in that: The method comprises the following steps: Acquire corpus data, and preprocess the corpus data to obtain a text to be evaluated; Obtaining a preset reference text, and determining a text quality score according to the frequency of the key words of the reference text contained in the text to be evaluated; Input the text to be evaluated into the trained target model, and output the text safety score through the target model based on the similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space respectively; Wherein, the target model is obtained by performing similarity analysis and learning of the first quality feature and the second quality feature based on different sample texts and the text safety scores of the corresponding sample texts by a preset model; A weighted calculation is performed based on the text quality score and the text safety score to obtain a target score for the text to be evaluated.

2. A data quality assessment method integrating security attributes according to claim 1, characterized in that: The acquiring of corpus data and preprocessing of the corpus data to obtain a text to be evaluated includes: Performing format conversion processing on the corpus data to obtain corpus data in a unified format; Performing data cleaning on the corpus data in the unified format to obtain cleaned corpus data; The cleaned corpus data is deduplicated to obtain the text to be evaluated.

3. The data quality assessment method for integrating security attributes according to claim 1 is characterized in that: The step of determining the text quality score according to the frequency of the key words of the reference text contained in the text to be evaluated comprises: Calculate the frequency of the key words of the reference text contained in the text to be evaluated to obtain a text frequency value; Calculating the probability of generating the reference text based on the text to be evaluated to obtain a generation probability value; The text quality score is obtained by performing a weighted calculation based on the text frequency value and the generation probability value.

4. The data quality assessment method for integrating security attributes according to claim 1 is characterized in that: The step of inputting the text to be evaluated into a trained target model comprises: Inputting the text to be evaluated into the bidirectional encoder of the target model to obtain encoding information; Inputting the encoded information into the attention layer of the target model for deep fusion to obtain attention layer output information; Inputting the attention layer output information and the word vectors of the preset local enhancement set into the target model to obtain local enhancement information, wherein the local enhancement set is composed of word vectors that maximize the mutual information in the text to be evaluated; The local enhancement information and the output embedding of the preset global enhancement set are input into the target model to obtain the text embedding vector, wherein the global enhancement set is composed of text information that maximizes the mutual information in the text to be evaluated.

5. A data quality assessment method integrating security attributes according to claim 4, characterized in that: The similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space respectively based on the target model includes: Calculating a cosine similarity value between the text embedding vector and a feature of the preset first quality feature space to obtain a first similarity; Calculating a cosine similarity value between the text embedding vector and a feature of the preset second quality feature space to obtain a second similarity; The text safety score is obtained by performing a weighted calculation based on the first similarity and the second similarity.

6. A data quality assessment method integrating security attributes according to claim 5, characterized in that: The target model is obtained by performing similarity analysis and learning of the first quality feature and the second quality feature based on different sample texts and the text safety scores of the corresponding sample texts by a preset model, including: Initializing the first quality feature space and the second quality feature space; Generate update information of the first quality feature space and the second quality feature space according to the text safety score of the sample text; The first quality feature space and the second quality feature space are updated according to the update information to obtain the target model.

7. The data quality assessment method for integrating security attributes according to claim 1 is characterized in that: The step of performing weighted calculation according to the text quality score and the text safety score to obtain a target score for the text to be evaluated includes: Obtaining a text quality parameter according to the text quality score and a preset first balance coefficient; Obtaining a text safety parameter according to the text safety score and a preset second balance coefficient; The text quality parameter and the text safety parameter are summed to obtain a target score for the text to be evaluated.

8. A data quality assessment device integrating security attributes, characterized in that: include: An acquisition module is used to acquire corpus data and preprocess the corpus data to obtain a text to be evaluated; A determination module, used to obtain a preset reference text, and determine a text quality score according to the frequency of the key words of the reference text contained in the text to be evaluated; An output module is used to input the text to be evaluated into a trained target model, and output a text safety score through the target model based on the similarity between the text embedding vector of the text to be evaluated and the features in the built-in first quality feature space and second quality feature space; wherein the target model is obtained by a preset model through similarity analysis and learning of the first quality feature and the second quality feature based on different sample texts and the text safety scores of the corresponding sample texts; The evaluation module is used to perform weighted calculation according to the text quality score and the text safety score to obtain a target score for the text to be evaluated.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for evaluating the data quality of the fused security attributes as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the data quality assessment method for fusion security attributes as described in any one of claims 1 to 7 is implemented.

Images