Quality evaluation method and device of model sample, storage medium and computer equipment

Abstract

The invention discloses a quality evaluation method and device of a model sample, a storage medium and computer equipment. The method comprises the steps of inputting sample data into an artificial intelligence generation content detection model, and obtaining a hit probability of the sample data; matching a content evaluation system based on the attribute information of the sample data; processing the sample data based on an evaluation rule in the content evaluation system, and determining a test value of the sample data relative to at least one preset evaluation index; and based on a target weight corresponding to the hit probability and a preset evaluation index, carrying out weight calculation on the hit probability and the test value to obtain a quality score of the sample data. According to the method, data which is generated by AI but may mislead model training can be filtered, the purity and credibility of the sample data set are remarkably improved, multi-dimensional and high-precision evaluation of the training data can be achieved, different task requirements are met, and the training efficiency is improved. And the generalization ability of the model based on sample data training and the adaptive ability to unknown data are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, storage medium and computer equipment for evaluating the quality of a model sample. Background Art

[0002] In recent years, with the improvement of big data and computing power, deep learning models have made significant progress in various fields. However, deep learning models usually have high requirements for training data, especially in complex tasks such as natural language processing and image recognition. At present, the training of large models is increasingly dependent on large and high-quality data sets. However, the uneven quality of data has become a key factor restricting the improvement of model performance. Low-quality data can easily lead to problems such as model overfitting, bias, and insufficient generalization ability.

[0003] In the field of training data evaluation, traditional methods often rely on manual labeling or simple statistical indicators. These methods have problems such as strong subjectivity, low efficiency, and difficulty in covering all data points. Summary of the invention

[0004] In view of this, the present application provides a model sample quality assessment method, apparatus, storage medium and computer equipment, which achieves a comprehensive and accurate assessment of training sample data by combining artificial intelligence to generate content detection technology and content assessment strategy.

[0005] According to one aspect of the present application, a method for evaluating the quality of a model sample is provided, comprising:

[0006] Input the sample data into the artificial intelligence generated content detection model to obtain the hit probability of the sample data;

[0007] Matching a content evaluation system based on the attribute information of the sample data, wherein the content evaluation system includes at least one preset evaluation index and an evaluation rule of the preset evaluation index;

[0008] Processing the sample data based on the evaluation rules to determine a test value of the sample data relative to at least one preset evaluation indicator;

[0009] Based on the target weights corresponding to the hit probability and the preset evaluation indicators, the hit probability and the test value are weighted to obtain the quality score of the sample data.

[0010] Optionally, the quality assessment method of the model sample further includes:

[0011] The target large model is trained based on sample data whose quality scores are greater than the score threshold;

[0012] Input the test data into the target large model to obtain the predicted data;

[0013] Compare the real data associated with the predicted data and the test data to determine the accuracy of the target large model;

[0014] If the accuracy is less than the accuracy threshold, adjust the target weight based on the accuracy;

[0015] If the accuracy is greater than or equal to the accuracy threshold, the target large model is output.

[0016] Optionally, the quality assessment method of the model sample further includes:

[0017] If the data type of the sample data is text, obtain pre-stored data whose attribute information is in the same range as the attribute information of the sample data, wherein the quality score of the pre-stored data is greater than the score threshold;

[0018] Using text similarity algorithm to determine the feature similarity between sample data and pre-stored data;

[0019] If the feature similarity is greater than the first similarity threshold, the processing of the sample data based on the evaluation rule is canceled, and the test value of the pre-stored data is used as the test value of the sample data.

[0020] Optionally, the quality assessment method of the model sample further includes:

[0021] Obtain manually created data of the target topic as positive samples;

[0022] Input the target topic into the artificial intelligence model to obtain the intelligently generated data of the target topic as a negative sample;

[0023] The positive samples and negative samples are divided into a training set and a validation set, wherein the difference in the number of positive samples and negative samples in the training set is less than a quantity threshold;

[0024] Train the classification model based on the training set to obtain a candidate model;

[0025] Input the validation set into the candidate model to obtain the predicted probability of the validation set;

[0026] If the predicted probability of the positive sample in the validation set is less than the first preset probability, and the predicted probability of the negative sample in the validation set is greater than the second preset probability, the candidate model is confirmed as an AI-generated content detection model;

[0027] If the predicted probability of the positive sample in the validation set is greater than or equal to the first preset probability, or the predicted probability of the negative sample in the validation set is less than or equal to the second preset probability, the positive sample or negative sample in the validation set is sent to the review node;

[0028] The candidate model is trained based on the target features fed back by the review nodes to obtain an artificial intelligence-generated content detection model.

[0029] Optionally, the quality assessment method of the model sample further includes:

[0030] Perform integrity check on sample data;

[0031] If the input part or output part of the sample data is missing, delete the sample data, or supplement the missing input part or output part of the sample data based on the existing input part or output part of the sample data.

[0032] Optionally, the attribute information includes at least one of the following: data application scenario, data type, data format, word count, and memory usage;

[0033] The preset evaluation indicators include at least one of the following: grammatical correctness, vocabulary diversity, whether the image or video has a watermark, content richness, content coherence, and noise ratio.

[0034] Optionally, the data type of the sample data is text, and the preset evaluation index includes content richness, and the sample data is processed based on the evaluation rule, including:

[0035] Perform word segmentation on the sample data to determine multiple words in the sample data;

[0036] Using natural language processing algorithms, determine the semantic similarity between different words in the sample data;

[0037] Different words whose semantic similarity is greater than a second similarity threshold form a similar word set;

[0038] Count the word frequency of similar vocabulary sets, the number of similar vocabulary sets and the number of words in the sample data;

[0039] The comparison between the word frequency range, quantity range and content richness based on the word count matching of sample data;

[0040] Based on the contrast relationship, the word frequency and the word frequency range of the similar vocabulary sets, as well as the number and the number range of the similar vocabulary sets are compared respectively to determine the content richness corresponding to the word frequency of the similar vocabulary sets and the number of similar vocabulary sets.

[0041] According to another aspect of the present application, a quality assessment device for a model sample is provided, comprising:

[0042] The first detection module is used to input sample data into an artificial intelligence generated content detection model to obtain a hit probability of the sample data;

[0043] A matching module, used for matching a content evaluation system based on the attribute information of the sample data, wherein the content evaluation system includes at least one preset evaluation index and an evaluation rule of the preset evaluation index;

[0044] A second detection module, used to process the sample data based on the evaluation rules and determine a test value of the sample data relative to at least one preset evaluation index;

[0045] The evaluation module is used to perform weight calculation on the hit probability and the test value based on the target weight corresponding to the hit probability and the preset evaluation index to obtain the quality score of the sample data.

[0046] Optionally, the quality assessment device of the model sample further includes:

[0047] A first training module is used to train a target large model based on sample data whose quality scores are greater than a score threshold;

[0048] A test module, for inputting test data into the target large model to obtain predicted data; and comparing the predicted data with real data associated with the test data to determine the accuracy of the target large model;

[0049] An update module, configured to adjust the target weight based on the accuracy if the accuracy is less than an accuracy threshold;

[0050] The first training module is also used to output the target large model if the accuracy is greater than or equal to the accuracy threshold.

[0051] Optionally, the second detection module is further configured to, if the data type of the sample data is text, obtain pre-stored data whose attribute information is in the same range as the attribute information of the sample data, wherein the quality score of the pre-stored data is greater than a score threshold;

[0052] Using text similarity algorithm to determine the feature similarity between sample data and pre-stored data;

[0053] If the feature similarity is greater than the first similarity threshold, the processing of the sample data based on the evaluation rule is canceled, and the test value of the pre-stored data is used as the test value of the sample data.

[0054] Optionally, the quality assessment device of the model sample further includes:

[0055] An acquisition module is used to acquire artificially created data of a target subject as a positive sample; and, input the target subject into an artificial intelligence model to obtain intelligently generated data of the target subject as a negative sample; and, divide the positive sample and the negative sample into a training set and a validation set, wherein the difference in quantity between the positive sample and the negative sample in the training set is less than a quantity threshold;

[0056] A second training module is used to train the classification model based on the training set to obtain a candidate model; and input the verification set into the candidate model to obtain the prediction probability of the verification set; and if the prediction probability of the positive sample in the verification set is less than the first preset probability, and the prediction probability of the negative sample in the verification set is greater than the second preset probability, the candidate model is confirmed as an artificial intelligence generated content detection model;

[0057] A review module, configured to send the positive sample or negative sample in the verification set to a review node if the predicted probability of the positive sample in the verification set is greater than or equal to the first preset probability, or the predicted probability of the negative sample in the verification set is less than or equal to the second preset probability;

[0058] The second training module is also used to train the candidate model based on the target features fed back by the review node to obtain an artificial intelligence generated content detection model.

[0059] Optionally, the quality assessment device of the model sample further includes:

[0060] An integrity check module is used to check the integrity of sample data;

[0061] If the input part or output part of the sample data is missing, delete the sample data, or supplement the missing input part or output part of the sample data based on the existing input part or output part of the sample data.

[0062] Optionally, the attribute information includes at least one of the following: data application scenario, data type, data format, word count, and memory usage;

[0063] The preset evaluation indicators include at least one of the following: grammatical correctness, vocabulary diversity, whether the image or video has a watermark, content richness, content coherence, and noise ratio.

[0064] Optionally, the second detection module is specifically used to perform word segmentation processing on the sample data to determine multiple words in the sample data;

[0065] Using natural language processing algorithms, determine the semantic similarity between different words in the sample data;

[0066] Different words whose semantic similarity is greater than a second similarity threshold form a similar word set;

[0067] Count the word frequency of similar vocabulary sets, the number of similar vocabulary sets and the number of words in the sample data;

[0068] The comparison between the word frequency range, quantity range and content richness based on the word count matching of sample data;

[0069] Based on the contrast relationship, the word frequency and the word frequency range of the similar vocabulary sets, as well as the number and the number range of the similar vocabulary sets are compared respectively to determine the content richness corresponding to the word frequency of the similar vocabulary sets and the number of similar vocabulary sets.

[0070] According to another aspect of the present application, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the quality assessment method of the above-mentioned model sample are implemented.

[0071] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and the processor implements the steps of the above-mentioned model sample quality assessment method when executing the program.

[0072] By means of the above technical solution, the probability that the sample data is generated by AI is determined by a pre-trained artificial intelligence generated content detection model, so as to screen out samples with high authenticity and low AI generation suspicion. At the same time, the attribute information is used to match at least one preset evaluation indicator and its corresponding evaluation rule suitable for the sample data. According to the evaluation rules, the sample data are tested for different preset evaluation indicators, and the test value of the sample data relative to at least one preset evaluation indicator is obtained. Finally, based on the target weight corresponding to the hit probability and the preset evaluation indicator, the hit probability and the test value are weighted to complete the quality scoring of the sample data. On the one hand, by combining advanced artificial intelligence generated content (AIGC) detection technology and content innovation evaluation strategy, a comprehensive and accurate evaluation of the training data is achieved, which not only effectively filters those data generated by AI but may mislead model training, but also significantly improves the purity and credibility of the sample data set, improves the authenticity and reliability of the training data, and lays a solid foundation for subsequent model training. On the other hand, by dynamically matching the attribute information of sample data to adjust the preset evaluation indicators and evaluation rules, it has a high degree of flexibility and realizes multi-dimensional and high-precision evaluation of training data, covering key aspects such as data integrity, accuracy, diversity and innovation, and meeting different task requirements. It helps to improve the generalization ability of models trained based on sample data and their adaptability to unknown data, eliminates the workload of redeveloping the model after introducing new evaluation rules, and reduces the operating cost and difficulty of data quality assessment.

[0073] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0075] Figure 1 One of the flow charts of the quality assessment method of the model sample provided in the embodiment of the present application is shown;

[0076] Figure 2 The second flowchart of the quality assessment method of the model sample provided in the embodiment of the present application is shown;

[0077] Figure 3 The third flowchart of the quality assessment method of the model sample provided in the embodiment of the present application is shown;

[0078] Figure 4 A structural block diagram of a quality assessment device for model samples provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0079] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0080] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0081] Those skilled in the art will appreciate that, unless expressly stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, or there may be intermediate elements. In addition, the "connection" or "connection" used herein may include wireless connection or wireless fusion. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0082] Now, exemplary embodiments according to the present application will be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments set forth herein. It should be understood that these embodiments are provided to make the disclosure of the present application thorough and complete, and to fully convey the concepts of these exemplary embodiments to those of ordinary skill in the art.

[0083] In this embodiment, a quality assessment method for a model sample is provided. Figure 1 As shown, the method includes:

[0084] Step 101, inputting sample data into an artificial intelligence generated content detection model to obtain a hit probability of the sample data;

[0085] The sample data includes text data, image data, video data, voice data, etc., which are not specifically limited in the present embodiment. The greater the hit probability, the higher the possibility that the sample data is generated by AI.

[0086] Specifically, the AIGC detection model is used to learn and capture the subtle differences between AI-generated data and human-created data, and derive the hit probability that the data is generated by AI. The AIGC detection model can be obtained by training a classification model using a large number of annotated data sets (including AI-generated and human-created data).

[0087] For example, the AIGC detection AI generation hit probability of text A is 0.001, indicating that the text is very likely written by humans, and therefore can be assigned a higher content innovation value; the AIGC detection AI generation hit probability of text B is 0.957, indicating that the text is likely generated by AI, and therefore the content innovation value should be reduced accordingly.

[0088] In actual application scenarios, such as Figure 2 As shown, before step 101, the quality assessment method of the model sample further includes:

[0089] Step 201, obtaining artificial creation data of the target subject as a positive sample;

[0090] Step 202, input the target topic into the artificial intelligence model to obtain intelligently generated data of the target topic as a negative sample;

[0091] It is understandable that the data generated by AI models is relatively simple compared to manually created data. Taking text as an example, the grammatical structure of negative samples generated by AI models is fixed and the wording is uniform, while users have different writing habits and expressions, and the articles created by users are complex and varied with a variety of complex grammars.

[0092] Step 203, dividing the positive samples and negative samples into a training set and a validation set;

[0093] The training set is used to train the classification model, while the validation set is used to verify the quality and classification effect of the classification model obtained by the training. Usually, the total number of positive / negative samples collected is divided into the training set and the validation set according to a preset ratio (such as a ratio of 8:2).

[0094] It is worth mentioning that the difference in the number of positive samples and negative samples in the training set or validation set is less than the quantity threshold. In this way, the number of positive samples and negative samples used in the training model and the validation model is equivalent, so that the results of balanced model training or validation will not be biased towards one side, effectively reducing the degree of sample imbalance, thereby optimizing the model quality and helping to improve the model classification accuracy.

[0095] Step 204, training the classification model based on the training set to obtain a candidate model;

[0096] In this embodiment, a large number of labeled AI-generated negative samples and manually created positive samples are used to train a classification model, so that the classification model can learn and capture the feature differences between AI generation and human creation, such as sentence structure, semantic coherence, video length, diversity, etc., so that the trained model can better determine the possibility that the sample data is generated by AI.

[0097] Step 205, input the validation set into the candidate model to obtain the prediction probability of the validation set;

[0098] Among them, the larger the prediction probability, the higher the possibility that the samples in the validation set are generated by AI.

[0099] Step 206: If the predicted probability of the positive sample in the validation set is less than the first preset probability, and the predicted probability of the negative sample in the validation set is greater than the second preset probability, the candidate model is confirmed as an artificial intelligence generated content detection model;

[0100] Step 207: If the predicted probability of the positive sample in the validation set is greater than or equal to the first preset probability, or the predicted probability of the negative sample in the validation set is less than or equal to the second preset probability, the positive sample or negative sample in the validation set is sent to the review node;

[0101] The first preset probability is less than or equal to the second preset probability.

[0102] Step 208: Train the candidate model based on the target features fed back by the review node to obtain an artificial intelligence generated content detection model.

[0103] In this embodiment, the validation set is input into the candidate model to obtain the predicted probability of whether the sample in the validation set is generated by AI. The probability of determining that the sample is generated by AI should be small for the positive sample, and the probability of determining that the sample is generated by AI should be large for the negative sample. By comparing the predicted probability of the positive sample with the first preset probability, and the predicted probability of the negative sample with the second preset probability, it is determined whether the prediction result of the candidate model is accurate. If the predicted probability of the positive sample in the validation set is less than the first preset probability, and the predicted probability of the negative sample in the validation set is greater than the second preset probability, it means that the predicted probability of the positive and negative samples is consistent with the sample label, and the model prediction result is relatively accurate, then the candidate model is output as an artificial intelligence generated content detection model. On the contrary, if the predicted probability of the positive sample in the validation set is greater than or equal to the first preset probability, or the predicted probability of the negative sample in the validation set is less than or equal to the second preset probability, it means that the model prediction result is inaccurate, and the system sends the positive or negative samples predicted to be abnormal in the validation set to the review node. The positive or negative samples are manually reviewed by the management personnel to the review node to find the target features between human creation and artificial intelligence creation that are not easily detected in the positive or negative samples predicted to be abnormal. The system then fine-tunes the candidate model again based on the target features fed back by the review node to form an AI-generated content detection model. This can not only continuously optimize the model's prediction ability for positive and negative samples, and improve the performance and scalability of the AI-generated content detection model, but also effectively optimize the utilization efficiency of computing resources through the verification processing of positive and negative samples in the verification set, avoiding waste of resources and unnecessary computing overhead.

[0104] In one embodiment, before step 101, the quality assessment method of the model sample further includes: performing integrity check on the sample data; if the input part or output part of the sample data is missing, deleting the sample data, or supplementing the missing input part or output part of the sample data based on the existing input part or output part of the sample data.

[0105] In this embodiment, the unprocessed missing data may lead to instability in model training and affect the generalization ability of the model. To this end, integrity verification can be used to find out whether there is data or label missing in the sample data. And delete or supplement the missing data. Thereby improving the overall quality of the sample data, ensuring that the data used in model training is complete and accurate, reducing the risk of overfitting and underfitting of model training, and helping to improve the reliability of model prediction. At the same time, the process of data quality assessment can be simplified and data processing efficiency can be improved.

[0106] Furthermore, in order to improve the efficiency of subsequent data screening and model training, the system can also convert sample data from different sources into a unified format to ensure data consistency and comparability.

[0107] Step 102, matching a content evaluation system based on attribute information of the sample data;

[0108] The content evaluation system includes at least one preset evaluation indicator and an evaluation rule of the preset evaluation indicator.

[0109] In actual application scenarios, the attribute information includes at least one of the following: data application scenario, data type, data format, word count, and memory usage. The preset evaluation indicators include at least one of the following: grammatical correctness, vocabulary diversity, whether the image or video has a watermark, content richness, content coherence, and noise ratio. For example, for sample data of text type, the appropriate preset evaluation indicators matched are grammatical correctness, vocabulary diversity, content richness, and content coherence; for sample data of video type, the appropriate preset evaluation indicators matched are content richness, content coherence, whether the video has a watermark, and noise ratio. For text data with a large number of words, the detection of content richness can be omitted; compared with voice data generated by recording, the detection of noise ratio can be omitted for voice data synthesized by computer translation.

[0110] It should be noted that, for the same preset evaluation index, the evaluation rules of the preset evaluation index obtained by matching different attribute information may be the same or different. For example, regarding the detection of content coherence, text data may be detected by detecting semantic coherence, and video data may be detected by video timestamp sequence.

[0111] Step 103, processing the sample data based on the evaluation rules to determine a test value of the sample data relative to at least one preset evaluation index;

[0112] In this embodiment, the preset evaluation indicators and evaluation rules are adjusted through dynamic matching of the attribute information of the sample data, which has a high degree of flexibility and realizes multi-dimensional and high-precision evaluation of the training data, covering key aspects such as data integrity, accuracy, diversity and innovation, meeting different task requirements, eliminating the workload of redeveloping the model after introducing new evaluation rules, and reducing the operating cost and difficulty of data quality assessment.

[0113] Exemplarily, if the data type of the sample data is text, and the preset evaluation index includes content richness, the sample data is processed based on the evaluation rules in step 103, including: performing word segmentation on the sample data to determine multiple words in the sample data; using a natural language processing algorithm to determine the semantic similarity between different words in the sample data; forming a similar vocabulary set from different words whose semantic similarity is greater than a second similarity threshold; counting the word frequency of the similar vocabulary set, the number of similar vocabulary sets and the number of words in the sample data; matching the comparative relationship between the word frequency range, the number range and the content richness based on the number of words in the sample data; and comparing the word frequency and word frequency range of the similar vocabulary set, as well as the number and number range of the similar vocabulary set based on the comparative relationship, to determine the content richness corresponding to the word frequency of the similar vocabulary set and the number of similar vocabulary sets.

[0114] In this embodiment, a word segmentation tool (such as Jieba Chinese word segmentation library) is used to divide the sample data of the text type into multiple words. The words are converted into word vectors by a natural language processing (NLP) algorithm, and the semantic similarity between these word vectors is compared. When the semantic similarity of any two words is greater than the second similarity threshold, the words can be determined to be synonyms, and the words with similar semantics are aggregated to form a similar vocabulary set. With similar vocabulary sets as units, the word frequencies of all words in the similar vocabulary sets, the number of different similar vocabulary sets and the number of words in the sample data as a whole are counted. The word count of the sample data is used to dynamically match the comparison relationship between the word frequency range, the number range and the content richness, so as to avoid the misjudgment caused by the higher word frequency of the longer text due to the use of more words when testing with a unified standard. Finally, the content richness corresponding to the word frequency of the similar vocabulary set and the number of similar vocabulary sets is determined based on the corresponding relationship, so as to automatically complete the accurate content richness evaluation.

[0115] Similarly, the detection of vocabulary diversity can be determined by the number of different similar vocabulary sets or the number of different words in the same similar vocabulary set. The more different similar vocabulary sets there are, the more words with different meanings are used in the text, and the higher the vocabulary diversity. The more different words there are in the same similar vocabulary set, the more synonyms are used in the text, the more varied the language forms are, and the higher the vocabulary diversity is.

[0116] If the data type of the sample data is text, and the preset evaluation index includes grammatical correctness, the sample data is processed based on the evaluation rules in step 103, including: segmenting the sample data based on punctuation marks in the sample data to obtain multiple sentences; performing grammatical analysis on the multiple sentences using a natural language processing algorithm to determine the grammatical structure of the sentences. If the grammatical structure of the sentence is the same as the standard grammatical structure, the sentence is determined to be grammatically correct; if the grammatical structure of the sentence is different from the standard grammatical structure, the sentence is determined to be grammatically incorrect.

[0117] In this embodiment, NLP libraries (such as spaCy, NLTK, Stanford NLP, etc.) are used for grammatical analysis. The grammatical structure of the text can be identified by methods such as dependency syntactic analysis and syntax tree generation, thereby discovering potential grammatical errors.

[0118] If the data type of the sample data is an image or a video, and the preset evaluation index includes whether the image or the video has a watermark, the sample data is processed based on the evaluation rule in step 103, including: inputting the sample data into a watermark detection model to obtain a detection result that the sample data contains a watermark, wherein the watermark detection model is trained based on historical images and videos and their watermark labels.

[0119] In this embodiment, the watermark detection model can be used to detect whether sample data of image or video type contains a watermark. If the sample data contains a watermark, it can be determined that the sample data is likely to be generated by AI with a high probability.

[0120] In one embodiment, before step 103, the quality assessment method of the model sample further includes: if the data type of the sample data is text, obtaining pre-stored data whose attribute information is in the same range as the attribute information of the sample data; using a text similarity algorithm to determine the feature similarity between the sample data and the pre-stored data; if the feature similarity is greater than a first similarity threshold, canceling processing of the sample data based on the evaluation rule, and using the test value of the pre-stored data as the test value of the sample data.

[0121] The quality score of the pre-stored data is greater than the score threshold. The score threshold and the first similarity threshold can be reasonably set according to the detection accuracy and experience.

[0122] In this embodiment, for sample data of text type, the attribute information of the current sample data to be detected is compared with the attribute information of the screened high-quality pre-stored data. If the attribute information of the two is in the same range, that is, the sample data and the pre-stored data are of the same type and can learn from each other. The feature similarity between the sample data and the pre-stored data is further calculated by a text similarity algorithm. If the feature similarity is greater than the first similarity threshold, it means that the semantics, grammar and other features of the sample data and the pre-stored data are similar, then the processing of the sample data based on the evaluation rules is cancelled, and the test value of the pre-stored data is directly used as the test value of the sample data. Thereby, repetitive content evaluation of similar data with high similarity is omitted, further reducing the waste of resources and unnecessary computing overhead, greatly improving the efficiency of the system in evaluating the quality of model samples, and facilitating the realization of batch data screening function.

[0123] Furthermore, if the feature similarity is less than the third similarity threshold, the sample data with the feature similarity less than the third similarity threshold is deleted. Thus, by comparing with high-quality data, low-value or abnormal sample data can be identified, and low-value or abnormal sample data can be filtered out in a targeted manner to improve the quality of sample data. Among them, the third similarity threshold is much smaller than the first similarity threshold.

[0124] Specifically, the text similarity algorithm may be a cosine similarity algorithm, a Jaccard similarity algorithm, a Manhattan distance algorithm, etc., which is not specifically limited in the embodiments of the present application.

[0125] Step 104, based on the target weight corresponding to the hit probability and the preset evaluation index, weight calculation is performed on the hit probability and the test value to obtain the quality score of the sample data.

[0126] The quality assessment method of the model sample provided in the embodiment of the present application determines the probability that the sample data is generated by AI through a pre-trained artificial intelligence generated content detection model, so as to screen out samples with high authenticity and low AI generation suspicion. At the same time, the attribute information is used to match at least one preset evaluation index and its corresponding evaluation rule suitable for the sample data. According to the evaluation rules, the sample data are tested for different preset evaluation indicators, and the test value of the sample data relative to at least one preset evaluation indicator is obtained. Finally, based on the target weight corresponding to the hit probability and the preset evaluation index, the hit probability and the test value are weighted to complete the quality scoring of the sample data. On the one hand, by combining advanced artificial intelligence generated content detection technology and content innovation evaluation strategy, a comprehensive and accurate evaluation of the training data is achieved, which not only effectively filters those data generated by AI but may mislead model training, but also significantly improves the purity and credibility of the sample data set, improves the authenticity and reliability of the sample data required for training, and lays a solid foundation for subsequent model training, and then uses high-quality sample data to train the model to achieve better prediction or classification effects on specific tasks. On the other hand, by dynamically matching the attribute information of sample data to adjust the preset evaluation indicators and evaluation rules, it has a high degree of flexibility and realizes multi-dimensional and high-precision evaluation of training data, covering key aspects such as data integrity, accuracy, diversity and innovation, and meeting different task requirements. It helps to improve the generalization ability of models trained based on sample data and their adaptability to unknown data, eliminates the workload of redeveloping the model after introducing new evaluation rules, and reduces the operating cost and difficulty of data quality assessment.

[0127] It is understandable that a quality report of sample data is generated based on the test values ​​of sample data under various evaluation indicators and the hit probability generated by AI. Users can intuitively obtain the quality of sample data through the quality report, and perceive sample quality problems at different stages, which plays a certain guiding role in improving data quality. In addition, users can also continuously adjust and optimize the parameters and algorithms of the evaluation system based on the data problems pointed out in the quality report and their impact on model performance, and introduce new evaluation dimensions and indicators to more comprehensively evaluate the quality of sample data.

[0128] In actual application scenarios, the target weight can be determined based on the prediction results of the target large model, which is trained by sample data with quality scores greater than the score threshold. Thus, the closed-loop feedback and adjustment mechanism ensures the close linkage between data screening and model training, which not only improves the overall performance of the trained model, but also enhances the adaptability and robustness of the data screening mechanism.

[0129] Specifically, Figure 3 As shown, the quality assessment method of the model sample also includes:

[0130] Step 301, training a target large model based on sample data whose quality scores are greater than a score threshold;

[0131] Step 302, input the test data into the target large model to obtain the predicted data;

[0132] Step 303, comparing the real data associated with the predicted data and the test data to determine the accuracy of the target large model;

[0133] It is understandable that if the target large model is also a generative model, an artificial intelligence-generated content detection model can also be used to detect the predicted data. By determining whether the predicted data is AI-generated, its innovation, logic, and difference from the sample data can be quantified, and the accuracy of the target large model can be determined.

[0134] Step 304: If the accuracy is less than the accuracy threshold, adjust the target weight based on the accuracy.

[0135] Step 305: If the accuracy is greater than or equal to the accuracy threshold, the target large model is output.

[0136] Among them, the accuracy threshold can be reasonably set according to the training accuracy required by the user.

[0137] In this embodiment, when it is detected that the quality score of the sample data is greater than the score threshold, it means that the sample data has a higher quality, and the sample data is used to train the target large model. The test data is input into the trained target large model to obtain the predicted data. The accuracy of the trained target large model is determined by comparing the difference between the predicted data and the real data associated with the test data. If the accuracy of the target large model is less than the accuracy threshold, it can be determined that there is an abnormality in the data screening, for example, the standard is too strict or too wide, resulting in the target large model prediction or classification effect not meeting the standard. At this time, the accuracy is used to adjust the hit probability and the target weights of different preset evaluation indicators when screening high-quality sample data. Thereby optimizing the rules of the data screening mechanism, so that the data screening mechanism can continuously adapt to new data requirements, so as to more comprehensively evaluate the quality of the sample data, and enhance the adaptability and robustness of the data screening mechanism.

[0138] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0139] The quality assessment method of the model sample provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the quality assessment method of the model sample, etc., but is not limited to the above forms.

[0140] Furthermore, if Figure 4 As shown, as a specific implementation of the above-mentioned model sample quality assessment method, an embodiment of the present application provides a model sample quality assessment device 400, and the model sample quality assessment device 400 includes: a first detection module 401, a matching module 402, a second detection module 403 and an assessment module 404.

[0141] The first detection module 401 is used to input sample data into an artificial intelligence generated content detection model to obtain a hit probability of the sample data;

[0142] A matching module 402, configured to match a content evaluation system based on the attribute information of the sample data, wherein the content evaluation system includes at least one preset evaluation index and an evaluation rule of the preset evaluation index;

[0143] The second detection module 403 is used to process the sample data based on the evaluation rules to determine the test value of the sample data relative to at least one preset evaluation index;

[0144] The evaluation module 404 is used to perform weight calculation on the hit probability and the test value based on the target weight corresponding to the hit probability and the preset evaluation index to obtain the quality score of the sample data.

[0145] In this embodiment, the probability that the sample data is generated by AI is determined by a pre-trained artificial intelligence generated content detection model, so as to screen out samples with high authenticity and low AI generation suspicion. At the same time, the attribute information is used to match at least one preset evaluation indicator and its corresponding evaluation rule suitable for the sample data. According to the evaluation rules, the sample data are subjected to relevant tests of different preset evaluation indicators, and the test value of the sample data relative to at least one preset evaluation indicator is obtained. Finally, based on the target weight corresponding to the hit probability and the preset evaluation indicator, the hit probability and the test value are weighted to complete the quality scoring of the sample data. On the one hand, by combining advanced artificial intelligence generated content detection technology and content innovation evaluation strategy, a comprehensive and accurate evaluation of the training data is achieved, which not only effectively filters those data generated by AI but may mislead model training, but also significantly improves the purity and credibility of the sample data set, improves the authenticity and reliability of the training data, and lays a solid foundation for subsequent model training. On the other hand, by dynamically matching the attribute information of sample data to adjust the preset evaluation indicators and evaluation rules, it has a high degree of flexibility and realizes multi-dimensional and high-precision evaluation of training data, covering key aspects such as data integrity, accuracy, diversity and innovation, and meeting different task requirements. It helps to improve the generalization ability of models trained based on sample data and their adaptability to unknown data, eliminates the workload of redeveloping the model after introducing new evaluation rules, and reduces the operating cost and difficulty of data quality assessment.

[0146] Furthermore, the quality assessment device 400 of the model sample further includes: a first training module (not shown in the figure), a testing module (not shown in the figure) and an updating module (not shown in the figure);

[0147] The first training module is used to train the target large model based on sample data whose quality score is greater than the score threshold;

[0148] A test module, for inputting test data into the target large model to obtain predicted data; and comparing the predicted data with real data associated with the test data to determine the accuracy of the target large model;

[0149] An update module, configured to adjust the target weight based on the accuracy if the accuracy is less than an accuracy threshold;

[0150] The first training module is also used to output the target large model if the accuracy is greater than or equal to the accuracy threshold.

[0151] Furthermore, the second detection module 403 is also used to obtain pre-stored data whose attribute information is within the same range as the attribute information of the sample data if the data type of the sample data is text, wherein the quality score of the pre-stored data is greater than a score threshold; use a text similarity algorithm to determine the feature similarity between the sample data and the pre-stored data; if the feature similarity is greater than a first similarity threshold, cancel processing of the sample data based on the evaluation rule, and use the test value of the pre-stored data as the test value of the sample data.

[0152] Furthermore, the quality assessment device 400 of the model sample further includes: an acquisition module (not shown in the figure), a second training module (not shown in the figure) and a review module (not shown in the figure);

[0153] The acquisition module is used to acquire artificially created data of the target subject as a positive sample; and the target subject is input into the artificial intelligence model to obtain intelligently generated data of the target subject as a negative sample; and the positive sample and the negative sample are divided into a training set and a validation set, wherein the difference in quantity between the positive sample and the negative sample in the training set is less than a quantity threshold;

[0154] A second training module is used to train the classification model based on the training set to obtain a candidate model; and input the verification set into the candidate model to obtain the prediction probability of the verification set; and if the prediction probability of the positive sample in the verification set is less than the first preset probability, and the prediction probability of the negative sample in the verification set is greater than the second preset probability, the candidate model is confirmed as an artificial intelligence generated content detection model;

[0155] A review module, configured to send the positive sample or negative sample in the verification set to a review node if the predicted probability of the positive sample in the verification set is greater than or equal to the first preset probability, or the predicted probability of the negative sample in the verification set is less than or equal to the second preset probability;

[0156] The second training module is also used to train the candidate model based on the target features fed back by the review node to obtain an artificial intelligence generated content detection model.

[0157] Furthermore, the model sample quality assessment device 400 further includes: an integrity verification module (not shown in the figure);

[0158] Among them, the integrity check module is used to perform integrity check on the sample data; if the input part or output part of the sample data is missing, the sample data is deleted, or the missing input part or output part of the sample data is supplemented based on the existing input part or output part of the sample data.

[0159] Furthermore, the attribute information includes at least one of the following: data application scenario, data type, data format, number of words, and memory usage; the preset evaluation indicators include at least one of the following: grammatical correctness, vocabulary diversity, whether the image or video has a watermark, content richness, content coherence, and noise ratio.

[0160] Furthermore, the second detection module 403 is specifically used to perform word segmentation processing on the sample data to determine multiple words in the sample data; use a natural language processing algorithm to determine the semantic similarity between different words in the sample data; group different words whose semantic similarity is greater than a second similarity threshold into a similar word set; count the word frequency of the similar word set, the number of similar word sets and the number of words in the sample data; match the comparative relationship between the word frequency range, the number range and the content richness based on the number of words in the sample data; compare the word frequency and word frequency range of the similar word set, as well as the number and number range of the similar word set based on the comparative relationship, and determine the content richness corresponding to the word frequency of the similar word set and the number of similar word sets.

[0161] For the specific definition of the quality assessment device for model samples, please refer to the definition of the quality assessment method for model samples above, which will not be repeated here. Each module in the above-mentioned quality assessment device for model samples can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0162] Based on the above Figures 1 to 3 The method shown in the embodiment of the present application accordingly provides a readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned Figures 1 to 3 Quality assessment method for the model samples shown.

[0163] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0164] Based on the above Figures 1 to 3 The method shown, and Figure 4In order to achieve the above-mentioned purpose, the embodiment of the present application further provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figures 1 to 3 Quality assessment method for the model samples shown.

[0165] Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0166] Those skilled in the art will appreciate that the computer device structure provided in this embodiment does not limit the computer device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0167] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and saves the hardware and software resources of the computer device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the physical device.

[0168] Through the description of the above implementation methods, the technicians in this field can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. The sample data can be input into the artificial intelligence generated content detection model to obtain the hit probability of the sample data; the content evaluation system is matched based on the attribute information of the sample data, wherein the content evaluation system includes at least one preset evaluation index and the evaluation rule of the preset evaluation index; the sample data is processed based on the evaluation rule to determine the test value of the sample data relative to at least one preset evaluation index; based on the target weight corresponding to the hit probability and the preset evaluation index, the hit probability and the test value are weighted to obtain the quality score of the sample data. In the embodiment of the present application, the probability that the sample data is generated by AI is determined by a pre-trained artificial intelligence generated content detection model, so as to screen out samples with high authenticity and low AI generation suspicion. At the same time, the attribute information is used to match at least one preset evaluation index and its corresponding evaluation rule suitable for the sample data. According to the evaluation rules, the sample data is subjected to relevant tests of different preset evaluation indicators, and the test value of the sample data relative to at least one preset evaluation index is obtained. Finally, based on the target weight corresponding to the hit probability and the preset evaluation index, the hit probability and the test value are weighted to complete the quality score of the sample data. On the one hand, by combining advanced AI-generated content detection technology and content innovation evaluation strategies, a comprehensive and accurate evaluation of training data is achieved, which not only effectively filters out data generated by AI but may mislead model training, but also significantly improves the purity and credibility of sample data sets, improves the authenticity and reliability of training data, and lays a solid foundation for subsequent model training. On the other hand, by dynamically matching the attribute information of sample data to adjust the preset evaluation indicators and evaluation rules, it has a high degree of flexibility and realizes a multi-dimensional and high-precision evaluation of training data, covering key aspects such as data integrity, accuracy, diversity and innovation, meeting different task requirements, and helping to improve the generalization ability of models trained based on sample data and their adaptability to unknown data, eliminating the workload of redeveloping models after introducing new evaluation rules, and reducing the operating cost and difficulty of data quality evaluation.

[0169] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0170] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A quality assessment method for a model sample, characterized in that: The method comprises: Inputting the sample data into an artificial intelligence generated content detection model to obtain a hit probability of the sample data; Matching a content evaluation system based on the attribute information of the sample data, wherein the content evaluation system includes at least one preset evaluation indicator and an evaluation rule of the preset evaluation indicator; Processing the sample data based on the evaluation rule to determine a test value of the sample data relative to at least one of the preset evaluation indicators; Based on the target weight corresponding to the hit probability and the preset evaluation index, the hit probability and the test value are weighted to obtain a quality score of the sample data.

2. The quality assessment method of model samples according to claim 1, characterized in that: The method further comprises: If the data type of the sample data is text, obtaining pre-stored data whose attribute information is in the same range as the attribute information of the sample data, wherein the quality score of the pre-stored data is greater than a score threshold; Using a text similarity algorithm to determine feature similarity between the sample data and the pre-stored data; If the feature similarity is greater than a first similarity threshold, the processing of the sample data based on the evaluation rule is canceled, and the test value of the pre-stored data is used as the test value of the sample data.

3. The quality assessment method of model samples according to claim 1, characterized in that: The method further comprises: Obtain manually created data of the target topic as positive samples; Inputting the target subject into an artificial intelligence model to obtain intelligently generated data of the target subject as a negative sample; Dividing the positive samples and the negative samples into a training set and a validation set, wherein the quantity difference between the positive samples and the negative samples in the training set is less than a quantity threshold; Training the classification model based on the training set to obtain a candidate model; Inputting the validation set into the candidate model to obtain the predicted probability of the validation set; If the predicted probability of the positive sample in the verification set is less than the first preset probability, and the predicted probability of the negative sample in the verification set is greater than the second preset probability, the candidate model is confirmed as the artificial intelligence generated content detection model; If the predicted probability of the positive sample in the verification set is greater than or equal to the first preset probability, or the predicted probability of the negative sample in the verification set is less than or equal to the second preset probability, the positive sample or negative sample in the verification set is sent to the review node; The candidate model is trained based on the target features fed back by the review node to obtain the artificial intelligence generated content detection model.

4. The quality assessment method of model samples according to claim 1, characterized in that: The method further comprises: Training a target large model based on the sample data whose quality score is greater than a score threshold; Inputting the test data into the target macro model to obtain prediction data; Comparing the predicted data with real data associated with the test data to determine the accuracy of the target macro model; If the accuracy is less than an accuracy threshold, adjusting the target weight based on the accuracy; If the accuracy is greater than or equal to the accuracy threshold, the target large model is output.

5. The quality assessment method of a model sample according to any one of claims 1 to 4, characterized in that: The method further comprises: Performing integrity check on the sample data; If the input part or the output part of the sample data is missing, the sample data is deleted, or the missing input part or the output part of the sample data is supplemented based on the existing input part or the output part of the sample data.

6. The quality assessment method of a model sample according to any one of claims 1 to 4, characterized in that: The attribute information includes at least one of the following: data application scenario, data type, data format, word count, and memory usage; The preset evaluation index includes at least one of the following: grammatical correctness, vocabulary diversity, whether the image or video has a watermark, content richness, content coherence, and noise ratio.

7. The method for evaluating the quality of a model sample according to claim 6, characterized in that: The data type of the sample data is text, and the preset evaluation index includes content richness, and the processing of the sample data based on the evaluation rule includes: Performing word segmentation processing on the sample data to determine a plurality of words in the sample data; Using a natural language processing algorithm, determining the semantic similarity between different words in the sample data; The different words whose semantic similarity is greater than a second similarity threshold form a similar word set; Counting the word frequency of the similar vocabulary set, the number of the similar vocabulary sets and the number of words in the sample data; Matching the word frequency range, quantity range and content richness based on the word count of the sample data; Based on the contrast relationship, the word frequency and the word frequency range of the similar vocabulary set, and the number and the number range of the similar vocabulary set are compared respectively to determine the content richness corresponding to the word frequency of the similar vocabulary set and the number of the similar vocabulary set.

8. A quality assessment device for a model sample, characterized in that: The device comprises: A first detection module, used to input sample data into an artificial intelligence generated content detection model to obtain a hit probability of the sample data; A matching module, used for matching a content evaluation system based on the attribute information of the sample data, wherein the content evaluation system includes at least one preset evaluation index and an evaluation rule of the preset evaluation index; A second detection module, configured to process the sample data based on the evaluation rule to determine a test value of the sample data relative to at least one of the preset evaluation indicators; An evaluation module is used to perform weight calculation on the hit probability and the test value based on a target weight corresponding to the hit probability and the preset evaluation index to obtain a quality score of the sample data.

9. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the quality assessment method of the model sample according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the program, the quality assessment method of the model sample according to any one of claims 1 to 7 is implemented.

Images