Text discrimination method and device, electronic equipment, readable storage medium and chip

Abstract

Embodiments of the present application provide a text discrimination method, device, electronic equipment, readable storage medium and chip, wherein the text discrimination method comprises: determining a reply text made by a first user to a question raised by a second user; performing data cleaning on the reply text to determine preprocessed data; determining a historical database; determining a plurality of labeled samples in the historical database; determining at least one unlabeled sample in the preprocessed data; labeling the unlabeled sample in the preprocessed data according to the labeled samples to determine labeled data; performing sampling evaluation on the labeled data to determine mined data; training a deep learning model according to the mined data; inputting the reply text into the deep learning model to determine a discrimination result corresponding to the reply text; wherein the discrimination result is used to indicate whether the reply text is associated with a data source corresponding to the historical database. Through the scheme of the present application, the robustness of artificial intelligence generated text discrimination in an interview scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a text discrimination method and device, electronic equipment, readable storage medium and chip. BACKGROUND

[0002] Under the background of rapid development of current artificial intelligence (Artificial Intelligence, AI) large language model, the response speed of its text generation is fast, the semantic quality is high, and the style is natural, which leads to the fact that in online interview, written interview and other actual business scenarios, some interviewees generate content by means of large model to pass the examination in a "cheating" way. This kind of behavior seriously affects the judgment of the real ability of the candidates by enterprises, and interferes with the normal screening process.

[0003] The existing identification means mainly rely on manual verification, which is not only inefficient and subjective, but also difficult to expand in the face of large-scale business scenarios, with low recall rate and high cost. Therefore, the existing technology cannot meet the actual needs of efficiently and accurately identifying AI generated text. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a text discrimination method, device, electronic equipment, readable storage medium and chip, which can solve the problems of low efficiency and strong subjectivity of AI generated text discrimination.

[0005] Therefore, the embodiments of the first aspect of the present application provide a text discrimination method.

[0006] The embodiments of the second aspect of the present application provide a text discrimination device.

[0007] The embodiments of the third aspect of the present application provide an electronic equipment.

[0008] The embodiments of the fourth aspect of the present application provide a readable storage medium.

[0009] The embodiments of the fifth aspect of the present application provide a chip.

[0010] To achieve the above object, the embodiment of the first aspect of the present application provides a text discrimination method, comprising: determining original text data associated with a first user and a second user, the original text data at least including reply text made by the first user to a question raised by the second user; performing data cleaning on the reply text to determine preprocessed data; determining a historical database; determining a plurality of labeled samples in the historical database; determining at least one unlabeled sample in the preprocessed data; labeling the unlabeled sample in the preprocessed data according to the labeled samples to determine labeled data; performing sampling evaluation on the labeled data to determine mined data; training a deep learning model according to the mined data; inputting the reply text into the deep learning model to determine a discrimination result corresponding to the reply text; wherein the discrimination result is used to represent whether the reply text is associated with a data source corresponding to the historical database.

[0011] The text discrimination method provided by the present application analyzes the reply text in the original text data corresponding to the second user of the first user, judges whether the content in the reply text is associated with the data source corresponding to the historical database, and outputs the corresponding judgment result. The original text data corresponding to the second user of the first user includes a plurality of original question texts of the first user to the second user and reply texts corresponding to the original question texts. The reply text includes the answers of the first user to each original question text of the second user. Through data preparation and preprocessing, preprocessed data corresponding to the reply text is determined. At least one unlabeled sample, i.e. a plurality of character strings or text contents, in the preprocessed data is determined. The unlabeled sample is labeled according to a plurality of labeled samples in the historical database to determine a plurality of mined data in the preprocessed data. The mined data all have labels and are used to distinguish whether they are associated with the data source corresponding to the historical database.

[0012] The plurality of labeled data associated with the data source corresponding to the historical database in the mined data and the plurality of labeled data not associated with the data source corresponding to the historical database in the mined data are used as a training set to train a deep learning model. After learning the training set, the deep learning model obtains the interaction information between each character in the reply text and other characters through a self-attention mechanism, judges the association degree between the plurality of labeled data associated with the data source corresponding to the historical database in the reply text through the interaction information, and determines the discrimination result of the reply text according to whether the reply text is associated with the data source corresponding to the historical database according to the association degree.

[0013] It can be understood that the deep learning model is automatically trained through historical labeled samples and automatic data mining and evaluation, the cost of manual labeling is reduced, the error caused by manual review is reduced, the accuracy and robustness of text discrimination are improved, and the false negative rate is greatly reduced to meet the discrimination requirements of text data.

[0014] In some embodiments, the unannotated samples in the preprocessed data are annotated according to the annotated samples to determine the annotated data, including: determining original question text corresponding to the reply text in the original text data; extracting key vocabularies from the preprocessed data to determine a key vocabulary set; determining a semantic correlation degree of the key vocabulary set and the original question text; determining a preset correlation threshold; annotating the unannotated samples according to the semantic correlation degree and the preset correlation threshold to determine the annotated data; wherein, when the semantic correlation degree is greater than or equal to the preset correlation threshold, a first label is annotated for the unannotated sample; and when the semantic correlation degree is less than the preset correlation threshold, a second label is annotated for the unannotated sample.

[0015] In the present scheme, the important vocabularies corresponding to the original question text in the reply text are extracted by the important word model, and the unannotated samples with a semantic correlation degree greater than a preset correlation threshold are queried and labeled in a plurality of unannotated samples. The label is used to reflect whether the reply text is generated by an artificial intelligence model.

[0016] The label includes a first label and a second label, and the semantics of the first label and the second label are opposite. The first label indicates that the semantic correlation degree between the reply text and the original question text is greater than or equal to the preset correlation threshold, that is, the reply content corresponding to the reply text is associated with the question in the original question text, and the reply text is not generated by an artificial intelligence model. The second label indicates that the semantic correlation degree between the reply text and the original question text is less than the preset correlation threshold, that is, the reply content corresponding to the reply text is irrelevant to the question in the original question text, and the reply text is generated by an artificial intelligence model.

[0017] It can be understood that, according to the characteristics of the AI large model in generating answers, the AI large model tends to generate content with universality and strong generalization. Therefore, the key vocabularies of the reply text are extracted, the logical relevance between the question text and the reply text is judged, and the generalization ability and robustness of the text discrimination in the interview scene are improved.

[0018] In some embodiments, the unannotated samples in the preprocessed data are annotated according to the annotated samples to determine the annotated data, further including: determining a text similarity model; inputting the annotated sample and at least one unannotated sample into the text similarity model to determine the text similarity of the annotated sample and the unannotated sample; determining a similarity threshold according to the historical database; annotating the unannotated sample according to the text similarity and the similarity threshold to determine the annotated data; wherein, when the text similarity is greater than or equal to the similarity threshold, the second label is annotated for the unannotated sample; and when the text similarity is less than the similarity threshold, the first label is annotated for the unannotated sample.

[0019] In the scheme, the similarity calculation model is used to calculate the text similarity of the labeled samples in the historical database and the unlabeled samples in the preprocessed data. By comparing the text similarity with the similarity threshold, the unlabeled samples are labeled.

[0020] Understandably, the artificial intelligence model will mark certain patterns or tend to use specific templates or sentence patterns when generating answers, especially for common questions. By batch identifying text highly similar to known AI-generated samples, i.e., labeled samples, the unlabeled samples are labeled, making the mined data more robust. Moreover, by determining the label through the similarity value, the discrimination result is more convincing and interpretable, facilitating manual review. For example, the reply text is highly similar to a known AI-generated answer (similarity of 92%).

[0021] In some technical solutions, the unlabeled samples in the preprocessed data are labeled according to the labeled samples to determine the labeled data, which further includes: determining a plurality of character strings corresponding to the unlabeled samples; determining the reply text corresponding to the character strings; determining the character type of the reply text corresponding to the character strings; replacing the character strings according to the character type to determine the replacement samples; and determining the labeled data according to the replacement samples.

[0022] In the scheme, labeling the unlabeled samples in the preprocessed data further includes replacing the entity words of the unlabeled samples to construct new samples, determining the replacement samples, and determining the labeled data corresponding to the replacement samples.

[0023] In some technical solutions, the labeled data is sampled and evaluated to determine the mined data, including: randomly sampling the plurality of labeled data to determine the evaluation sample set; determining the labeling completeness parameter of each labeled data in the evaluation sample set; determining the quality threshold; when the labeling completeness parameter is greater than or equal to the quality threshold, determining the labeled data as the mined data; when the labeling completeness parameter is less than the quality threshold, randomly sampling the plurality of labeled data to determine the evaluation sample set again until the labeling completeness parameter is greater than or equal to the quality threshold.

[0024] In the scheme, after determining the labeled data, the preprocessed data labeled with the new label is automatically verified. The labeling completeness parameter of each labeled data is compared with the quality threshold. If it does not meet the standard, i.e., the labeling completeness parameter is less than the quality threshold, the mining rule is further optimized, the unlabeled samples are labeled again to determine new labeled data until the labeling completeness parameter is greater than or equal to the quality threshold. If it meets the standard, i.e., the labeling completeness parameter is greater than or equal to the quality threshold, the labeled data is determined as the mined data, which is used as the training set of the deep learning model for training.

[0025] In some embodiments, training the deep learning model according to the mined data comprises: determining an initial deep learning model; dividing the mined data into a training set and a validation set; training the initial deep learning model according to the training set to determine a training learning model; verifying the model performance of the deep learning model according to the validation set to determine a verification result; when the accuracy corresponding to the verification result reaches a preset target, determining the deep learning model according to the training learning model; and when the accuracy corresponding to the verification result does not reach the preset target, re-dividing the training set and the validation set.

[0026] In this embodiment, the deep learning model is trained by mining data, and the trained deep learning model is used to determine whether the reply text of the first user is generated by AI.

[0027] The embodiment of the second aspect of the application provides a text determination device, which comprises: a data acquisition module configured to determine original text data associated with a first user and a second user, the original text data comprising at least a reply text made by the first user in response to a question raised by the second user; a preprocessing module configured to perform data cleaning on the reply text to determine preprocessed data; a data source module configured to determine a historical database; a sample determination module configured to determine a plurality of labeled samples in the historical database; a label recognition module configured to determine at least one unlabeled sample in the preprocessed data; a data labeling module configured to label the unlabeled sample in the preprocessed data according to the labeled sample to determine labeled data; a data mining module configured to sample and evaluate the labeled data to determine mined data; a model training module configured to train a deep learning model according to the mined data; and a text determination module configured to input the reply text into the deep learning model to determine a determination result corresponding to the reply text, wherein the determination result is used to indicate whether the reply text is associated with a data source corresponding to the historical database.

[0028] The embodiment of the third aspect of the application provides an electronic device, which comprises a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the text determination method of the first aspect.

[0029] The embodiment of the fourth aspect of the application provides a readable storage medium, and the readable storage medium stores a program or instructions, and the program or instructions are executed by the processor to implement the steps of the text determination method of the first aspect.

[0030] The embodiment of the fifth aspect of the application provides a chip, which comprises a processor and a communication interface, the communication interface and the processor are coupled, the processor is used to run a program or instructions, and the steps of the text determination method of the first aspect are implemented.

[0031] The additional aspects and advantages of the technical solutions of the present application will become apparent in the following description part or can be learned through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flowchart of a text discrimination method according to one embodiment of the present application is shown;

[0033] Figure 2 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown;

[0034] Figure 3 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown;

[0035] Figure 4 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown;

[0036] Figure 5 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown;

[0037] Figure 6 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown;

[0038] Figure 7 A structural schematic block diagram of an electronic device according to one embodiment of the present application is shown;

[0039] Figure 8 A structural schematic block diagram of a text discrimination device according to one embodiment of the present application is shown;

[0040] Figure 9 A flowchart of a text discrimination method according to one embodiment of the present application is shown;

[0041] Figure 10 A flowchart of data mining according to one embodiment of the present application is shown.

[0042] Wherein, Figure 7 And Figure 8 The correspondence between the reference signs and the component names in the drawings is as follows:

[0043] 1000: electronic device; 1109: memory; 1110: processor; 900: text discrimination device; 902: data acquisition module; 904: preprocessing module; 906: data source module; 908: sample determination module; 910: label recognition module; 912: data labeling module; 914: data mining module; 916: model training module; 918: text discrimination module. DETAILED DESCRIPTION

[0044] To better understand the above-described objectives, features, and advantages of the embodiments of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, embodiments of the invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0046] Among related technologies, techniques for identifying suspected AI-generated text mainly fall into the following two categories:

[0047] I. Keyword matching and manual verification method: This method typically involves screening text using a pre-set keyword database and combining it with manual review to determine whether the text was generated by AI. This approach relies on empirical rules and human judgment, resulting in low efficiency and significant processing bottlenecks when dealing with large-scale text data.

[0048] II. Machine Learning Model-Based Approaches: These methods collect large amounts of manually labeled data and train classification models (such as support vector machines, random forests, and deep neural networks) to distinguish between AI-generated text and human-written text. While machine learning model-based approaches introduce automation, they still face many challenges in practical applications.

[0049] However, both methods have the following drawbacks: strong sample dependence, with machine learning models heavily reliant on high-quality, large-scale manually labeled data, resulting in high data collection and labeling costs and difficulty in continuous iteration; weak generalization ability, with the model's recognition ability declining in texts outside the training corpus, especially in scenarios with diverse language styles and domain backgrounds, leading to insufficient accuracy; low recall, in practical use, the system cannot effectively identify all suspected AI-generated text, resulting in a high false negative rate, making it difficult to meet the needs of scenarios with high text authenticity requirements, such as interviews and exams; and difficulty in handling variant expressions, as AI text generation technology continues to evolve (such as the widespread adoption of large models like ChatGPT and Claude), AI-generated text is becoming increasingly closer to human language styles, gradually weakening the recognition ability of traditional methods.

[0050] Therefore, the existing technologies still have many shortcomings in identifying suspected AI-generated text. A new method with greater robustness, high recall, and low human dependence is needed to meet the requirements for judging the authenticity and originality of text in interview scenarios.

[0051] The following is in conjunction with the appendix Figures 1 to 10The text discrimination method, device, electronic device, readable storage medium and chip provided by the embodiments of the present application are described in detail through specific embodiments and application scenarios.

[0052] As shown in Figure 1 The present embodiment provides a text discrimination method, which comprises the following steps:

[0053] Step S100: determining original text data associated with a first user and a second user, wherein the original text data at least comprises reply text made by the first user in response to a question raised by the second user;

[0054] Step S102: performing data cleaning on the reply text to determine preprocessed data;

[0055] Step S104: determining a historical database;

[0056] Step S106: determining a plurality of labeled samples in the historical database;

[0057] Step S108: determining at least one unlabeled sample in the preprocessed data;

[0058] Step S110: labeling the unlabeled sample in the preprocessed data according to the labeled samples to determine labeled data;

[0059] Step S112: performing sampling evaluation on the labeled data to determine mined data;

[0060] Step S114: training a deep learning model according to the mined data;

[0061] Step S116: inputting the reply text into the deep learning model to determine a discrimination result corresponding to the reply text;

[0062] The discrimination result is used to indicate whether the reply text is associated with a data source corresponding to the historical database.

[0063] The text discrimination method provided by the application analyzes the reply text in the original text data corresponding to the second user of the first user, judges whether the content in the reply text is associated with the data source corresponding to the historical database, and outputs the corresponding judgment result. Among them, the original text data corresponding to the second user of the first user includes multiple original question texts of the second user to the first user, and the reply text corresponding to the original question text. The reply text includes the answer of the first user to each original question text of the second user. Through data preparation and preprocessing, the preprocessed data corresponding to the reply text is determined. At least one unlabeled sample in the preprocessed data, i.e. multiple strings or text contents, is determined. According to the multiple labeled samples in the historical database, the unlabeled samples are labeled to determine multiple mining data in the preprocessed data. The mining data all have labels, which are used to distinguish whether they are associated with the data source corresponding to the historical database.

[0064] The multiple labeled data associated with the data source corresponding to the historical database in the mining data and the multiple labeled data not associated with the data source corresponding to the historical database in the mining data are used as a training set to train a deep learning model. After learning the training set, the deep learning model obtains the interaction information between each word in the reply text and other words through the self-attention mechanism, judges the association degree between the multiple labeled data associated with the data source corresponding to the historical database in the reply text through the interaction information, and determines whether the reply text is associated with the data source corresponding to the historical database according to the association degree, to determine the discrimination result of the reply text.

[0065] It can be understood that through the historical labeled samples and the automatic data mining and evaluation, the automatic deep learning model reduces the cost of manual labeling and the error caused by manual review, improves the accuracy and robustness of text discrimination, and greatly reduces the missed detection rate to meet the discrimination requirements of text data.

[0066] Exemplarily, the first user is a candidate participating in an interview, and the second user is an interviewer.

[0067] In some embodiments, optionally, the data source of the historical database corresponds to at least one artificial intelligence model.

[0068] In some embodiments, optionally, the multiple labeled samples in the historical database are reply texts generated by the artificial intelligence model.

[0069] Specifically, after determining the reply text of the candidate to the question of the interviewer from the log of the interview system, the multiple reply texts are data cleaned. Data cleaning includes deletion of stop words, deletion of punctuation marks, and handling of garbled codes, etc., to determine preprocessed data. Through the way of determining preprocessed data through data cleaning, the data processing amount is reduced, thereby improving the text discrimination efficiency.

[0070] After determining the plurality of labeled data, the newly labeled labeled data is automatically checked. Whether the plurality of labeled data is the same as the text content in the preprocessed data is judged by sampling detection, and the quality of the labeled data is evaluated. If the quality meets the standard, the output is mining data for training of the deep learning model; if it does not meet the standard, data mining is performed again, and the plurality of unlabeled samples is relabeled to improve the quality and robustness of the data set for training.

[0071] In some embodiments, optionally, in the process of labeling the unlabeled samples in the preprocessed data, the preprocessed data is labeled by small sample data, the unlabeled samples are data mined, the mining data is determined, and the efficiency of text discrimination is improved.

[0072] In some embodiments, optionally, the discrimination result output by the model includes a classification label or a discrimination probability. The classification label is used to indicate whether the reply text is from an artificial intelligence model; and the discrimination probability is used to indicate the probability that the reply text is generated by an artificial intelligence model.

[0073] In some embodiments, optionally, the reply text includes but is not limited to the input text of the first user, the voice input text and the handwriting image text.

[0074] In some embodiments, optionally, the original text data includes not only pure text replies, but also metadata associated with the text content, such as answer time stamp, delay time from question display to start answering, editing times during answering and cursor activity log. The discrimination result of the reply text can be combined with the plurality of metadata for comprehensive judgment, and the accurate discrimination result can be determined according to the discrimination result and the metadata to improve the text discrimination accuracy in the interview scenario.

[0075] In some embodiments, optionally, the step of labeling the unlabeled samples is performed based on a configurable mining rule library, the mining rule library includes a plurality of mining strategies, and one or more mining strategies are combined according to the text type and text features of the unlabeled samples to determine the mining data corresponding to the preprocessed data, so as to improve the correlation between the mining data and the reply text.

[0076] In some embodiments, optionally, after the reply text of the first user is discriminated and the discrimination result is generated, the text content from the artificial intelligence model in the reply text of the first user is stored in the historical database, and the historical database is automatically updated.

[0077] In some embodiments, optionally, as shown in Figure 2 Step S110: according to the labeled samples, labeling the unlabeled samples in the preprocessed data to determine the labeled data, including:

[0078] Step S1100: determining original question text corresponding to the reply text in the original text data;

[0079] Step S1102: extracting key vocabularies from the preprocessed data to determine a key vocabulary set;

[0080] Step S1104: determining semantic correlation degree of the key vocabulary set and the original question text;

[0081] Step S1106: determining a preset correlation threshold;

[0082] Step S1108: labeling the unlabeled samples according to the semantic correlation degree and the preset correlation threshold to determine labeled data;

[0083] When the semantic correlation degree is greater than or equal to the preset correlation threshold, the first label is labeled for the unlabeled sample; when the semantic correlation degree is less than the preset correlation threshold, the second label is labeled for the unlabeled sample.

[0084] In this embodiment, the important vocabularies corresponding to the original question text in the reply text are extracted by the important word model, and the unlabeled samples with a semantic correlation degree greater than a preset correlation threshold are queried and labeled in a plurality of unlabeled samples. The label is used to reflect whether the reply text is generated by an artificial intelligence model.

[0085] The label includes a first label and a second label, and the semantics of the first label and the second label are opposite. The first label indicates that the semantic correlation degree between the reply text and the original question text is greater than or equal to the preset correlation threshold, that is, the reply content corresponding to the reply text is associated with the question in the original question text, and the reply text is not generated by an artificial intelligence model. The second label indicates that the semantic correlation degree between the reply text and the original question text is less than the preset correlation threshold, that is, the reply content corresponding to the reply text is irrelevant to the question in the original question text, and the reply text is generated by an artificial intelligence model.

[0086] It can be understood that according to the characteristics of the AI large model in generating answers, the key vocabularies of the reply text are extracted from the perspective of the correlation degree between the question and the answer, the logical relevance between the question text and the reply text is judged, and the generalization ability and robustness of the text discrimination in the interview scene are improved.

[0087] Wherein, after determining the original question text corresponding to the reply text in the original text data, keyword extraction is performed on the question text, that is, feature extraction is performed on the answer, and a keyword extraction technology in natural language processing (NLP) (such as TF-IDF, TextRank algorithm or more advanced deep learning model) is used to process the cleaned reply text. The purpose is to extract the word group in the reply text that best represents the core theme and content of the reply text, form a keyword set, and the keyword set reflects the essence of the question text.

[0088] In some embodiments, optionally, the keyword in the reply text is extracted by an important word vector model.

[0089] Specifically, the keyword set and the original question text are input into a semantic representation model (such as a sentence embedding model generated by Word2Vec, GloVe, BERT, etc.), which is used to convert text into vectors in a high-dimensional space, which captures the semantic information of the original question text.

[0090] In some embodiments, optionally, the semantic correlation degree is obtained by calculating the cosine similarity or other distance measurement between the two vectors. The higher the value, the higher the correlation between the keyword in the reply text and the original question text.

[0091] In some embodiments, optionally, the preset correlation threshold is determined according to the experimental data corresponding to a plurality of labeled samples in the historical database, and the preset correlation threshold is a critical value for dividing text content and artificial intelligence model related or unrelated.

[0092] In some embodiments, optionally, the keyword extraction process in the reply text uses a domain-specific dictionary (such as a technical term library, an industry slang dictionary) for weighting to ensure that the extracted keywords have higher weights and representativeness in the context related to the interview field.

[0093] In some embodiments, optionally, the preset correlation threshold is not a fixed value, but is dynamically adjusted according to the complexity, length or field of the original question. For example, for complex open questions, the preset correlation threshold can be appropriately reduced; for simple factual questions, the preset correlation threshold is increased.

[0094] In some embodiments, optionally, the label labeled for the unlabeled sample is a continuous probability value (for example, a probability of 0.7 for AI generation), which can be mapped by a semantic correlation degree through an activation function (sigmoid) to provide more rich information for the subsequent fusion model.

[0095] In some embodiments, optionally, as shown in Figure 3 Step S110: according to the labeled samples, labeling the unlabeled samples in the preprocessed data to determine the labeled data, further comprising:

[0096] Step S1110: determining the text similarity model;

[0097] Step S1112: inputting the labeled samples and at least one unlabeled sample into the text similarity model to determine the text similarity of the labeled samples and the unlabeled samples;

[0098] Step S1114: determining the similarity threshold according to the historical database;

[0099] Step S1116: labeling the unlabeled samples according to the text similarity and the similarity threshold to determine the labeled data;

[0100] Wherein, when the text similarity is greater than or equal to the similarity threshold, the second label is labeled for the unlabeled sample; when the text similarity is less than the similarity threshold, the first label is labeled for the unlabeled sample.

[0101] In this embodiment, the similarity calculation model is used to calculate the text similarity of the labeled samples in the historical database and the unlabeled samples in the preprocessed data, and the unlabeled samples are labeled by comparing the text similarity with the similarity threshold.

[0102] It can be understood that the artificial intelligence model will mark a certain pattern or tend to use a specific template or sentence when generating an answer, especially for common questions. By identifying a batch of text highly similar to known AI-generated samples, i.e., labeled samples, the unlabeled samples are labeled, making the mined data more robust. Moreover, the label is determined by the similarity value, making the discrimination result more convincing and interpretable, and facilitating manual review. For example, the reply text is highly similar to a known AI-generated answer (similarity of 92%).

[0103] Specifically, the text similarity model is a trained calculation model that can quantify the similarity between two text segments. This model can be a traditional natural language processing-based model, for example, a cosine similarity calculation based on TF-IDF vectors. It can also be a more advanced deep learning model, such as a sentence embedding model generated by Sentence-BERT, SimCSE, etc. The choice of model determines the accuracy and robustness of similarity calculation.

[0104] Select one or more known-labeled samples, i.e. labeled samples, from the historical database, e.g. samples known to be AI-generated or human-written, and input them into the similarity model together with the unlabeled sample to be labeled. The model outputs a numerical similarity score (e.g. between 0 and 1) that objectively reflects the distance or closeness of the two texts in the feature space understood by the model.

[0105] In some embodiments, optionally, a threshold value that best distinguishes the two classes of samples, i.e. the similarity threshold, is determined by calculating the average similarity between samples of the same class (AI-generated samples) in the historical database, and their average similarity with samples of the other class (human-written samples).

[0106] If the unlabeled sample is highly similar to a known labeled sample, it is very likely to come from the same source or have the same attribute. Therefore, it is given the same second label as the known sample.

[0107] If the unlabeled sample is not similar to any known labeled sample in the historical database, it means it is a new and different expression. In the absence of evidence that it is AI-generated, it is defaulted to the first label.

[0108] In some embodiments, optionally, the labeled samples in the historical database cover common AI-generated patterns.

[0109] In some embodiments, optionally, multiple labeled samples are compared with an unlabeled sample. The maximum similarity or the average similarity is finally taken as the final similarity value of the unlabeled sample.

[0110] In one embodiment, optionally, when human review finds that the automatic labeling is wrong, the wrong sample and its correct label are added to the historical database and trigger a threshold value recalculation, so that the system can continuously evolve and cope with new AI models.

[0111] In one embodiment, optionally, the output of the labeling step is not only a binary label, but also includes a confidence score (e.g. the text similarity itself or a calibrated probability). This confidence score can be used as a weight for downstream processes, or to filter high-confidence samples to be directly added to the training set.

[0112] In some embodiments, optionally, as shown in FIG. 1, step S110: labeling the unlabeled samples in the preprocessed data according to the labeled samples to determine the labeled data, further includes: Figure 4

[0113] Step S1130: determining a plurality of character strings corresponding to the unlabeled sample;

[0114] ​Step S1132: determining the reply text corresponding to the string;

[0115] Step S1134: determining the character type of the reply text corresponding to the string;

[0116] Step S1136: replacing the string according to the character type to determine the replacement sample;

[0117] Step S1138: determining the labeled data according to the replacement sample.

[0118] In the embodiment, the labeling of the unlabeled sample in the preprocessed data further includes constructing a new sample from the unlabeled sample by replacing the entity word in the unlabeled sample, determining a replacement sample, and determining labeled data corresponding to the replacement sample.

[0119] It can be understood that when generating text, the AI large model often produces behaviors such as synonym replacement and number format conversion. By actively replacing the string in the reply text, the replacement behavior of the AI is simulated, and new training samples are automatically generated from the unlabeled data. When encountering new AI-generated text, the replacement sample is used for identification, so that the text discrimination method is more robust and general.

[0120] Specifically, the keyword extraction technology in NLP is used to determine the string that can keep the sentence meaning unchanged after replacement from multiple unlabeled samples. The character type of the target string in the specific context of the reply text to which it belongs is determined, such as Chinese numbers, Arabic numbers, full-width characters, half-width characters, abbreviations, or complete terms, etc. According to the analyzed character type, a preset replacement rule library is called to replace the original string.

[0121] For example, some quantifiers in Chinese numbers are changed to Arabic numbers, or formal terms are replaced by abbreviations.

[0122] By replacing the original string, a replacement sample that is different in surface form but consistent in core semantics is generated, and the replacement sample is labeled. The label of the replacement sample is inherited from the original unlabeled sample, and the labeled data is determined.

[0123] In some embodiments, optionally, the mining strategy includes but is not limited to the ways of semantic correlation comparison, text similarity comparison, and sample replacement. The preprocessed data corresponding to the mining data is determined by one or more mining strategies in combination.

[0124] In some embodiments, optionally, as shown in Figure 5 Step S112: sampling and evaluating the labeled data to determine the mining data, including:

[0125] Step S1120: Randomly sample multiple labeled data to determine the evaluation sample set;

[0126] Step S1122: Determine the complete annotation parameters for each labeled data in the evaluation sample set;

[0127] Step S1124: Determine the quality threshold;

[0128] Step S1126: When the complete annotation parameters are greater than or equal to the quality threshold, the annotated data is determined to be mining data;

[0129] Step S1128: When the complete annotation parameters are less than the quality threshold, randomly sample multiple annotation data to determine the evaluation sample set until the complete annotation parameters are greater than or equal to the quality threshold.

[0130] In this embodiment, after the labeled data is determined, the newly labeled preprocessed data is automatically validated. The complete annotation parameters of each labeled data point are compared with a quality threshold. If the threshold is not met (i.e., the complete annotation parameters are less than the quality threshold), the mining rules are further optimized, and the unprocessed samples are re-labeled to determine new labeled data, until the complete annotation parameters are greater than or equal to the quality threshold. If the threshold is met (i.e., the complete annotation parameters are greater than or equal to the quality threshold), the labeled data is determined as mining data and used as the training set for the deep learning model.

[0131] Understandably, given the potentially massive amount of automatically labeled data, manual evaluation of all of it would be prohibitively expensive. By employing random sampling, a statistically representative evaluation sample set is drawn from the large-scale labeled data for quality control. This ensures that only high-quality data is used in the model training phase, guaranteeing the accuracy, reliability, and stability of the final model from the outset.

[0132] The complete annotation parameters include one or more of the following indicators: the proportion of samples whose automatically labeled labels match the correct labels, the probability value or similarity score output by the model during the automatic annotation process, and the completeness of multiple labeled data corresponding to the preprocessed data.

[0133] In some embodiments, the quality threshold may optionally be the lowest acceptable data quality standard for model training.

[0134] In some embodiments, optionally, such as Figure 6 As shown, step S114: Training a deep learning model based on the mined data includes:

[0135] Step S1140: Determine the initial deep learning model;

[0136] Step S1142: Divide the mined data into a training set and a validation set;

[0137] Step S1144: training the initial deep learning model according to the training set to determine a training learning model;

[0138] Step S1146: verifying the model performance of the deep learning model according to the verification set to determine a verification result;

[0139] Step S1148: when the accuracy corresponding to the verification result reaches a preset target, determining the deep learning model according to the training learning model;

[0140] Step S1150: when the accuracy corresponding to the verification result does not reach the preset target, re-dividing the training set and the verification set.

[0141] In this embodiment, the deep learning model is trained by mining data, and the trained deep learning model is used to determine whether the reply text of the first user is generated by AI.

[0142] The initial deep learning model is a pre-trained model (Pre-trained Model). For example, an ALBERT, BERT or the like pre-trained on a large number of general texts is used as a starting point. This is equivalent to letting the model already have general language understanding ability, and we only need to fine-tune it on this basis to adapt to the specific task of distinguishing AI text, which is faster and better than training the model from scratch.

[0143] High-quality mining data is randomly divided into two parts in a certain proportion (such as 8:2): a training set (Training Set) is used to directly update the model parameters, so that the model learns the feature patterns in the data; a verification set (Validation Set) is used to evaluate the performance of the model on unseen data, monitor the training process, prevent overfitting (Overfitting), and serve as a basis for deciding whether to stop training.

[0144] The training set data is input into the initial model, and the model internal parameters are iteratively adjusted through back propagation and optimization algorithms (such as AdamW) to minimize the prediction error or loss function, and finally the training learning model is obtained.

[0145] The training learning model is tested using the verification set data that did not participate in the training. The performance of the training learning model is measured by accuracy (Accuracy), recall (Recall), score and other indicators, and the comprehensive evaluation of these indicators is the verification result.

[0146] When the accuracy of the verification result reaches the preset target, it means that the model has successfully learned, and its generalization performance has reached the level required by the business. At this time, the training process is completed, and the current training learning model is determined as the final available deep learning model; when the accuracy of the verification result does not reach the preset target, it means that the training effect is not as expected. At this time, the training set and the validation set are re-divided, and the training learning model is re-trained to reduce accidental errors and improve the robustness of text discrimination.

[0147] Exemplarily, the deep learning model adopts an (Albert model). Compared with the deep learning model (Bert), Albert reduces the overall parameter quantity, speeds up the training, and increases the model effect. The Albert model can obtain the interaction information of each word with other words through a self-attention mechanism, and better express the text.

[0148] In some embodiments, optionally, the discrimination result output by the deep learning model includes the probability of the specified label.

[0149] In some embodiments, optionally, a plurality of different specified labels are determined according to different business scenarios.

[0150] In one specific embodiment, optionally, the text discrimination method of the present application mainly includes four stages, as shown in Figure 9 After determining the interview log data source in step S200, the four stages are:

[0151] Step S202: Obtain the data source and clean the data in the data source;

[0152] Step S204: Label and mine the data;

[0153] Step S206: Model training and optimization;

[0154] Step S208: Output the category corresponding to the interview log data source.

[0155] In step S202: obtaining the data source and cleaning the data in the data source, a large number of answer texts are obtained from the logs of the AI interview process of the business. These texts need to be data cleaned, including stop word deletion, punctuation deletion, and code processing. Data determines the upper limit of the model, so data preprocessing is necessary.

[0156] In the data mining stage, the main process is to mine the data of the sample by labeling and preprocessing the data through small sample data. There are mainly three methods for mining.

[0157] 1. Replace the entity words in the sample to form a new sample, for example, change some quantifiers to Arabic numerals;

[0158] 2. Using the existing similarity calculation model, the similarity of the labeled samples and unlabeled samples is calculated, and the unlabeled samples under a certain threshold are labeled;

[0159] 3. The important words of the text are extracted through the important word model, and the labeled samples are queried in the unlabeled samples.

[0160] After mining, the newly labeled data is manually checked, and if it does not meet the standard, the mining rules will be further optimized, and finally the data mining is completed.

[0161] The flowchart of data mining is shown in Figure 10 , including:

[0162] Step S300: Determine the preprocessed data;

[0163] Step S302: Small sample data labeling;

[0164] Step S304: Data mining;

[0165] Step S306: Data sampling evaluation is performed to determine whether the standard is met, and when the judgment result of step S306 is no, return to step S304 until the sampling evaluation meets the standard;

[0166] When the judgment result of step S306 is yes, step S308: training data mining is performed.

[0167] In the model training and optimization stage, the model mainly uses the deep learning model (Albert). Compared with the deep learning model (Bert), Albert reduces the overall parameter quantity, speeds up the training speed, and increases the model effect. Mainly in reducing the mapping dimension of the word, from the experiment, the size of the word embedding (embedding) is not completely positively correlated with the effect, so 64 dimensions are selected; and cross-layer parameter sharing, whether it is stacked 12 layers or 24 layers, an encoder is selected. Compared with the shallow model, Albert can better obtain the text semantics, and through the self-attention mechanism, it can obtain the interaction information of each word with other words, and better express the text.

[0168] In the output corresponding to the interview log data source category stage, the output category is the probability of predicting the specified label (label) through the model, and the concerned label is selected according to the specific business scenario, and the data set corresponding to the label is obtained through the threshold.

[0169] Understandably, the present application is mainly to reduce the cost of manual annotation, provide more recall data, anti-cheat for interviewers suspected of AI-generated text, ensure the fairness of the interview, and improve the efficiency and quality of Human Resources (HR) interview screening.

[0170] As Figure 8 shown, the present application also provides a text discrimination device 900, which comprises: a data acquisition module 902, configured to determine original text data associated with a first user and a second user, the original text data at least comprising reply text made by the first user to a question raised by the second user; a preprocessing module 904, configured to perform data cleaning on the reply text to determine preprocessed data; a data source module 906, configured to determine a historical database; a sample determination module 908, configured to determine a plurality of labeled samples in the historical database; a label identification module 910, configured to determine at least one unlabeled sample in the preprocessed data; a data labeling module 912, configured to label the unlabeled sample in the preprocessed data according to the labeled sample to determine labeled data; a data mining module 914, configured to sample and evaluate the labeled data to determine mined data; a model training module 916, configured to train a deep learning model according to the mined data; a text discrimination module 918, configured to input the reply text into the deep learning model to determine a discrimination result corresponding to the reply text; wherein the discrimination result is used to indicate whether the reply text is associated with a data source corresponding to the historical database.

[0171] As Figure 7 shown, the present application also provides an electronic device 1000, which comprises a processor 1110, a memory 1109, a program or instruction stored on the memory 1109 and executable on the processor 1110, the program or instruction being executed by the processor 1110 to implement various processes of the embodiments of the above-mentioned text discrimination method and achieve the same technical effects. To avoid repetition, details are not repeated here.

[0172] Optionally, the processor 1110 is configured to determine original text data associated with a first user and a second user, the original text data at least comprising reply text made by the first user to a question raised by the second user;

[0173] Optionally, the processor 1110 is further configured to perform data cleaning on the reply text to determine preprocessed data;

[0174] Optionally, the processor 1110 is further configured to determine a historical database;

[0175] Optionally, the processor 1110 is further configured to determine a plurality of labeled samples in the historical database;

[0176] Optionally, the processor 1110 is further configured to determine at least one unlabeled sample in the preprocessed data.

[0177] Optionally, the processor 1110 is further configured to label the unlabeled sample in the preprocessed data according to the labeled sample, and determine labeled data.

[0178] Optionally, the processor 1110 is further configured to sample and evaluate the labeled data, and determine mined data.

[0179] Optionally, the processor 1110 is further configured to train the deep learning model according to the mined data.

[0180] Optionally, the processor 1110 is further configured to input the reply text into the deep learning model, and determine a discrimination result corresponding to the reply text.

[0181] The memory 1109 can be configured to store software programs and various data. The memory 1109 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 1109 can include a volatile memory or a non-volatile memory, or the memory 1109 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 1109 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0182] The embodiment of the present application further provides a readable storage medium, and the readable storage medium stores programs or instructions. The programs or instructions are executed by a processor to realize the processes of the text discrimination method and achieve the same technical effects. To avoid repetition, details are not described herein. In addition, the readable storage medium improves the data storage capacity and data processing speed of the text discrimination method.

[0183] The readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or a grooved medium having a recording, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media, or electrical signals transmitted through a wire.

[0184] The processor is a processor in the electronic device in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like.

[0185] The embodiment of the present application further provides a chip, and the chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to execute programs or instructions to realize the processes of the text discrimination method and achieve the same technical effects. To avoid repetition, details are not described herein. In addition, the chip improves the data processing speed of the text discrimination method.

[0186] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip, and the like.

[0187] In the present application, the terms "first", "second", "third" are only used for descriptive purpose, and should not be understood as indicating or implying relative importance. The term "multiple" refers to two or more, unless otherwise explicitly limited. The terms "mount", "connect", "connection", "fix", and the like should be interpreted broadly, for example, "connection" can be fixed connection, or detachable connection, or integral connection; "connection" can be direct connection, or indirect connection through intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0188] In the description of the present application, it should be understood that the terms "upper", "lower", "left", "right", "front", "back", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or unit referred to must have a particular direction, be constructed and operated in a particular orientation, therefore, should not be understood as a limitation on the present application.

[0189] In the description of the present application, the terms "one embodiment", "some embodiments", "a specific embodiment", and the like, mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0190] The above is only the preferred embodiment of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A text discrimination method, characterized in that, include: Determine the raw text data associated with the first user and the second user, the raw text data including at least the text of the first user's response to a question raised by the second user; The response text is cleaned to determine the preprocessed data; Determine the historical database; Identify multiple labeled samples in the historical database, wherein the labeled samples are response texts generated by an artificial intelligence model; Identify at least one unlabeled sample in the preprocessed data; Based on the labeled samples, the unlabeled samples in the preprocessed data are labeled to determine the labeled data; The labeled data is sampled and evaluated to determine the data to be mined; A deep learning model is trained based on the mined data; The response text is input into the deep learning model to determine the discrimination result corresponding to the response text; The discrimination result is used to indicate whether the response text is associated with the data source corresponding to the historical database; The step of labeling unlabeled samples in the preprocessed data based on the labeled samples to determine the labeled data includes: Determine the original question text in the original text data that corresponds to the response text; Extract keywords from the preprocessed data to determine the keyword set; Determine the semantic relevance between the keyword set and the original question text; Determine the preset association threshold; The unlabeled samples are labeled according to the semantic relevance and the preset relevance threshold to determine the labeled data; Specifically, when the semantic relevance is greater than or equal to the preset relevance threshold, the unlabeled sample is labeled with a first label; When the semantic relevance is less than the preset relevance threshold, a second label is assigned to the unlabeled sample; The first and second tags have opposite semantics and are used to reflect whether the response text was generated by the artificial intelligence model.

2. The text discrimination method according to claim 1, characterized in that, The step of labeling unlabeled samples in the preprocessed data based on the labeled samples to determine the labeled data further includes: Determine the text similarity model; The labeled sample and at least one unlabeled sample are input into the text similarity model to determine the text similarity between the labeled sample and the unlabeled sample; A similarity threshold is determined based on the historical database; The unlabeled samples are labeled based on the text similarity and the similarity threshold to determine the labeled data; Specifically, when the text similarity is greater than or equal to the similarity threshold, the unlabeled sample is labeled with the second label; When the text similarity is less than the similarity threshold, the first label is assigned to the unlabeled sample.

3. The text discrimination method according to claim 1, characterized in that, The step of labeling unlabeled samples in the preprocessed data based on the labeled samples to determine the labeled data further includes: Identify multiple strings corresponding to the unlabeled samples; Determine the response text corresponding to the string; Determine the text type of the response text corresponding to the string; Replace the string according to the text type to determine the replacement sample; The labeled data is determined based on the replacement sample.

4. The text discrimination method according to claim 1, characterized in that, The step of sampling and evaluating the labeled data to determine the data to be mined includes: Random sampling is performed on multiple labeled data sets to determine the evaluation sample set; Determine the complete annotation parameters for each of the labeled data in the evaluation sample set; Determine the quality threshold; When the complete annotation parameters are greater than or equal to the quality threshold, the annotation data is determined to be mining data; When the completeness parameter of the annotation is less than the quality threshold, the multiple annotation data are randomly sampled again to determine the evaluation sample set until the completeness parameter of the annotation is greater than or equal to the quality threshold.

5. The text discrimination method according to any one of claims 1 to 4, characterized in that, The step of training a deep learning model based on the mined data includes: Determine the initial deep learning model; The mined data is divided into a training set and a validation set; The initial deep learning model is trained based on the training set to determine the training learning model; The performance of the deep learning model is verified based on the verification set, and the verification result is determined. When the accuracy corresponding to the verification result reaches the preset target, a deep learning model is determined based on the training learning model. When the accuracy corresponding to the verification result does not reach the preset target, the training set and the verification set are re-divided.

6. A text discrimination device, characterized in that, include: The data acquisition module is used to determine the original text data associated with the first user and the second user, wherein the original text data includes at least the text of the first user's reply to the question raised by the second user; The preprocessing module is used to clean the response text and determine the preprocessed data. The data source module is used to determine the historical database; A sample determination module is used to determine multiple labeled samples in the historical database, wherein the labeled samples are response texts generated by an artificial intelligence model; The annotation and recognition module is used to identify at least one unlabeled sample in the preprocessed data; The data annotation module is used to annotate the unannotated samples in the preprocessed data according to the annotated samples, and determine the annotated data; The data mining module is used to sample and evaluate the labeled data to determine the data to be mined. The model training module is used to train a deep learning model based on the mined data; The text discrimination module is used to input the response text into the deep learning model and determine the discrimination result corresponding to the response text; The discrimination result is used to indicate whether the response text is associated with the data source corresponding to the historical database; The step of labeling unlabeled samples in the preprocessed data based on the labeled samples to determine the labeled data includes: Determine the original question text in the original text data that corresponds to the response text; Extract keywords from the preprocessed data to determine the keyword set; Determine the semantic relevance between the keyword set and the original question text; Determine the preset association threshold; The unlabeled samples are labeled according to the semantic relevance and the preset relevance threshold to determine the labeled data; Specifically, when the semantic relevance is greater than or equal to the preset relevance threshold, the unlabeled sample is labeled with a first label; When the semantic relevance is less than the preset relevance threshold, a second label is assigned to the unlabeled sample; The first and second tags have opposite semantics and are used to reflect whether the response text was generated by the artificial intelligence model.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the text discrimination method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the text discrimination method as described in any one of claims 1 to 5.

9. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the text discrimination method as described in any one of claims 1 to 5.

Images