Text discrimination method and device, electronic equipment, readable storage medium and chip

Abstract

The embodiment of the invention provides a text discrimination method and device, electronic equipment, a readable storage medium and a chip, and the text discrimination method comprises the steps: determining a reply text made by a first user for a question proposed by a second user; performing data cleaning on the reply text, and determining preprocessed data; determining a historical database; determining a plurality of annotation samples in a historical database; determining at least one unlabeled sample in the preprocessed data; according to the labeled samples, labeling unlabeled samples in the preprocessed data, and determining labeled data; carrying out sampling evaluation on the annotated data, and determining mining data; training a deep learning model according to the mined data; inputting the reply text into a deep learning model, and determining a judgment result corresponding to the reply text; wherein the judgment result is used for representing whether the reply text is associated with a data source corresponding to the historical database or not. Through the scheme of the invention, the robustness of distinguishing the text generated by artificial intelligence in the interview scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a text discrimination method and device, electronic equipment, readable storage medium and chip. BACKGROUND

[0002] Under the background of rapid development of current artificial intelligence (Artificial Intelligence, AI) large language model, the response speed of its text generation is fast, the semantic quality is high, and the style is natural, which leads to the fact that in online interview, written interview and other actual business scenarios, some interviewees generate content by means of large model to pass the examination in a "cheating" way. This kind of behavior seriously affects the judgment of the real ability of the candidates by enterprises, and interferes with the normal screening process.

[0003] The existing identification means mainly rely on manual verification, which is not only inefficient and subjective, but also difficult to expand in the face of large-scale business scenarios, with low recall rate and high cost. Therefore, the existing technology cannot meet the actual needs of efficiently and accurately identifying AI generated text. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a text discrimination method, device, electronic equipment, readable storage medium and chip, which can solve the problems of low efficiency and strong subjectivity of AI generated text discrimination.

[0005] Therefore, the embodiments of the first aspect of the present application provide a text discrimination method.

[0006] The embodiments of the second aspect of the present application provide a text discrimination device.

[0007] The embodiments of the third aspect of the present application provide an electronic equipment.

[0008] The embodiments of the fourth aspect of the present application provide a readable storage medium.

[0009] The embodiments of the fifth aspect of the present application provide a chip.

[0010] To achieve the above object, the embodiment of the first aspect of the present application provides a text discrimination method, comprising: determining original text data associated with a first user and a second user, the original text data at least including reply text made by the first user to a question raised by the second user; performing data cleaning on the reply text to determine preprocessed data; determining a historical database; determining a plurality of labeled samples in the historical database; determining at least one unlabeled sample in the preprocessed data; labeling the unlabeled sample in the preprocessed data according to the labeled samples to determine labeled data; performing sampling evaluation on the labeled data to determine mined data; training a deep learning model according to the mined data; inputting the reply text into the deep learning model to determine a discrimination result corresponding to the reply text; wherein the discrimination result is used to represent whether the reply text is associated with a data source corresponding to the historical database.

[0011] The text discrimination method provided by the present application analyzes the reply text in the original text data corresponding to the second user of the first user, judges whether the content in the reply text is associated with the data source corresponding to the historical database, and outputs the corresponding judgment result. The original text data corresponding to the second user of the first user includes a plurality of original question texts of the first user to the second user and reply texts corresponding to the original question texts. The reply text includes the answers of the first user to each original question text of the second user. Through data preparation and preprocessing, preprocessed data corresponding to the reply text is determined. At least one unlabeled sample, i.e. a plurality of character strings or text contents, in the preprocessed data is determined. The unlabeled sample is labeled according to a plurality of labeled samples in the historical database to determine a plurality of mined data in the preprocessed data. The mined data all have labels and are used to distinguish whether they are associated with the data source corresponding to the historical database.

[0012] The plurality of labeled data associated with the data source corresponding to the historical database in the mined data and the plurality of labeled data not associated with the data source corresponding to the historical database in the mined data are used as a training set to train a deep learning model. After learning the training set, the deep learning model obtains the interaction information between each character in the reply text and other characters through a self-attention mechanism, judges the association degree between the plurality of labeled data associated with the data source corresponding to the historical database in the reply text through the interaction information, and determines the discrimination result of the reply text according to whether the reply text is associated with the data source corresponding to the historical database according to the association degree.

[0013] It can be understood that the deep learning model is automatically trained through historical labeled samples and automatic data mining and evaluation, the cost of manual labeling is reduced, the error caused by manual review is reduced, the accuracy and robustness of text discrimination are improved, and the false negative rate is greatly reduced to meet the discrimination requirements of text data.

[0014] In some embodiments, the unannotated samples in the preprocessed data are annotated according to the annotated samples to determine the annotated data, including: determining original question text corresponding to the reply text in the original text data; extracting key vocabularies from the preprocessed data to determine a key vocabulary set; determining a semantic correlation degree of the key vocabulary set and the original question text; determining a preset correlation threshold; annotating the unannotated samples according to the semantic correlation degree and the preset correlation threshold to determine the annotated data; wherein, when the semantic correlation degree is greater than or equal to the preset correlation threshold, a first label is annotated for the unannotated sample; and when the semantic correlation degree is less than the preset correlation threshold, a second label is annotated for the unannotated sample.

[0015] In the present scheme, the important vocabularies corresponding to the original question text in the reply text are extracted by the important word model, and the unannotated samples with a semantic correlation degree greater than a preset correlation threshold are queried and labeled in a plurality of unannotated samples. The label is used to reflect whether the reply text is generated by an artificial intelligence model.

[0016] The label includes a first label and a second label, and the semantics of the first label and the second label are opposite. The first label indicates that the semantic correlation degree between the reply text and the original question text is greater than or equal to the preset correlation threshold, that is, the reply content corresponding to the reply text is associated with the question in the original question text, and the reply text is not generated by an artificial intelligence model. The second label indicates that the semantic correlation degree between the reply text and the original question text is less than the preset correlation threshold, that is, the reply content corresponding to the reply text is irrelevant to the question in the original question text, and the reply text is generated by an artificial intelligence model.

[0017] It can be understood that, according to the characteristics of the AI large model in generating answers, the AI large model tends to generate content with universality and strong generalization. Therefore, the key vocabularies of the reply text are extracted, the logical relevance between the question text and the reply text is judged, and the generalization ability and robustness of the text discrimination in the interview scene are improved.

[0018] In some embodiments, the unannotated samples in the preprocessed data are annotated according to the annotated samples to determine the annotated data, further including: determining a text similarity model; inputting the annotated sample and at least one unannotated sample into the text similarity model to determine the text similarity of the annotated sample and the unannotated sample; determining a similarity threshold according to the historical database; annotating the unannotated sample according to the text similarity and the similarity threshold to determine the annotated data; wherein, when the text similarity is greater than or equal to the similarity threshold, the second label is annotated for the unannotated sample; and when the text similarity is less than the similarity threshold, the first label is annotated for the unannotated sample.

[0019] In the scheme, the similarity calculation model is used to calculate the text similarity of the labeled samples in the historical database and the unlabeled samples in the preprocessed data. By comparing the text similarity with the similarity threshold, the unlabeled samples are labeled.

[0020] Understandably, the artificial intelligence model will mark certain patterns or tend to use specific templates or sentence patterns when generating answers, especially for common questions. By batch identifying text highly similar to known AI-generated samples, i.e., labeled samples, the unlabeled samples are labeled, making the mined data more robust. Moreover, by determining the label through the similarity value, the discrimination result is more convincing and interpretable, facilitating manual review. For example, the reply text is highly similar to a known AI-generated answer (similarity of 92%).

[0021] In some technical solutions, the unlabeled samples in the preprocessed data are labeled according to the labeled samples to determine the labeled data, which further includes: determining a plurality of character strings corresponding to the unlabeled samples; determining the reply text corresponding to the character strings; determining the character type of the reply text corresponding to the character strings; replacing the character strings according to the character type to determine the replacement samples; and determining the labeled data according to the replacement samples.

[0022] In the scheme, labeling the unlabeled samples in the preprocessed data further includes replacing the entity words of the unlabeled samples to construct new samples, determining the replacement samples, and determining the labeled data corresponding to the replacement samples.

[0023] In some technical solutions, the labeled data is sampled and evaluated to determine the mined data, including: randomly sampling the plurality of labeled data to determine the evaluation sample set; determining the labeling completeness parameter of each labeled data in the evaluation sample set; determining the quality threshold; when the labeling completeness parameter is greater than or equal to the quality threshold, determining the labeled data as the mined data; when the labeling completeness parameter is less than the quality threshold, randomly sampling the plurality of labeled data to determine the evaluation sample set again until the labeling completeness parameter is greater than or equal to the quality threshold.

[0024] In the scheme, after determining the labeled data, the preprocessed data labeled with the new label is automatically verified. The labeling completeness parameter of each labeled data is compared with the quality threshold. If it does not meet the standard, i.e., the labeling completeness parameter is less than the quality threshold, the mining rule is further optimized, the unlabeled samples are labeled again to determine new labeled data until the labeling completeness parameter is greater than or equal to the quality threshold. If it meets the standard, i.e., the labeling completeness parameter is greater than or equal to the quality threshold, the labeled data is determined as the mined data, which is used as the training set of the deep learning model for training.

[0025] In some embodiments, training the deep learning model according to the mined data comprises: determining an initial deep learning model; dividing the mined data into a training set and a validation set; training the initial deep learning model according to the training set to determine a training learning model; verifying the model performance of the deep learning model according to the validation set to determine a verification result; when the accuracy corresponding to the verification result reaches a preset target, determining the deep learning model according to the training learning model; and when the accuracy corresponding to the verification result does not reach the preset target, re-dividing the training set and the validation set.

[0026] In this embodiment, the deep learning model is trained by mining data, and the trained deep learning model is used to determine whether the reply text of the first user is generated by AI.

[0027] The embodiment of the second aspect of the application provides a text determination device, which comprises: a data acquisition module configured to determine original text data associated with a first user and a second user, the original text data comprising at least a reply text made by the first user in response to a question raised by the second user; a preprocessing module configured to perform data cleaning on the reply text to determine preprocessed data; a data source module configured to determine a historical database; a sample determination module configured to determine a plurality of labeled samples in the historical database; a label recognition module configured to determine at least one unlabeled sample in the preprocessed data; a data labeling module configured to label the unlabeled sample in the preprocessed data according to the labeled sample to determine labeled data; a data mining module configured to sample and evaluate the labeled data to determine mined data; a model training module configured to train a deep learning model according to the mined data; and a text determination module configured to input the reply text into the deep learning model to determine a determination result corresponding to the reply text, wherein the determination result is used to indicate whether the reply text is associated with a data source corresponding to the historical database.

[0028] The embodiment of the third aspect of the application provides an electronic device, which comprises a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the text determination method of the first aspect.

[0029] The embodiment of the fourth aspect of the application provides a readable storage medium, and the readable storage medium stores a program or instructions, and the program or instructions are executed by the processor to implement the steps of the text determination method of the first aspect.

[0030] The embodiment of the fifth aspect of the application provides a chip, which comprises a processor and a communication interface, the communication interface and the processor are coupled, the processor is used to run a program or instructions, and the steps of the text determination method of the first aspect are implemented.

[0031] Additional aspects and advantages of the technical solutions of the present application will become apparent from the following description with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flowchart of a text discrimination method according to one embodiment of the present application is shown; Figure 2 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown; Figure 3 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown; Figure 4 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown; Figure 5 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown; Figure 6 A partial flowchart of a text discrimination method according to one embodiment of the present application is shown; Figure 7 A structural schematic block diagram of an electronic device according to one embodiment of the present application is shown; Figure 8 A structural schematic block diagram of a text discrimination device according to one embodiment of the present application is shown; Figure 9 A flowchart of a text discrimination method according to one embodiment of the present application is shown; Figure 10 A flowchart of data mining according to one embodiment of the present application is shown.

[0033] wherein, Figure 7 and Figure 8 The correspondence between the reference signs and the component names in the drawings is as follows: 1000: electronic device; 1109: memory; 1110: processor; 900: text discrimination device; 902: data acquisition module; 904: preprocessing module; 906: data source module; 908: sample determination module; 910: label recognition module; 912: data labeling module; 914: data mining module; 916: model training module; 918: text discrimination module. DETAILED DESCRIPTION

[0034] In order to more clearly understand the above-mentioned purposes, features and advantages of the embodiments of the present application, the embodiments of the present application are further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0035] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but embodiments of the present application can also be implemented in other ways different from those described herein, and therefore the scope of protection of the present application is not limited to the specific embodiments disclosed below.

[0036] In the related art, the technology for identifying suspected AI-generated text mainly includes the following two types: I. Keyword matching and manual verification method: usually through a pre-set keyword library to screen the text, and combined with manual review to determine whether the text is AI-generated. This method relies on experience rules and manual judgment, and is low in efficiency, and has obvious processing bottlenecks when dealing with large-scale text data.

[0037] II. Machine learning model-based method: by collecting a large amount of manually annotated data, training a classification model (such as support vector machine, random forest, deep neural network, etc.) to distinguish AI-generated text from human-written text. Although the machine learning model-based method introduces automation, it still faces many challenges in practical application.

[0038] However, the two methods have the following defects: strong sample dependence, the machine learning model is highly dependent on high-quality large-scale manually annotated data, the data collection and annotation cost is high, and it is difficult to continuously iterate; weak generalization ability, the model's recognition ability decreases outside the training corpus, especially in scenarios with diverse language styles and domain backgrounds, the accuracy is insufficient; low recall rate, in actual use, the system cannot effectively identify all suspected AI-generated texts, and there is a high rate of missed detection, which is difficult to meet the needs of scenarios such as interviews, examinations, etc. that require high text authenticity; difficult to cope with variant expressions, as AI text generation technology continues to evolve (such as the popularity of large models like ChatGPT and Claude), AI-generated text is increasingly similar to human language style, and the recognition ability of traditional methods is gradually weakened.

[0039] Therefore, the related art still has many deficiencies in identifying suspected AI-generated text, and a new method with higher robustness, higher recall rate, and lower human dependence is needed to meet the needs of distinguishing text authenticity and originality in the interview scenario.

[0040] The text discrimination method, device, electronic equipment, readable storage medium and chip provided by the embodiments of the present application will be described in detail below in combination with the accompanying Figures 1 to 10 , through specific embodiments and application scenarios.

[0041] As shown in Figure 1 , the present embodiment provides a text discrimination method, the text discrimination method comprising: Step S100: determining original text data associated with the first user and the second user, the original text data at least including reply text made by the first user for a question raised by the second user; Step S102: data cleaning on the reply text to determine preprocessed data; Step S104: determining a historical database; Step S106: determining a plurality of labeled samples in the historical database; Step S108: determining at least one unlabeled sample in the preprocessed data; Step S110: labeling the unlabeled sample in the preprocessed data according to the labeled samples to determine labeled data; Step S112: sampling evaluation on the labeled data to determine mined data; Step S114: training a deep learning model according to the mined data; Step S116: inputting the reply text into the deep learning model to determine a discrimination result corresponding to the reply text; The discrimination result is used to represent whether the reply text is associated with a data source corresponding to the historical database.

[0042] The text discrimination method provided by the application analyzes the reply text in the original text data of the first user corresponding to the second user, judges whether the content in the reply text is associated with a data source corresponding to the historical database, and outputs the corresponding judgment result. The original text data of the first user corresponding to the second user includes a plurality of original question texts of the second user to the first user and reply texts corresponding to the original question texts. The reply text includes the answers of the first user to each original question text of the second user. Through data preparation and preprocessing, preprocessed data corresponding to the reply text is determined. At least one unlabeled sample in the preprocessed data is determined, i.e. a plurality of character strings or text contents. The unlabeled sample is labeled according to a plurality of labeled samples in the historical database to determine a plurality of mined data in the preprocessed data. The mined data all have labels and are used to distinguish whether they are associated with a data source corresponding to the historical database.

[0043] The plurality of labeled data in the mined data associated with the data source corresponding to the historical database and the plurality of labeled data in the mined data not associated with the data source corresponding to the historical database are used as a training set to train a deep learning model. After learning the training set, the deep learning model obtains the interaction information between each character in the reply text and other characters through a self-attention mechanism, judges the association degree between the plurality of labeled data associated with the data source corresponding to the historical database in the reply text through the interaction information, and determines the discrimination result of the reply text according to whether the reply text is associated with the data source corresponding to the historical database.

[0044] Understandably, through historical annotation samples and automated data mining and evaluation, automatic deep learning models reduce the cost of manual annotation, reduce the errors brought by manual review, improve the accuracy and robustness of text discrimination, and significantly reduce the missed detection rate to meet the requirements of text data discrimination.

[0045] Exemplarily, the first user is a candidate participating in an interview, and the second user is an interviewer conducting an assessment.

[0046] In some embodiments, optionally, the data source of the historical database corresponds to at least one artificial intelligence model.

[0047] In some embodiments, optionally, the plurality of annotation samples in the historical database are reply texts generated by the artificial intelligence model.

[0048] Specifically, after determining the reply texts of the candidate to the questions raised by the interviewer from the logs of the interview system, data cleaning is performed on the plurality of reply texts. Data cleaning includes deletion of stop words, deletion of punctuation marks, and handling of garbled codes, etc., to determine preprocessed data. By determining preprocessed data through data cleaning, the data processing amount is reduced, thereby improving the text discrimination efficiency.

[0049] After determining the plurality of annotation data, automatic review is performed on the newly labeled annotation data. Through sampling detection, it is determined whether the plurality of annotation data is the same as the text content in the preprocessed data, thereby realizing quality evaluation of the annotation data. If the quality meets the standard, the output is mining data, which is used for training of the deep learning model; if it does not meet the standard, data mining is performed again, and the plurality of unannotated samples are reannotated to improve the quality and robustness of the data set used for training.

[0050] In some embodiments, optionally, in the process of annotating the unannotated samples in the preprocessed data, small sample data is used to annotate the preprocessed data, data mining is performed on the unannotated samples, and mining data is determined to improve the efficiency of text discrimination.

[0051] In some embodiments, optionally, the discrimination result output by the model includes a classification label or a discrimination probability. The classification label is used to indicate whether the reply text is from an artificial intelligence model; the discrimination probability is used to indicate the probability that the reply text is generated by an artificial intelligence model.

[0052] In some embodiments, optionally, the reply text includes but is not limited to input text, voice input text, and handwritten image text of the first user.

[0053] In some embodiments, the raw text data may optionally include not only the plain text response but also metadata associated with the text content, such as the response timestamp, the delay time from the question being displayed to the start of the response, the number of edits during the response process, and cursor activity logs. A comprehensive judgment can be made by combining the response text with multiple metadata sources, and an accurate judgment result can be determined based on the judgment result and metadata to improve the accuracy of text judgment in interview scenarios.

[0054] In some embodiments, optionally, the step of labeling unlabeled samples is performed based on a configurable mining rule base, which includes multiple mining strategies. The mining data corresponding to the preprocessed data is determined by determining a combination of one or more mining strategies based on the text type and text features of the unlabeled samples, so as to improve the correlation between the mining data and the response text.

[0055] In some embodiments, optionally, after judging the first user's reply text and generating the judgment result, the text content from the artificial intelligence model in the first user's reply text is stored in the historical database to realize the automatic updating of the historical database.

[0056] In some embodiments, optionally, such as Figure 2 As shown, step S110: Based on the labeled samples, label the unlabeled samples in the preprocessed data to determine the labeled data, including: Step S1100: Determine the original question text in the original text data that corresponds to the response text; Step S1102: Extract keywords from preprocessed data and determine the keyword set; Step S1104: Determine the semantic relevance between the keyword set and the original question text; Step S1106: Determine the preset association threshold; Step S1108: Label the unlabeled samples according to the semantic relevance and the preset relevance threshold to determine the labeled data; Specifically, when the semantic relevance is greater than or equal to the preset relevance threshold, the unlabeled sample is labeled with the first label; when the semantic relevance is less than the preset relevance threshold, the unlabeled sample is labeled with the second label.

[0057] In this embodiment, important words corresponding to the original question text are extracted from the response text using an important word model. The response text is then queried among multiple unlabeled samples, and unlabeled samples with semantic relevance greater than a preset relevance threshold are labeled. The labels are used to reflect whether the response text was generated by an artificial intelligence model.

[0058] The label includes a first label and a second label, and semantics of the first label and the second label are opposite. The first label indicates that a semantic association degree between the reply text and the original question text is greater than or equal to a preset association threshold, that is, the reply content corresponding to the reply text is associated with the question in the original question text, and the reply text is not generated by the artificial intelligence model. The second label indicates that the semantic association degree between the reply text and the original question text is less than the preset association threshold, that is, the reply content corresponding to the reply text is irrelevant to the question in the original question text, and the reply text is generated by the artificial intelligence model.

[0059] Understandably, according to the characteristics of the AI large model in generating answers, the key words of the reply text are extracted from the perspective of the association degree between the question and the answer, the logical relevance between the question text and the reply text is judged, and the generalization ability and robustness of the text discrimination in the interview scene are improved.

[0060] The key words of the reply text are extracted, that is, the features of the answer are extracted, the key word extraction technology in natural language processing (NLP) (for example, based on TF-IDF, TextRank algorithm or more advanced deep learning model) is used to process the cleaned reply text. The purpose is to extract the word group that best represents the core theme and content of the reply text, form a key word set, and the key word set reflects the essential content of the question text.

[0061] In some embodiments, the key words in the reply text are extracted by an important word vector model.

[0062] Specifically, the key word set and the original question text are input into a semantic representation model (for example, a sentence embedding model generated by Word2Vec, GloVe, BERT, etc.), and the semantic representation model is used to convert the text into a vector in a high-dimensional space, which captures the semantic information of the original question text.

[0063] In some embodiments, the semantic association degree is obtained by calculating the cosine similarity or other distance measurement between the two vectors. The higher the value, the higher the association degree between the key words in the reply text and the original question text.

[0064] In some embodiments, the preset association threshold is determined according to experimental data corresponding to a plurality of labeled samples in a historical database, and the preset association threshold is a critical value for dividing text content and artificial intelligence model related or unrelated.

[0065] In some embodiments, optionally, the keyword extraction process in the reply text uses a domain-specific dictionary (such as a technical term library, an industry slang dictionary) for weighting to ensure that the extracted keywords have higher weights and representativeness in the context related to the interview field.

[0066] In some embodiments, optionally, the preset correlation threshold is not a fixed value, but is dynamically adjusted according to the complexity, length or field of the original question. For example, for a complex open question, the preset correlation threshold can be appropriately reduced; for a simple factual question, the preset correlation threshold is increased.

[0067] In some embodiments, optionally, the label annotated for the unlabeled sample is a continuous probability value (for example, a probability of 0.7 is generated by AI), which can be mapped by a activation function (sigmoid) from the semantic correlation degree to provide more rich information for the subsequent fusion model.

[0068] In some embodiments, optionally, as shown in Figure 3 Step S110: according to the labeled sample, annotating the unlabeled sample in the preprocessed data to determine the annotated data, further comprising: Step S1110: determining a text similarity model; Step S1112: inputting the labeled sample and at least one unlabeled sample into the text similarity model to determine the text similarity of the labeled sample and the unlabeled sample; Step S1114: determining a similarity threshold according to the historical database; Step S1116: annotating the unlabeled sample according to the text similarity and the similarity threshold to determine the annotated data; Wherein, when the text similarity is greater than or equal to the similarity threshold, the second label is annotated for the unlabeled sample; when the text similarity is less than the similarity threshold, the first label is annotated for the unlabeled sample.

[0069] In this embodiment, the similarity calculation model is used to calculate the text similarity of the labeled sample in the historical database and the unlabeled sample in the preprocessed data, and the unlabeled sample is labeled by comparing the text similarity with the similarity threshold.

[0070] It is understandable that the artificial intelligence model will mark a certain pattern or tend to use a specific template or sentence pattern when generating answers, especially for common questions. By batch identifying text highly similar to known AI-generated samples, i.e. labeled samples, the unlabeled samples are labeled, making the mined data more robust. Moreover, by determining the label through the similarity value, the discrimination result is more convincing and interpretable, facilitating manual review. For example, the reply text is highly similar to a certain known AI-generated answer (similarity of 92%).

[0071] In particular, the text similarity model is a trained calculation model that can quantify the similarity between two text segments. This model can be a traditional natural language processing-based model, such as cosine similarity calculation based on TF-IDF vectors. It can also be a more advanced deep learning model, such as a sentence embedding model generated by Sentence-BERT, SimCSE, etc. The choice of model determines the accuracy and robustness of similarity calculation.

[0072] Select one or more known labeled samples, i.e. labeled samples, from the historical database, such as samples known to be AI-generated or human-written, and input them together with the unlabeled sample to be labeled into the similarity model. The model will output a numerical similarity score (e.g. between 0 and 1), which objectively reflects the distance or closeness of the two texts in the feature space understood by the model.

[0073] In some embodiments, optionally, by calculating the average similarity between the same class samples (AI-generated samples) in the historical database and the average similarity between them and the different class samples (human-written samples), a threshold value that best distinguishes the two classes of samples is determined, i.e. the similarity threshold.

[0074] If the unlabeled sample is highly similar to a certain known labeled sample, it is very likely to come from the same source or have the same attribute. Therefore, it is given the same second label as the known sample.

[0075] If the unlabeled sample is not similar to any known labeled sample in the historical database, it means it is a new and different expression. In the absence of evidence that it is AI-generated, it is defaulted to the first label.

[0076] In some embodiments, optionally, the labeled samples in the historical database cover common AI-generated patterns.

[0077] In some embodiments, optionally, multiple labeled samples are compared with an unlabeled sample. Finally, the maximum similarity or average similarity is taken as the final similarity value of the unlabeled sample.

[0078] In one embodiment, optionally, when human review finds that the automatic annotation is wrong, the wrong sample and its correct label are added to the historical database, and a threshold value recalculation is triggered, so that the system can continuously evolve and cope with new AI models.

[0079] In one embodiment, optionally, the output of the annotation step is not only a binary label, but also includes a confidence score (for example, the text similarity itself or a calibrated probability). This confidence score can be used as a weight for downstream processes, or used to filter high-confidence samples to directly add to the training set.

[0080] In some embodiments, optionally, as shown in FIG. 1 1, the step S110 of annotating the unannotated samples in the preprocessed data according to the annotated samples to determine annotated data further includes: Figure 4 Step S1130: determining a plurality of character strings corresponding to the unannotated sample; Step S1132: determining the reply text corresponding to the character string; Step S1134: determining the character type of the reply text corresponding to the character string; Step S1136: replacing the character string according to the character type to determine a replacement sample; Step S1138: determining the annotated data according to the replacement sample.

[0081] In this embodiment, annotating the unannotated samples in the preprocessed data further includes constructing new samples by replacing entity words in the unannotated samples, determining replacement samples, and determining annotated data corresponding to the replacement samples.

[0082] It can be understood that when generating text, AI large models often produce synonymous replacements, numerical format conversions, and other behaviors. By actively replacing character strings in the reply text, the replacement behavior of AI is simulated, and new training samples are automatically generated from unannotated data. When encountering new AI-generated text, identification is performed according to the replacement sample, so that the text discrimination method is more robust and general.

[0083] Specifically, the keyword extraction technology in NLP is used to determine the character string that can maintain the meaning of the sentence after replacement from a plurality of unannotated samples. The character type of the target character string in the specific context of the reply text to which it belongs is determined, for example, Chinese numerals, Arabic numerals, full-width characters, half-width characters, abbreviations, or complete terms, etc. According to the analyzed character type, a preset replacement rule library is called to replace the original character string.

[0084] For example, part of the quantifier in Chinese numerals is changed to Arabic numerals, or formal terms are replaced by abbreviations. ​

[0085] A replacement sample that is different in surface form but consistent in core semantics from the original string is generated by replacing the original string, and the replacement sample is labeled, with the label inherited from the original unlabeled sample, to determine the labeled data.

[0086] In some embodiments, the mining strategies include, but are not limited to, semantic correlation comparison, text similarity comparison, and sample replacement. The mining data corresponding to the preprocessed data is determined by one or more combinations of mining strategies.

[0087] In some embodiments, as shown in Figure 5 Step S112: sampling and evaluating the labeled data to determine the mining data, including: Step S1120: randomly sampling the plurality of labeled data to determine an evaluation sample set; Step S1122: determining the labeling completeness parameter of each labeled data in the evaluation sample set; Step S1124: determining the quality threshold; Step S1126: when the labeling completeness parameter is greater than or equal to the quality threshold, determining the labeled data as the mining data; Step S1128: when the labeling completeness parameter is less than the quality threshold, randomly sampling the plurality of labeled data to determine an evaluation sample set again until the labeling completeness parameter is greater than or equal to the quality threshold.

[0088] In this embodiment, after determining the labeled data, the newly labeled preprocessed data is automatically verified. The labeling completeness parameter of each labeled data is compared with the quality threshold. If it does not meet the standard, i.e., the labeling completeness parameter is less than the quality threshold, the mining rule is further optimized, and the unprocessed sample is labeled again to determine new labeled data until the labeling completeness parameter is greater than or equal to the quality threshold. If it meets the standard, i.e., the labeling completeness parameter is greater than or equal to the quality threshold, the labeled data is determined as the mining data, and the mining data is used as the training set of the deep learning model for training.

[0089] It can be understood that the amount of automatically labeled data can be very large, and the cost of manual evaluation is high. By using the random sampling method, a statistically representative evaluation sample set is extracted from the large-scale labeled data for quality inspection, ensuring that only high-quality data can enter the model training stage, thereby ensuring the accuracy, reliability, and stability of the final discrimination model from the source.

[0090] The labeling completeness parameter includes one or more of the following indicators: the proportion of samples with consistent labels and correct labels in the automatic labeling process, the probability value or similarity score output by the model in the automatic labeling process, and the completeness of the plurality of labeled data corresponding to the preprocessed data.

[0091] In some embodiments, optionally, the quality threshold is a minimum data quality standard acceptable for model training.

[0092] In some embodiments, optionally, as shown in FIG. 11B, step S114: training the deep learning model according to the mined data, comprises: Figure 6 Step S1140: determining an initial deep learning model; Step S1142: dividing the mined data into a training set and a validation set; Step S1144: training the initial deep learning model according to the training set to determine a training learning model; Step S1146: verifying the model performance of the deep learning model according to the validation set to determine a verification result; Step S1148: when the accuracy corresponding to the verification result reaches a preset target, determining the deep learning model according to the training learning model; Step S1150: when the accuracy corresponding to the verification result does not reach the preset target, re-dividing the training set and the validation set.

[0093] In this embodiment, the deep learning model is trained by the mined data, and the trained deep learning model is used to determine whether the reply text of the first user is generated by AI.

[0094] The initial deep learning model is a pre-trained model (Pre-trained Model). For example, an ALBERT, BERT or the like model pre-trained on a large number of general texts is used as a starting point. This is equivalent to letting the model already have general language understanding ability, and we only need to fine-tune it on this basis to adapt to the specific task of distinguishing AI text, which is faster and better than training the model from scratch.

[0095] The high-quality mined data is randomly divided into two parts in a certain proportion (such as 8:2): a training set (Training Set) for directly updating model parameters to let the model learn the feature patterns in the data; and a validation set (Validation Set) for evaluating the performance of the model on unseen data, monitoring the training process to prevent overfitting (Overfitting), and serving as a basis for decision-making whether to stop training.

[0096] The training set data is input into the initial model, and the model internal parameters are iteratively adjusted through back propagation and optimization algorithms (such as AdamW) to minimize the prediction error or loss function, and finally the training learning model is obtained.

[0097] ​The trained learning model is tested using validation set data that was not used in the training process. The performance of the trained learning model is measured by metrics such as accuracy, recall, and score; the overall evaluation of these metrics constitutes the validation result.

[0098] When the accuracy of the validation results reaches the preset target, it indicates that the model has successfully learned and its generalization performance has reached the level required by the business. At this point, the training process is complete, and the current trained model is determined as the final usable deep learning model. When the accuracy of the validation results does not reach the preset target, it indicates that the training effect has not met expectations. In this case, the training set and validation set are re-divided, and the trained model is retrained to reduce random errors and improve the robustness of text discrimination.

[0099] For example, the deep learning model uses the Albert model. Compared to the BERT deep learning model, the Albert model reduces the overall number of parameters, speeds up training, and improves model performance. The Albert model uses a self-attention mechanism to acquire interaction information between each character and other characters, thus better representing the text.

[0100] In some embodiments, the discrimination result output by the deep learning model may optionally include the probability of a specified label.

[0101] In some embodiments, optionally, multiple different designated labels may be determined according to different business scenarios.

[0102] In one specific embodiment, optionally, the text discrimination method of the present invention mainly includes four stages, such as... Figure 9 As shown, after determining the interview log data source in step S200, the four stages are as follows: Step S202: Obtain the data source and clean the data in the data source; Step S204: Label and mine the data; Step S206: Model training and optimization; Step S208: Output the category corresponding to the interview log data source.

[0103] In step S202: obtaining the data source and cleaning the data in the data source, a large amount of answer text is obtained from the logs of the business's AI interview process. This text needs to be cleaned, including the removal of stop words, punctuation marks, and garbled text. Data determines the upper limit of the model, so preprocessing the data is necessary.

[0104] The main process in the data mining stage is to label and preprocess the data with small sample data, and then perform data mining on the samples. There are three main mining methods.

[0105] 1. Through the replacement of entity words on the sample, a new sample is constructed, for example, part of the quantifier is changed into Arabic numerals; 2. Using the existing similarity calculation model, the similarity of the labeled sample and the unlabeled sample is calculated, and the unlabeled sample under a certain threshold is labeled; 3. The important words of the text are extracted through the important word model, and the labeled sample is queried in the unlabeled sample.

[0106] After mining, the newly labeled data is manually checked, and if it does not meet the standard, the mining rules will be further optimized, and finally the data mining is completed.

[0107] The flowchart of data mining is shown in Figure 10 , which includes: Step S300: Determine the preprocessed data; Step S302: Small sample data labeling; Step S304: Data mining; Step S306: Data sampling evaluation is performed to determine whether it meets the standard, and when the result of step S306 is no, return to step S304 until the sampling evaluation meets the standard; When the result of step S306 is yes, execute step S308: training data mining is completed.

[0108] In the model training and optimization stage, the model mainly uses a deep learning model (Albert). Compared with the deep learning model (Bert), Albert reduces the overall parameter quantity, speeds up the training speed, and increases the model effect. Mainly in reducing the word mapping dimension, from the experiment, the size of the word embedding (embedding) is not completely positively correlated with the effect, so 64 dimensions are selected; and cross-layer parameter sharing, whether it is stacked 12 layers or 24 layers, an encoder is selected. Compared with the shallow model, Albert can better obtain the text semantics, and through the self-attention mechanism, the interaction information of each word with other words can be obtained, and the text can be better expressed.

[0109] In the output corresponding to the interview log data source category stage, the output category is the probability of predicting the specified label (label) by the model, the label concerned is selected according to the specific business scenario, and the data set corresponding to the label is obtained through the threshold.

[0110] Understandably, the present application mainly reduces the cost of manual labeling, provides more recall data, anti-cheating for interviewers suspected of AI generated text, ensures the fairness of the interview, and improves the efficiency and quality of human resources (Human Resources, HR) interview screening.

[0111] As Figure 8 shown, the embodiment of the present application also provides a text discrimination device 900, which comprises: a data acquisition module 902, configured to determine original text data associated with a first user and a second user, the original text data at least comprising reply text made by the first user to a question raised by the second user; a preprocessing module 904, configured to perform data cleaning on the reply text to determine preprocessed data; a data source module 906, configured to determine a historical database; a sample determination module 908, configured to determine a plurality of labeled samples in the historical database; a label identification module 910, configured to determine at least one unlabeled sample in the preprocessed data; a data labeling module 912, configured to label the unlabeled sample in the preprocessed data according to the labeled samples to determine labeled data; a data mining module 914, configured to perform sampling evaluation on the labeled data to determine mined data; a model training module 916, configured to train a deep learning model according to the mined data; and a text discrimination module 918, configured to input the reply text into the deep learning model to determine a discrimination result corresponding to the reply text; wherein the discrimination result is used to indicate whether the reply text is associated with a data source corresponding to the historical database.

[0112] As Figure 7 shown, the embodiment of the present application also provides an electronic device 1000, which comprises a processor 1110, a memory 1109, programs or instructions stored on the memory 1109 and executable on the processor 1110, the programs or instructions being executed by the processor 1110 to implement various processes of the above-mentioned embodiments of the text discrimination method and achieve the same technical effects. To avoid repetition, details are not described herein.

[0113] Optionally, the processor 1110 is configured to determine original text data associated with a first user and a second user, the original text data at least comprising reply text made by the first user to a question raised by the second user. Optionally, the processor 1110 is further configured to perform data cleaning on the reply text to determine preprocessed data. Optionally, the processor 1110 is further configured to determine a historical database. Optionally, the processor 1110 is further configured to determine a plurality of labeled samples in the historical database. Optionally, the processor 1110 is further configured to determine at least one unlabeled sample in the preprocessed data. Optionally, the processor 1110 is further configured to label the unlabeled sample in the preprocessed data according to the labeled samples to determine labeled data. Optionally, the processor 1110 is further configured to perform sampling evaluation on the labeled data to determine mined data. Optionally, the processor 1110 is further configured to train a deep learning model according to the mined data. Optionally, the processor 1110 is further configured to input the reply text into the deep learning model to determine a discrimination result corresponding to the reply text.

[0114] The memory 1109 can be configured to store software programs and various data. The memory 1109 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 1109 can include a volatile memory or a non-volatile memory, or the memory 1109 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 1109 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0115] The embodiments of the present application further provide a readable storage medium, and the readable storage medium stores programs or instructions. When the programs or instructions are executed by a processor, each process of the above-mentioned text discrimination method embodiments is realized, and the same technical effects can be achieved. To avoid repetition, details are not described herein. In addition, the readable storage medium improves the data storage capacity and data processing speed of the text discrimination method of the present application.

[0116] The readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media, or electrical signals transmitted through a wire.

[0117] The processor is a processor in the electronic device in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like.

[0118] The chip provided by the embodiment of the present application includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is used to run programs or instructions to realize the processes of the above text discrimination method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein. In addition, the chip improves the data processing speed of the text discrimination method of the present application.

[0119] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip, etc.

[0120] In the present application, the terms "first", "second", "third" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance; the term "multiple" refers to two or more, unless otherwise explicitly limited. The terms "mounting", "connecting", "connecting", "fixing" and the like should be understood in a broad sense, for example, "connecting" can be fixed connection, or detachable connection, or integrally connected; "connected" can be directly connected, or indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0121] In the description of the application, it should be understood that the terms "upper", "lower", "left", "right", "front", "rear", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the application and simplifying the description, and do not indicate or imply that the device or unit referred to must have a particular direction, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application.

[0122] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "a specific embodiment" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The above is only the preferred embodiment of the present application, and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A text discrimination method, characterized in that, include: Determine the raw text data associated with the first user and the second user, the raw text data including at least the text of the first user's response to a question raised by the second user; The response text is cleaned to determine the preprocessed data; Determine the historical database; Identify multiple labeled samples in the historical database; Identify at least one unlabeled sample in the preprocessed data; Based on the labeled samples, the unlabeled samples in the preprocessed data are labeled to determine the labeled data; The labeled data is sampled and evaluated to determine the data to be mined; A deep learning model is trained based on the mined data; The response text is input into the deep learning model to determine the discrimination result corresponding to the response text; The discrimination result is used to indicate whether the response text is associated with the data source corresponding to the historical database.

2. The text discrimination method according to claim 1, characterized in that, The step of labeling unlabeled samples in the preprocessed data based on the labeled samples to determine the labeled data includes: Determine the original question text in the original text data that corresponds to the response text; Extract keywords from the preprocessed data to determine the keyword set; Determine the semantic relevance between the keyword set and the original question text; Determine the preset association threshold; The unlabeled samples are labeled according to the semantic relevance and the preset relevance threshold to determine the labeled data; Specifically, when the semantic relevance is greater than or equal to the preset relevance threshold, the unlabeled sample is labeled with a first label; When the semantic relevance is less than the preset relevance threshold, a second label is assigned to the unlabeled sample.

3. The text discrimination method according to claim 2, characterized in that, The step of labeling unlabeled samples in the preprocessed data based on the labeled samples to determine the labeled data further includes: Determine the text similarity model; The labeled sample and at least one unlabeled sample are input into the text similarity model to determine the text similarity between the labeled sample and the unlabeled sample; A similarity threshold is determined based on the historical database; The unlabeled samples are labeled based on the text similarity and the similarity threshold to determine the labeled data; Specifically, when the text similarity is greater than or equal to the similarity threshold, the unlabeled sample is labeled with the second label; When the text similarity is less than the similarity threshold, the first label is assigned to the unlabeled sample.

4. The text discrimination method according to claim 1, characterized in that, The step of labeling unlabeled samples in the preprocessed data based on the labeled samples to determine the labeled data further includes: Identify multiple strings corresponding to the unlabeled samples; Determine the response text corresponding to the string; Determine the text type of the response text corresponding to the string; Replace the string according to the text type to determine the replacement sample; The labeled data is determined based on the replacement sample.

5. The text discrimination method according to claim 1, characterized in that, The step of sampling and evaluating the labeled data to determine the data to be mined includes: Random sampling is performed on multiple labeled data sets to determine the evaluation sample set; Determine the complete annotation parameters for each of the labeled data in the evaluation sample set; Determine the quality threshold; When the complete annotation parameters are greater than or equal to the quality threshold, the annotation data is determined to be mining data; When the completeness parameter of the annotation is less than the quality threshold, the multiple annotation data are randomly sampled again to determine the evaluation sample set until the completeness parameter of the annotation is greater than or equal to the quality threshold.

6. The text discrimination method according to any one of claims 1 to 5, characterized in that, The step of training a deep learning model based on the mined data includes: Determine the initial deep learning model; The mined data is divided into a training set and a validation set; The initial deep learning model is trained based on the training set to determine the training learning model; The performance of the deep learning model is verified based on the verification set, and the verification result is determined. When the accuracy corresponding to the verification result reaches the preset target, a deep learning model is determined based on the training learning model. When the accuracy corresponding to the verification result does not reach the preset target, the training set and the verification set are re-divided.

7. A text discrimination device, characterized in that, include: The data acquisition module is used to determine the original text data associated with the first user and the second user, wherein the original text data includes at least the text of the first user's reply to the question raised by the second user; The preprocessing module is used to clean the response text and determine the preprocessed data. The data source module is used to determine the historical database; A sample determination module is used to determine multiple labeled samples in the historical database; The annotation and recognition module is used to identify at least one unlabeled sample in the preprocessed data; The data annotation module is used to annotate the unannotated samples in the preprocessed data according to the annotated samples, and determine the annotated data; The data mining module is used to sample and evaluate the labeled data to determine the data to be mined. The model training module is used to train a deep learning model based on the mined data; The text discrimination module is used to input the response text into the deep learning model and determine the discrimination result corresponding to the response text; The discrimination result is used to indicate whether the response text is associated with the data source corresponding to the historical database.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the text discrimination method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the text discrimination method as described in any one of claims 1 to 6.

10. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the text discrimination method as described in any one of claims 1 to 6.

Images