Text detection method, training method, device, apparatus, medium, and program

Abstract

To provide a text detection method, a training method, a device, an apparatus, a medium, and a program.SOLUTION: A training method includes processing a sample text pair and a sample tag by using an initial model in response to the reliability of a first sample detection result being lower than a prescribed threshold and obtaining a second sample detection result, obtaining a first loss value based on the second sample detection result and the sample tag on the basis of a first loss function, and adjusting a model parameter of the initial model on the basis of the first loss value and obtaining a trained target model. The reliability of the first sample detection result is obtained by detection of the sample text pair by using a first model. The sample text pair includes a sample inquiry text and a sample answer text. The second sample detection result includes a determination result and a determination factor for indicating an answer matching degree between the sample answer text and the sample inquiry text.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as large-scale language models, deep learning, and text processing, and more specifically to a text detection method, a training method, an apparatus, a device, a medium, and a program. [Background technology]

[0002] With the spread of Internet technology, a user can post a question via the Internet and obtain answers to the question from other users.

[0003] Since the questions relate to many fields, the phenomenon of "answers being incorrect" or "questions not being answered" often occurs. Therefore, a method is required that can accurately detect the matching degree between answers and questions, thereby improving the user experience. Summary of the Invention

[0004] The present disclosure provides a text detection method, a training method, an apparatus, a device, a medium, and a program.

[0005] According to an aspect of the present disclosure, a model training method is provided, which includes: in response to a confidence level of the first sample detection result being smaller than a predetermined threshold, processing the sample text pairs and the sample tags using an initial model to obtain a second sample detection result; obtaining a first loss value based on the second sample detection result and the sample tags based on a first loss function; and adjusting model parameters of the initial model based on the first loss value to obtain a trained target model, where the confidence level of the first sample detection result is obtained by detecting the sample text pairs using the first model, the sample text pairs include corresponding sample question texts and sample answer texts, the second sample detection result includes a discrimination result for representing an answer matching degree between the sample answer text and the sample question text and a discrimination cause corresponding to the discrimination result, the sample tags include a discrimination result tag and a discrimination cause tag, and the initial model is a type different from the first model.

[0006] According to another aspect of the present disclosure, there is provided a text detection method, including: detecting a plurality of to-be-detected text pairs using a first model, and obtaining a first detection result and a confidence corresponding to the first detection result, wherein the first model is a discriminant model, and the plurality of to-be-detected text pairs include a plurality of question texts and a plurality of answer texts corresponding to each other; determining a target to-be-detected text pair from the plurality of to-be-detected text pairs based on the confidence, wherein the confidence of the detection result corresponding to the target to-be-detected text pair is smaller than a predetermined threshold; detecting a target to-be-detected text pair using a second model, and obtaining a second detection result, wherein the second model is a trained target model obtained by the training method according to any one of claims 1-4; and generating a target detection result representing an answer matching degree between the answer texts and the corresponding question texts based on the first detection result and the second detection result.

[0007] According to another aspect of the present disclosure, there is provided a model training apparatus including a first processing module, a first loss calculation module, and a first adjustment module.

[0008] The first processing module, in response to the confidence of the first sample detection result being smaller than a predetermined threshold, processes the sample text pair and the sample tag using an initial model to obtain a second sample detection result, where the confidence of the first sample detection result is obtained by detecting the sample text pair using the first model, the sample text pair includes a sample question text and a sample answer text corresponding to each other, the second sample detection result includes a discrimination result for representing an answer matching degree between the sample answer text and the sample question text and a discrimination cause corresponding to the discrimination result, the sample tag includes a discrimination result tag and a discrimination cause tag, and the initial model is different from a type of the first model.

[0009] The first loss calculation module obtains a first loss value based on the second sample detection result and the sample tag based on the first loss function.

[0010] The first adjustment module adjusts model parameters of the initial model based on the first loss value to obtain a trained target model.

[0011] According to another aspect of the present disclosure, there is provided a text detection apparatus including a second detection module, a second determination module, a third detection module, and a generation module.

[0012] The second detection module detects a plurality of text pairs to be detected using a first model, obtains a first detection result and a confidence corresponding to the first detection result, where the first model is a discriminant model, and the plurality of text pairs to be detected include a plurality of question texts and a plurality of answer texts corresponding to each other.

[0013] The second determination module determines a target text pair to be detected from the plurality of text pairs to be detected based on the confidence level, and the confidence level of the detection result corresponding to the target text pair to be detected is smaller than a predetermined threshold value.

[0014] The third detection module detects the text pairs to be targeted by using the second model to obtain a second detection result, where the second model is the trained target model obtained by the above training method.

[0015] The generation module generates a target detection result representing an answer matching degree between the answer text and the corresponding question text based on the first detection result and the second detection result.

[0016] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-mentioned method.

[0017] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions that cause the computer to perform the method set forth above.

[0018] According to another aspect of the present disclosure, there is provided a computer program product which, when executed by a processor, implements the above method.

[0019] It should be understood that the subject matter described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following specification. [Brief description of the drawings]

[0020] The drawings are provided for a better understanding of the invention and are not intended to limit the disclosure. [Figure 1] FIG. 1 illustrates a schematic diagram of an exemplary system architecture in which the model training method or the text detection method and apparatus according to the embodiments of the present disclosure can be applied. [Diagram 2] FIG. 2 illustrates a flowchart of a model training method according to an embodiment of the present disclosure. [Figure 3A] FIG. 3A illustrates a schematic diagram of model training according to an embodiment of the present disclosure. [Figure 3B] FIG. 3B illustrates a schematic diagram of model training according to another embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates a schematic flow chart of a text detection method according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 illustrates a schematic diagram of text detection according to an embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates a block diagram of a model training apparatus according to an embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates a block diagram of a text detection device according to an embodiment of the present disclosure. [Figure 8] FIG. 8 illustrates a block diagram of an electronic device suitable for implementing the model training method or the text detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings, and various details of the embodiments of the present disclosure will be included for ease of understanding, which are merely illustrative. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the following description will omit the description of known functions and structures.

[0022] In the scenario of determining the quality of text question answering, there are many fields related to the question text in the sample data. In the process of training the discriminant model multiple times based on a large amount of sample data, the model is prone to overfitting.

[0023] If the input question-answer text pair to be detected deviates slightly from the sample data distribution, the model inference performance fluctuates significantly, and the accuracy of the model output results decreases significantly.

[0024] In addition, the model in the related example detects text question answering quality based on surface features such as the linguistic fluency of the answer text and the lexical relationships between the question and the answer, and does not adequately process deep features such as the logical relevance between the question and the answer, resulting in low model accuracy.

[0025] Generative Large Language Models (LLMs) are natural language processing models with pre-trained logical reasoning capabilities using deep learning techniques.

[0026] Therefore, an embodiment of the present disclosure provides a model training method, and uses the discrimination results and discrimination causes as tags to guide an initial model to detect based on deep features such as logical relationships between question texts and answer texts, thereby improving the model accuracy, realizing accurate detection of sample text pairs that are difficult for the first model to detect, and obtaining quality detection results of question-answer text pairs with high confidence.

[0027] FIG. 1 illustrates a schematic diagram of an exemplary system architecture in which the model training method or the text detection method and apparatus according to the embodiments of the present disclosure can be applied.

[0028] In order for those skilled in the art to understand the technical contents of the present disclosure, Fig. 1 is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, and does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments, or scenes. For example, in other embodiments, an exemplary system architecture to which the model training method and device can be applied may include a terminal device, but the terminal device does not need to interact with a server and can realize the model training method and device provided by the embodiments of the present disclosure.

[0029] 1, the system architecture 100 according to the present embodiment includes a terminal device 110, a first server 150_1, and a second server 150_2. The terminal device 110 may be various electronic devices having processing capabilities, including, but not limited to, smartphones, tablet computers, laptop portable computers, desktop computers, and smart wearable devices.

[0030] The first server 150_1 executes a training method for a first model in the model training method provided by the embodiment of the present disclosure to obtain a discriminant model 140_1. The second server 150_2 executes a training method for a target model in the model training method provided by the embodiment of the present disclosure to obtain a generative formula model 140_2.

[0031] The terminal device can load the trained discriminant model 140_1 and the trained generative model 140_2, and process the question text answer text 120 based on the loaded discriminant model 140_1 to obtain a preliminary detection result. The question text answer text whose reliability in the preliminary detection result is lower than a predetermined threshold can be processed using the generative model 140_2 to obtain a detection result 130.

[0032] It should be noted that the model training method or the text detection method according to the embodiments of the present disclosure may generally be performed by the terminal device 110. Accordingly, the model training device or the text detection device provided by the embodiments of the present disclosure may be provided in the terminal device 110.

[0033] Alternatively, the model training method or text detection method provided by the embodiment of the present disclosure may be generally performed by the first server 150_1 and the second server 150_2. Accordingly, the model training device or text detection device provided by the embodiment of the present disclosure may be generally installed in the first server 150_1 and the second server 150_2. The model training method or text detection method provided by the embodiment of the present disclosure may be performed by a server or server cluster different from the first server 150_1 and the second server 150_2 and capable of communicating with the terminal device 110 and / or the first server 150_1 and the second server 150_2. Accordingly, the model training device or text detection device provided by the embodiment of the present disclosure may be installed in a server or server cluster different from the first server 150_1 and the second server 150_ and capable of communicating with the terminal device 110 and / or the first server 150_1 and the second server 150_2.

[0034] It should be understood that the number of terminal devices and servers in Figure 1 is merely an example, and any number of terminal devices and servers may be included as desired.

[0035] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, application and other processing of such user personal information shall all comply with the provisions of relevant laws and regulations, take necessary security measures, and not violate public order and morals.

[0036] In the technical solution disclosed herein, the user's approval or consent is obtained before obtaining or collecting the user's personal information.

[0037] FIG. 2 illustrates a flowchart of a model training method according to an embodiment of the present disclosure.

[0038] As shown in FIG. 2, the method 200 includes operations S210 to S230.

[0039] In operation S210, in response to the confidence of the first sample detection result being less than a predefined threshold, the sample text pairs and the sample tags are processed using the initial model to obtain a second sample detection result.

[0040] In operation S220, a first loss value is obtained based on the first loss function, based on the second sample detection result and the sample tag.

[0041] In operation S230, based on the first loss value, adjust model parameters of the initial model to obtain a trained target model.

[0042] According to an embodiment of the present disclosure, the initial model is different from the type of the first model. The initial model may be a generative model, such as a sentence mind model, a large language model (Large Language Model Meta AI, LlaMA), a generative pre-trained transformer (GPT) model, etc. The first model may be a discriminant model, such as a convolutional neural network model, an enhanced language representation with informative entities (ERNIE), etc.

[0043] According to an embodiment of the present disclosure, the first model and the initial model may be trained in two stages, and in the first stage, the first model may be supervised trained using the initial sample text to obtain a trained first model, so that the loss value between the result output by the first model and the manually labeled result tag reaches a convergence condition. The convergence condition may be a maximum number of training times or a predetermined loss value, and the present disclosure is not specifically limited thereto.

[0044] After the first stage training is completed, the trained first model processes the initial sample text pairs to obtain a first sample detection result. The first sample detection result may include a discrimination result corresponding to the initial sample text pairs and a confidence level corresponding to the discrimination result. The confidence level corresponding to the discrimination result may represent the discrimination difficulty of the first model for the initial sample text pairs.

[0045] For example, the initial sample text pairs include multiple text pairs consisting of sample question texts and sample answer texts: Q1-A1, Q2-A2, …, Q n -A n The first sample detection results obtained by processing the initial sample text pairs using the first model may include Q1-A1 (I, 95%), Q2-A2 (II, 55%), ..., Q n -A n (III, 75%), where I indicates that the answer text A1 does not solve the question text Q1, II indicates that the answer text A2 partially solves the question text Q2, and III indicates that the answer text A n is the question text Q n It can be shown that the method perfectly solves the above problem. 95%, 55%, and 75% respectively indicate the confidence level of the detection result corresponding to each text pair by the first model, and the higher the confidence level, the lower the difficulty of the first model in discriminating the text pair, and the more accurate the detection result.

[0046] Therefore, by setting a predetermined threshold, based on the first sample detection result, sample text pairs whose confidence is smaller than the predetermined threshold are screened out and used for training the generative formula model in the second stage.

[0047] For example, the predetermined threshold may be 80%, and the sample text pairs screened based on the first sample detection result are at least Q2-A2(II, 55%), ..., Q n -A n (III, 75%).

[0048] According to an embodiment of the present disclosure, before starting training in the second stage, the manually marked digital discrimination result tags I-III corresponding to the screened sample text pairs can be converted into corresponding text tags: <The question is not resolved>, <The question is partially resolved> and <The question is completely resolved> to meet the input needs of the generative model.

[0049] A generative model may be trained using the discrimination causes corresponding to the discrimination result tags as tags to guide the initial model to detect deep features such as logical relationships in the sample text pairs. Exemplarily, the discrimination cause tags may be manually marked based on the discrimination result tags.

[0050] For example, for the sample text pair Q2-A2, the discrimination result tag may be “partially resolves the question” and the discrimination cause tag may be “A2 resolves only the first part of Q2’s question, and the second part of the question is not resolved.”

[0051] According to an embodiment of the present disclosure, the initial model can be used to process the sample text pairs and the sample tags to obtain a second sample detection result. The second sample detection result may include a discrimination result for representing the answer matching degree between the sample answer text and the sample question text, and a discrimination cause corresponding to the discrimination result. For example, [partially solves the question; Cause: A2 only solves the first part of the question in Q2].

[0052] According to the embodiment of the present disclosure, a first loss value can be obtained based on the second sample detection result and the sample tag based on a first loss function. The first loss function may be a cross-entropy loss function, for example:

[0053]

number

[0054] According to the embodiment of the present disclosure, the model parameters of the initial model can be adjusted based on the first loss value until a convergence condition is satisfied, and a trained target model can be obtained. The convergence condition can be a maximum number of training times, a loss value reaching a predetermined threshold, or a loss value converging, and the embodiment of the present disclosure does not specifically limit the convergence condition.

[0055] According to an embodiment of the present disclosure, the discrimination results and discrimination causes are used as tags to guide an initial model, and deep features such as logical relationships between question texts and answer texts are used for detection, thereby improving the model accuracy, realizing accurate detection of sample text pairs that are difficult for the first model to detect, and obtaining quality detection results of question-answer text pairs with high reliability.

[0056] According to the embodiment of the present disclosure, the first model is fine-tuned based on sample text pairs in a specific application scene, thereby improving the discrimination accuracy of the first model.

[0057] For example, the first model may be an Enhanced Representation from knowledge Integration (ERNIE) model, which is built based on a Transformer encoder, and in the pre-training stage, the pre-training tasks performed may include a Masked Language Modeling (MLM) task and a Next Sentence Prediction (NSP) task.

[0058] In an embodiment of the present disclosure, a branch for outputting a confidence level can be added to the output layer of the ERNIE model to determine the discrimination difficulty of the ERNIE model for the input sample text pair.

[0059] Then, the preliminary training model is used to detect sample text pairs, a sample detection result and a confidence level of the sample detection result are generated, and a second loss value is obtained based on the sample detection result and the sample discrimination result tag according to a second loss function, and a model parameter of the preliminary training model is adjusted based on the second loss value to obtain a first model.

[0060] According to an embodiment of the present disclosure, the construction format of the sample text pair may be <question text><delimiter><answer text>. The sample discrimination result tag may be manually marked.

[0061] According to an embodiment of the present disclosure, the second loss function may be a cross-entropy loss function, for example:

[0062]

number

[0063] In an embodiment of the present disclosure, the categories of sample discrimination results may include three, and the sample discrimination result tags are: I-question not resolved; II-question partially resolved; and III-question completely resolved, respectively.

[0064] The category of the sample discrimination result can be set according to the needs of the actual application scene, and the present disclosure does not specifically limit it.

[0065] According to an embodiment of the present disclosure, the pre-training model may at least include a feature extraction layer and a feature processing layer. When adjusting the model parameters of the pre-training model based on the second loss value, the model parameters of the feature extraction layer may be fixed first, and the model parameters of the feature processing layer may be fine-tuned to improve the model training efficiency. If the convergence speed of the second loss value is slow, the model parameters of the feature extraction layer and the feature processing layer may be adjusted in parallel to improve the convergence speed of the second loss value and shorten the model training period.

[0066] According to an embodiment of the present disclosure, by increasing the confidence output branch, the discrimination difficulty of the first model for the sample text pairs can be determined based on the confidence, and the sample text pairs with high discrimination difficulty can be screened to train the generative model.

[0067] Hereinafter, the method shown in FIG. 2 will be further described in combination with a specific embodiment with reference to FIG. 3A and FIG. 3B.

[0068] FIG. 3A illustrates a schematic diagram of model training according to an embodiment of the present disclosure.

[0069] As shown in FIG. 3A, the embodiment 300A may include an ERNIE model 310 and an LlaMA model 320.

[0070] First, we consider a Q&A (Query&Answer) text pair [Q1-A1, , Q i -A i , ,Q I -A I ] 311 is input to the ERNIE model 310, which outputs a detection result 312 and a confidence 313. The detection result 312 and the confidence 313 correspond to the Q&A text pair, for example, the detection result for the Q1-A1 text pair is I (the question is not resolved) and the confidence is 67%. The detection result for the Q1-A1 text pair is II (the question is partially resolved) and the confidence is 97%.

[0071] Then, based on a predetermined threshold of the reliability, for example, 90%, text pairs with reliability lower than the predetermined threshold are screened and used as sample data for training the LlaMA model 320. j -A j , …, Q j -A j] 314 is obtained. The low confidence of the detection results corresponding to the screened Q&A text pairs indicates that it is difficult for the ERNIE model 310 to distinguish these text pairs, and accordingly, the accuracy of the detection results for these text pairs by the ERNIE model 310 is also low.

[0072] Next, we apply the manually marked tags,321,to the Q&A text pairs [Q1-A1, …, Q j -A j , …, Q j -A j ] 314 are input to an LlaMA model 320 for training, and a discrimination result + discrimination cause 323 is generated.

[0073] According to an embodiment of the present disclosure, processing the sample text pairs and the sample tags using the initial model to obtain a second sample detection result may include: stitching the sample text pairs and the sample tags to generate a sample target text; word-segmenting the sample target text to generate a sample word sequence; encoding each word in the sample word sequence according to its sequence position to generate a sample feature sequence; and processing the sample feature sequence based on an attention mechanism to generate a second sample detection result.

[0074] According to an embodiment of the present disclosure, the sample target text may be a text paragraph consisting of a sample text pair, a sample discrimination result tag, and a sample discrimination cause tag. The sample target text may be word-segmented, and the sample target text may be word-segmented into words or phrases to form a sample word sequence. Then, the sample word sequence may be converted into an integer index sequence based on the index value in the corpus of words or phrases. The integer index sequence may then be mapped into a real vector, and encoded based on the position of each word or phrase in the token sequence to generate a sample feature sequence.

[0075] According to an embodiment of the present disclosure, the LlaMA model 320 is constructed based on the Transformer decoder and uses an autoregressive method to generate each token (text unit) in the output sequence one by one. The text unit may be a word, a punctuation mark, a number, or other language element. In the decoding process, each time a token is generated, the content generated before it is used as a context to help predict the next token. The generated token sequence is usually linearly transformed by one output layer and the probability distribution at each position is converted into a probability corresponding to the token by a Softmax function. According to the probability, the token with the highest probability or the prediction result as the model is selected. In the generation task, the above autoregressive generation process is repeated for multiple tokens until an end mark (e.g., a period or an end symbol) is reached or a predetermined maximum output length is reached, and a second detection result including a discrimination result and a discrimination cause is output.

[0076] According to an embodiment of the present disclosure, a tag text can be generated by stitching the discrimination result tag and the discrimination cause tag, and a first loss value can be obtained based on the tag text and the second sample detection result based on a first loss function.

[0077] For example, for Q1-A1, the discrimination result tag may be "question is not solved" and the discrimination cause may be "answer is unrelated to the question". "Question is not solved" and "answer is unrelated to the question" are stitched together to generate tag text "question is not solved, and the cause is that the answer is unrelated to the question". Since the second detection result output by the LlaMA model 320 is a text including the discrimination result and the discrimination cause, there is no need to perform two stitching operations, and for example, the second detection result may be "question is not solved, and the cause is that the answer is unrelated to the question". Then, based on the above formula (1), a loss value 324 is calculated from the tag text and discrimination result + discrimination cause 323.

[0078] According to an embodiment of the present disclosure, the manually marked tag 321 includes a discrimination result tag 321_1 and a discrimination cause tag 321_2, so that during the training process of the LlaMA model 320, the LlaMA model 320 learns not only the discrimination result but also the discrimination cause corresponding to the discrimination result based on context learning for the input data, so that the LlaMA model 320 has inference and discrimination capabilities for deep logical features.

[0079] In the scene of quality inspection of question answering text, the technical field of the related question answering is very broad, and the manually marked discrimination causes are limited to the knowledge accumulation and subjective cognition of the person involved, so the accuracy of the marked discrimination causes varies. Therefore, the LlaMA model 320 trained using the manually marked discrimination cause tags has a certain accuracy bottleneck.

[0080] The generative large-scale language model has a strong logical reasoning ability that is pre-trained based on a rich corpus. Therefore, the embodiment of the present disclosure can break through the bottleneck of the accuracy of the LlaMA model 320 by using the discrimination cause output by the generative large-scale language model as the discrimination cause tag to train the LlaMA model 320.

[0081] According to an embodiment of the present disclosure, a suggested text can be constructed based on a sample text pair, a first discrimination result tag corresponding to the sample text pair, and a reference text, where the reference text includes a text pair related to a target sample text, a second discrimination result tag corresponding to the text pair, and a discrimination cause tag corresponding to the text pair, where the first discrimination result tag and the second discrimination result tag are identical, and a large-scale language model is used to process the suggested text to generate a second discrimination cause corresponding to the sample text pair, and determine the second discrimination cause as the discrimination cause tag.

[0082] FIG. 3B illustrates a schematic diagram of model training according to another embodiment of the present disclosure.

[0083] As shown in Fig. 3B, the embodiment 300B may include an ERNIE model 310, an LlaMA model 320 and a large-scale language model 330. The process of using the ERNIE model 310 to process a Q&A text pair 311 to obtain a detection result 312, and screening a Q&A text pair 314 for training the LlaMA model 320 from the Q&A text pair 311 based on a confidence 313 is the same as that of the embodiment 300A, and thus is not described here.

[0084] Then, a presentation text 331 is constructed based on the Q&A text pair 314 obtained by screening, the manually marked classification results, and reference texts. The reference texts are other text pairs Q&A text pairs in a field close to the Q&A text pair 314. t -A t and text pair Q t -A t may include manually marked discrimination results + discrimination causes corresponding to the above.

[0085] For example, the text pair Q j -A j may be a question and answer related to the field of autonomous driving technology, and may randomly screen 1-3 text pairs from other question and answer text pairs in the field of autonomous driving technology, and mark the discrimination results and discrimination reasons for the 1-3 text pairs. j -A j and text pair Q j -A j The manually marked judgment results corresponding to the above 1-3 text pairs, and the judgment results and judgment causes corresponding to the above 1-3 text pairs are called Prompt (presentation text).

[0086] The classification result tag type for the 1-3 text pairs is text pair Q. j -A j The result of the classification is the same as the tag type. For example, for a text pair Q j -A jis “I-Question unresolved”, the discrimination result tag type of the 1-3 text pairs is also “I-Question unresolved”, so that the generative large model can output a cause tag matching the discrimination result based on the discrimination cause of the 1-3 text pairs.

[0087] Next, the presented text 331 is input to the large-scale language model 330, and the text pair Q j -A j Then, the discrimination cause corresponding to the determined result tag is output, and the discrimination cause is identified as the discrimination cause tag 321_3. Then, the discrimination cause tag 321_3 is used to combine the manually marked discrimination result tag, and the LlaMA model 320 is trained. The training process is the same as that of the embodiment 300A described above, and therefore will not be described here.

[0088] According to an embodiment of the present disclosure, the discrimination results and discrimination causes of other text pairs in a field related to the text pair are input into the large model together with the reference text, and discrimination cause tags are output for training the LlaMA model 320. By making full use of the rich corpus and logical inference ability of the generative large model, and learning the logical inference ability of the generative large model by the LlaMA model 320 in a manner similar to result distillation, the model accuracy can be improved and the hardware resource needs of the model can be reduced, so that the LlaMA model 320 can efficiently complete the quality detection task of text pairs under conditions of limited hardware resources.

[0089] In the scene of question-answering text pair quality detection, not only the accuracy requirements of the detection result need to be met, but also the hardware resource requirements of the model operation need to be considered. Therefore, the embodiment of the present disclosure provides a text detection method that combines a discriminant model and a generative model.

[0090] FIG. 4 illustrates a schematic flow chart of a text detection method according to an embodiment of the present disclosure.

[0091] As shown in FIG. 4, the text detection method 400 includes operations S410 to S440.

[0092] In operation S410, a first model is used to detect a plurality of text pairs to be detected, and a first detection result and a confidence measure corresponding to the first detection result are obtained.

[0093] In operation S420, a target text pair to be detected is determined from the plurality of text pairs to be detected based on the confidence level.

[0094] In operation S430, the second model is used to detect the text pairs to be targeted, and a second detection result is obtained.

[0095] In operation S440, a target detection result is generated based on the first detection result and the second detection result.

[0096] According to an embodiment of the present disclosure, the first model may be a discriminant model, such as the ERNIE model, and the second model may be a target model, such as the LlaMA model, trained using the model training method described above.

[0097] According to an embodiment of the present disclosure, the plurality of text pairs to be detected include a plurality of question texts and a plurality of answer texts that correspond to each other.

[0098] According to the embodiment of the present disclosure, the first model can be used to detect all text pairs to be detected, and a first detection result and a corresponding confidence level for each text pair can be obtained. For example, the first detection result for the text pair Q1-A1 is II (partially solves the question) and has a confidence level of 95%. The first detection result for the text pair Q2-A2 is II (partially solves the question) and has a confidence level of 55%.

[0099] According to an embodiment of the present disclosure, the predetermined threshold of the confidence level may be set to 80%. Since the confidence level of the first detection result of the text pair Q1-A1 is greater than 80%, the first detection result of the text pair Q1-A1 can be taken as the final detection result. However, the confidence level of the first detection result of the text pair Q2-A2 is less than 80%, so the text pair Q2-A2 can be determined as the target text pair and used as the input data of the second model. This can reduce the amount of data processing of the second model and improve the detection efficiency.

[0100] According to an embodiment of the present disclosure, the second detection result obtained by detecting the target text pair using the second model includes a discrimination result category of the text pair and a discrimination cause corresponding to the discrimination result category.

[0101] For example, the second model may be used to process the text pair Q2-A2, and the second detection result obtained may be that the reason (the question is unsolved) is that answer A2 is unrelated to question Q2.

[0102] Finally, the first detection result output by the first model, the confidence of which is greater than a predetermined threshold, and the second detection result output by the second model are integrated into a target detection result, which represents the answer matching degree between the answer text and the corresponding question text.

[0103] According to the embodiment of the present disclosure, by comprehensively using the discriminant model and the generative model, compared with the single discriminant model in the related examples, the problem that the discriminant model has low detection accuracy due to overfitting of data can be solved, and the robustness of the model can be improved. In addition, when hardware resources are limited, the generative model can be used to detect only low-confidence text pairs, thereby reducing the amount of data processing of the generative model and improving the detection efficiency.

[0104] FIG. 5 illustrates a schematic diagram of text detection according to an embodiment of the present disclosure.

[0105] As shown in FIG. 5, an embodiment 500 may include an ERNIE model 510 and an LlaMA model 520 .

[0106] First, the Q&A text pair 501 “Q1-A1, Q2-A2, Q3-A3” is input to the ERNIE model 510, and the detection result 502 “Q1-A1_partially resolves the question_80%; Q2-A2_completely resolves the question_60%; Q3-A3_question unresolved_50%” is output. Based on the confidence, the Q&A text pair 501 is screened to obtain the Q&A text pair 503 “Q2-A2, Q3-A3”.

[0107] Then, the Q&A text pair 503 “Q2-A2, Q3-A3” is input to the LlaMA model 520, and the detection result 504 “Q2-A2_Completely resolves the question_Discriminating cause XXX; Q3-A3_Partially resolves the question_Discriminating cause YYY” is output.

[0108] For example, the text "Q2-A2" to be detected as a target may be divided into words to generate a word sequence. Then, the sample word sequence may be converted into an integer index sequence based on the index value in the corpus of words or phrases. The integer index sequence may then be mapped to a real vector, and encoding may be performed based on the position of each word or phrase in the token sequence to generate a feature sequence. Next, an autoregressive method is used to generate each token in the output sequence one by one (text unit) based on the attention mechanism. The text unit may be a word, punctuation mark, number, or other language element. In the decoding process, each time a token is generated, the previously generated content is used as a context to help predict the next token. The generated token sequence is converted into a probability corresponding to the token by one output layer, usually by a linear transformation and a Softmax function, from the probability distribution of each position. Depending on the probability, the token with the highest probability or the prediction result as the model is selected. In the generation task, the above autoregressive generation process is repeated until an end mark (e.g., a period or an end code) or a predetermined maximum output length is reached, multiple tokens are generated, and the output second detection result may be “Q2-A2_Perfectly resolves the question_Discrimination cause XXX”.

[0109] Finally, the detection result 502 whose confidence level is greater than a predetermined threshold is integrated with the detection result 504 to obtain the detection result 505 "Q1-A1_partially resolves the question_80%; Q2-A2_perfectly resolves the question_discriminating cause XXX; Q3-A3_partially resolves the question_discriminating cause YYY".

[0110] FIG. 6 illustrates a block diagram of a model training apparatus according to an embodiment of the present disclosure.

[0111] As shown in FIG. 6 , the model training apparatus 600 may include a first processing module 610, a first loss calculation module 620 and a first adjustment module 630.

[0112] In response to the confidence level of the first sample detection result being less than a predetermined threshold, the first processing module 610 processes the sample text pairs and the sample tags using the initial model to obtain a second sample detection result, where the confidence level of the first sample detection result is obtained by detecting the sample text pairs using the first model, the sample text pairs include corresponding sample question texts and sample answer texts, the second sample detection result includes a discrimination result for representing an answer matching degree between the sample answer texts and the sample question texts and a discrimination cause corresponding to the discrimination result, the sample tags include a discrimination result tag and a discrimination cause tag, and the initial model is a type different from that of the first model.

[0113] The first loss calculation module 620 obtains a first loss value based on the second sample detection result and the sample tag based on the first loss function.

[0114] The first adjustment module 630 adjusts model parameters of the initial model based on the first loss value to obtain a trained target model.

[0115] According to an embodiment of the present disclosure, the first loss calculation module may include a first stitching sub-module and a calculation sub-module. The first stitching sub-module stitches the discrimination result tag and the discrimination cause tag to generate a tag text. The calculation sub-module obtains a first loss value based on the tag text and the second sample detection result according to a first loss function.

[0116] According to an embodiment of the present disclosure, the first processing module may include a first stitching sub-module, a first word segmentation sub-module, a first encoding sub-module and a first processing sub-module.

[0117] The first stitching sub-module stitches the sample text pairs and the sample tags to generate a sample target text. The first word segmentation sub-module segments the sample target text into words to generate a sample word sequence. The first encoding sub-module encodes each word in the sample word sequence according to its sequence position to generate a sample feature sequence. The first processing sub-module processes the sample feature sequence based on the attention mechanism to generate a second sample detection result.

[0118] According to an embodiment of the present disclosure, the training apparatus further includes a construction module, a second processing module, and a first determination module.

[0119] The construction module constructs a presentation text based on the sample text pair, a first discrimination result tag corresponding to the sample text pair, and a reference text, where the reference text includes a text pair related to the target sample text, a second discrimination result tag corresponding to the text pair, and a discrimination cause tag corresponding to the text pair, and the first discrimination result tag and the second discrimination result tag are the same.

[0120] A second processing module processes the presented text using the large-scale language model to generate second discriminant causes corresponding to the sample text pairs.

[0121] The first determination module determines the second discrimination cause as a discrimination cause tag.

[0122] According to an embodiment of the present disclosure, the training apparatus further includes a first detection module, a second loss calculation module, and a second adjustment module. The first detection module detects sample text pairs using a pre-trained model, and generates sample detection results and confidence levels of the sample detection results. The second loss calculation module obtains a second loss value based on the sample detection results and the sample discrimination result tags according to a second loss function. The second adjustment module adjusts model parameters of the pre-trained model based on the second loss value to obtain a first model.

[0123] FIG. 7 illustrates a block diagram of a text detection device according to an embodiment of the present disclosure.

[0124] As shown in FIG. 7, the text detection apparatus 700 may include a second detection module 710, a second determination module 720, a third detection module 730 and a generation module 740.

[0125] The second detection module 710 detects a plurality of text pairs to be detected using a first model, and obtains a first detection result and a confidence corresponding to the first detection result, where the first model is a discriminant model, and the plurality of text pairs to be detected include a plurality of question texts and a plurality of answer texts corresponding to each other.

[0126] The second determination module 720 determines a target text pair to be detected from the multiple text pairs to be detected based on the confidence level, and the confidence level of the detection result corresponding to the target text pair to be detected is smaller than a predetermined threshold.

[0127] The third detection module 730 detects the text pairs to be targeted using the second model, and obtains a second detection result, where the second model is the trained target model obtained by the aforementioned training method.

[0128] The generating module 740 generates a target detection result based on the first detection result and the second detection result, where the target detection result represents an answer matching degree between the answer text and the corresponding question text.

[0129] According to an embodiment of the present disclosure, the third detection module may include a second word segmentation sub-module, a second encoding sub-module, and a second processing sub-module.

[0130] The second word segmentation sub-module segments the target text to be detected into words and generates a word sequence. The second encoding sub-module encodes each word in the word sequence according to its sequence position and generates a feature sequence. The second processing sub-module processes the feature sequence based on the attention mechanism and generates a second detection result.

[0131] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program.

[0132] According to an embodiment of the present disclosure, an electronic device includes at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor, and the at least one processor being capable of performing the above-mentioned method.

[0133] According to an embodiment of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to perform the above-described method.

[0134] According to an embodiment of the present disclosure, there is provided a computer program, which, when executed by a processor, implements the above method.

[0135] 8 shows an exemplary block diagram for implementing an exemplary electronic device 800 according to an embodiment of the present disclosure. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure as described and / or claimed herein.

[0136] 8, the device 800 includes a computing unit 801, which can perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 can further store various programs and data required for the operation of the device 800. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0137] The components of the device 800 are connected to an I / O interface 805, including an input unit 806, e.g., a keyboard, a mouse, etc., an output unit 807, e.g., various types of displays, speakers, etc., a storage unit 808, e.g., a magnetic disk, an optical disk, etc., and a communication unit 809, e.g., a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 enables the device 800 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various types of telecommunication networks.

[0138] The computing unit 801 may be various general-purpose and / or dedicated processing modules having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graph processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units for executing machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs each of the methods and processes described above, such as the model training method or the text detection method. For example, in some embodiments, the model training method or the text detection method may be realized as a computer software program tangibly included in a machine-readable medium, such as the storage unit 808. In some embodiments, some or all of the computer program may be loaded and / or installed in the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, it may perform one or more steps of the model training method or the text detection method described above. Alternatively, in another embodiment, the computing unit 801 may be configured to perform the model training method or the text detection method in any other suitable form (eg, via firmware).

[0139] Various embodiments of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may include a processor that is capable of receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

[0140] The program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program codes may be fully executed on the device, partially executed on the device, partially executed on the device as a separate software package and partially executed on a remote device or fully executed on a remote device or server.

[0141] In the context of this disclosure, a machine-readable medium may be a tangible medium, and may contain or store a program for use in or in combination with an instruction execution system, device, or electronic device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or electronic device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections through one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, a computer may implement the systems and techniques described herein and include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the computer. Other types of devices may also provide interaction with a user, for example, the feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and may receive input from the user in any form (including voice input, speech input, or tactile input).

[0143] The systems and techniques described herein may be implemented in a computing system including background components (e.g., a data server), or a computing system including middleware components (e.g., an application server), or a computing system including front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user may interact with embodiments of the systems and techniques described herein), or a computing system including any combination of such background, middleware, or front-end components. The components of the system may be connected to each other by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks illustratively include a local area network (LAN), a wide area network (WAN), and the Internet.

[0144] The computer system may include clients and servers. The clients and servers are generally remote and typically interact through a communication network. The relationship between clients and servers is created by computer programs running on the corresponding computers and having a client-server relationship. The server may be a cloud server, a server in a distributed system, or a server coupled with a blockchain.

[0145] It should be understood that various types of flows shown above may be used, and steps may be rearranged, added, or deleted. For example, each step described in the present invention may be performed in parallel, sequentially, or in a different order, and the present specification is not limited thereto as long as the desired results of the technical solution of the present disclosure can be achieved.

[0146] The specific embodiments described above do not limit the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. In response to the confidence of the first sample detection result being less than a predetermined threshold, processing the sample text pairs and the sample tags using the initial model to obtain a second sample detection result; Obtaining a first loss value based on the second sample detection result and the sample tag based on a first loss function; and adjusting model parameters of the initial model based on the first loss value to obtain a trained target model. Here, the reliability of the first sample detection result is obtained by detecting a sample text pair using a first model, the sample text pair includes a sample question text and a sample answer text corresponding to each other, the second sample detection result includes a discrimination result for indicating an answer matching degree between the sample answer text and the sample question text and a discrimination cause corresponding to the discrimination result, the sample tag includes a discrimination result tag and a discrimination cause tag, and the initial model is a type different from that of the first model. Model training methods.

2. Obtaining a first loss value based on the second sample detection result and the sample tag based on a first loss function, stitching the discrimination result tag and the discrimination cause tag to generate a tag text; obtaining the first loss value based on the tag text and the second sample detection result based on the first loss function. The method of claim 1.

3. Processing the sample text pairs and the sample tags with the initial model to obtain second sample detection results, stitching the sample text pairs and the sample tags to generate a sample target text; word segmenting the sample target text to generate a sample word sequence; encoding each word in the sample word sequence according to its sequence position to generate a sample feature sequence; and processing the sample feature sequence based on an attention mechanism to generate the second sample detection result. The method of claim 1.

4. constructing a presentation text based on the sample text pairs, the first determination result tags corresponding to the sample text pairs, and a reference text; processing the submitted text using a large-scale language model to generate second discriminant causes corresponding to the sample text pairs; determining the second discrimination cause as the discrimination cause tag; Here, the reference text includes a text pair related to the target sample text, a second discrimination result tag corresponding to the text pair, and a discrimination cause tag corresponding to the text pair, and the first discrimination result tag is the same as the second discrimination result tag. The method according to any one of claims 1 to 3.

5. detecting the sample text pairs using a pre-trained model and generating sample detection results and confidence measures for the sample detection results; Obtaining a second loss value based on the sample detection result and the sample discrimination result tag according to a second loss function; and adjusting model parameters of the pre-trained model based on the second loss value to obtain the first model. The method according to any one of claims 1 to 3.

6. Detecting a plurality of text pairs to be detected using a first model, obtaining a first detection result and a confidence level corresponding to the first detection result, the first model being a discriminant model, and the plurality of text pairs to be detected including a plurality of question texts and a plurality of answer texts corresponding to each other; determining a target text pair to be detected from the plurality of text pairs to be detected based on the confidence level, where the confidence level of the detection result corresponding to the target text pair to be detected is less than a predetermined threshold; Detecting the target text pair to be detected using a second model to obtain a second detection result, where the second model is a trained target model obtained by the training method according to any one of claims 1 to 3; generating a target detection result representing an answer matching degree between the answer text and the corresponding question text based on the first detection result and the second detection result. Text detection methods.

7. Detecting the target text pair using a second model to obtain a second detection result, Segmenting the text to be detected into words and generating a word sequence; encoding each word in the word sequence according to its position in the sequence to generate a sequence of features; and processing the sequence of features based on an attention mechanism to generate the second detection result. The method according to claim 6.

8. a first processing module that, in response to a confidence level of the first sample detection result being less than a predetermined threshold, processes the sample text pairs and the sample tags using the initial model to obtain a second sample detection result; a first loss calculation module for obtaining a first loss value based on the second sample detection result and the sample tag according to a first loss function; a first adjustment module for adjusting model parameters of the initial model based on the first loss value to obtain a trained target model; Here, the reliability of the first sample detection result is obtained by detecting a sample text pair using a first model, the sample text pair includes a sample question text and a sample answer text corresponding to each other, the second sample detection result includes a discrimination result for indicating an answer matching degree between the sample answer text and the sample question text and a discrimination cause corresponding to the discrimination result, the sample tag includes a discrimination result tag and a discrimination cause tag, and the initial model is a type different from that of the first model. Model training equipment.

9. The first loss calculation module: a first stitching sub-module for stitching the discrimination result tag and the discrimination cause tag to generate a tag text; and a calculation submodule for obtaining the first loss value based on the tag text and the second sample detection result based on the first loss function.

9. The apparatus of claim 8.

10. The first processing module is a first stitching sub-module for stitching the sample text pairs and the sample tags to generate a sample target text; a first word segmentation submodule for word segmenting the sample target text to generate a sample word sequence; a first encoding sub-module for encoding each word in the sample word sequence according to its sequence position to generate a sample feature sequence; a first processing submodule for processing the sample feature sequence based on an attention mechanism to generate the second sample detection result.

9. The apparatus of claim 8.

11. a construction module for constructing a presentation text based on the sample text pairs, the first determination result tags corresponding to the sample text pairs, and a reference text; a second processing module for processing the submitted text using a large-scale language model to generate second discriminant causes corresponding to the sample text pairs; A first determination module for determining the second discrimination cause as a discrimination cause tag, wherein the reference text includes a text pair related to the target sample text, a second discrimination result tag corresponding to the text pair, and a discrimination cause tag corresponding to the text pair, and the first discrimination result tag and the second discrimination result tag are the same. An apparatus according to any one of claims 8 to 10.

12. a first detection module for detecting the sample text pairs using a pre-trained model and generating a sample detection result and a confidence level for the sample detection result; A second loss calculation module obtains a second loss value based on the sample detection result and the sample discrimination result tag according to a second loss function; and a second adjustment module for adjusting model parameters of the preliminary training model based on the second loss value to obtain the first model. An apparatus according to any one of claims 8 to 10.

13. a second detection module for detecting a plurality of text pairs to be detected using a first model, obtaining a first detection result and a confidence level corresponding to the first detection result, the first model being a discriminant model, and the plurality of text pairs to be detected including a plurality of question texts and a plurality of answer texts corresponding to each other; a second determination module for determining a target text pair from the plurality of text pairs to be detected based on the confidence level, the confidence level of the detection result corresponding to the target text pair to be detected being less than a predetermined threshold; a third detection module for detecting the target text pair using a second model to obtain a second detection result, the second model being a trained target model obtained by the training method according to any one of claims 1 to 3; a generating module for generating a target detection result representing an answer matching degree between the answer text and the corresponding question text based on the first detection result and the second detection result. Text detector.

14. The third detection module is a second word segmentation submodule for segmenting the text to be detected into words and generating a word sequence; a second encoding sub-module for encoding each word in the word sequence according to its position in the sequence to generate a feature sequence; and a second processing sub-module for processing the feature sequence based on an attention mechanism to generate the second detection result.

14. The apparatus of claim 13.

15. At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform the method of any one of claims 1-3. electronic equipment.

16. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions cause the computer to carry out the method according to any one of claims 1-3.

17. A computer program which, when executed by a processor, implements the method according to any one of claims 1-3.

Images