Credit dialogue data construction method

Abstract

The invention provides a credit review dialogue data construction method which specifically comprises the following steps: (1) collecting credit review dialogue texts, scoring the credit review dialogue texts by adopting a large model, and taking the credit review dialogue texts with high scores as sample data; and (2) taking the large model as a teacher model, taking the small model as a student model, distilling the teacher model based on the sample data, distilling the scoring ability of the teacher model to the student model, and using the trained small model for scoring the quality of the credit dialogue text. Clear and score screening of credit review dialogue data is driven through a large model, credit review dialogue text data is remarkably improved, the follow-up training effect of the large language model can be effectively enhanced through high-quality review dialogue data, the professional requirement of the credit review dialogue field is met, credit review call-out precision, intelligence and digital transformation is assisted, and the credit review dialogue training efficiency is improved. In addition, the manual intervention amount is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of credit review, and provides a method for constructing credit review dialogue data. Background Art

[0002] With the rapid development of information technology, natural language processing technology has emerged. By endowing machines with the ability to understand and generate natural language, machines can communicate with humans naturally without barriers. As a crucial branch in the field of artificial intelligence, it covers multiple directions such as machine translation, language understanding, text summarization, and information extraction. Among them, the dialogue system plays a unique and key role, which needs to comprehensively apply multiple domain technologies in natural language processing and is one of the most widely used and complex fields.

[0003] Credit review dialogue is an effective scenario for the practical application of intelligent dialogue systems, demonstrating their ability to automate tasks. Such dialogue systems can replace traditional human customer service and complete complex credit review processes through interactive conversations with users, greatly saving manpower and improving work efficiency. Although current large language models have shown strong dialogue capabilities in general fields, in vertical scenarios such as financial credit review, due to the strong constraints of professional terms and the compliance requirements of dialogue logic, directly applying general models has problems of insufficient domain adaptation.

[0004] Fine-tuning training based on specific vertical scenario data of pre-trained general large language models can effectively solve this problem, but it poses strict requirements on the quality, scale, and diversity of training data. Existing data construction schemes mainly rely on manual annotation and rule template matching. By constructing a data processing rule library, outlier filtering and semantic annotation of data are realized, and manual annotation is relied on to further improve data quality. It improves the construction efficiency of structured data through templatized processing, but there are significant problems of incomplete rule coverage in complex semantic scenarios, and if manual calibration is used, the cost is extremely high and it is difficult to achieve data scale. Summary of the Invention

[0005] In view of this, the present application provides a method for constructing credit review dialogue data, aiming to improve at least one of the above problems.

[0006] Specifically, it includes the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a method for constructing credit review dialogue data, and the method is as follows:

[0008] (1) Collect credit review dialogue texts, score the credit review dialogue texts using a large model, and use the credit review dialogue texts with high scores as sample data;

[0009] (2) Use the large model as the teacher model and the small model as the student model. Distill the teacher model based on the sample data, distill the scoring ability of the teacher model into the student model, and use the trained small model to score the quality of the credit review dialogue text.

[0010] In some embodiments of the present invention, the method further includes:

[0011] (3) Enhance the quality of the credit review dialogue text with intermediate scores, score the enhanced credit review dialogue text through the large model, and use the credit review dialogue text with high scores as sample data.

[0012] In some embodiments of the present invention, the scoring process of the credit review dialogue text based on the large model is specifically as follows:

[0013] Input the credit review dialogue text into two parallel large model evaluators. The two large model evaluators respectively output evaluation reports of the credit review dialogue text. The two evaluation reports are aggregated by the large model evaluation aggregator to output the final evaluation report of the credit review dialogue text. The large model scorer generates the score of the credit review dialogue text based on the final evaluation report.

[0014] In some embodiments of the present invention, the scoring process of the credit review dialogue text based on the large model is specifically as follows:

[0015] Input the credit review dialogue text into the large model scorer. The large model scorer outputs tokens to the tokenizer. The tokenizer outputs the score of the corresponding credit review dialogue text and the distribution probabilities of each corresponding score. The fusion unit uses the generation probability of the score as the weight to perform weighted fusion on the scores, and uses the weighted fusion score as the score of the corresponding credit review dialogue text.

[0016] In some embodiments of the present invention, the quality enhancement method of the credit review dialogue text is specifically as follows:

[0017] Extract the questions of the credit reviewer and the answers of the loan application user in the first n rounds of the credit review dialogue text with intermediate scores, and input them into the credit review question model and the credit review answer model respectively. The credit review question model and the credit review answer model respectively conduct dialogues based on the input questions and answers;

[0018] After the nth round, the credit review question model automatically generates questions based on the answers given by the credit review answer model, and the credit review answer model automatically generates answers based on the questions given by the credit review question model. After the question and answer in the current round are completed, score the credit review dialogue text between the credit review question model and the credit review answer model. If the scored value is greater than the set scoring threshold, proceed to the next round until m rounds are completed, and use the credit review dialogue text between the credit review question model and the credit review answer model as a sample.

[0019] In some embodiments of the present invention, when the large model scorer adopts the LLaMA3 series model, two digits correspond to one token, and the formula for calculating the score s of the credit review dialogue text is specifically as follows:

[0020]

[0021] Wherein, i represents the score output by the tokenizer, and P(i) represents the generation probability of the score i output by the tokenizer.

[0022] In some embodiments of the present invention, when the large model scorer adopts the Qwen2.5 series model, two digits correspond to two tokens, and the formula for calculating the score s of the credit review dialogue text is specifically as follows:

[0023]

[0024] Wherein, i1 and i2 are respectively the values in the tens and units digits of the score i, P(i1) represents the probability of the tokenizer outputting the digit i1 in the tens place, and P(i2) represents the probability of the tokenizer outputting the digit i2 in the units place under the probability P(i1) of generating the digit i1 in the tens place.

[0025] In some embodiments of the present invention, select the score with the highest generation probability from the sample data, and the maximum generation probability is greater than the probability threshold. Use the sample with a higher score as the positive sample, and the sample with a lower score as the negative sample. Use the positive sample and the negative sample as the distillation samples and put them into the distillation dataset D. Distill the teacher model based on the distillation dataset.

[0026] In some embodiments of the present invention, the large model adopts the Qwen2.5-72B model, and the small model adopts the Qwen2.5-7B model.

[0027] The method for constructing credit review dialogue data provided by the present invention has the following beneficial technical effects:

[0028] (1) By driving the clarity and scoring screening of credit review dialogue data with a large model, the credit review dialogue text data is significantly improved. High-quality credit review dialogue data can effectively enhance the subsequent training effect of the large language model, meet the professional requirements of the credit review dialogue field, help the accuracy, intelligence, and digital transformation of credit review outbound calls. In addition, it greatly reduces the amount of manual intervention.

[0029] (2) By distilling the scoring function of the large model into the small model, while maintaining the existing performance of the model, the consumption of computing resources is effectively reduced, and then it can be better used for actual credit review operations. Description of the Drawings

[0030] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0031] Figure 1 It is a flowchart of the method for constructing credit review dialogue data provided by the embodiments of the present invention;

[0032] Through the above accompanying drawings, the clear embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These accompanying drawings and text descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope protected by the present application. Unless otherwise defined, all technical terms used in the embodiments of the present application have the same meaning as commonly understood by those of ordinary skill in the art.

[0034] Figure 1 It is a flowchart of the method for constructing credit review dialogue data provided by the embodiments of the present invention, and the method is as follows:

[0035] (1) Collect credit review dialogue texts, score the credit review dialogue texts using a large model, and use the credit review dialogue texts with high scores as sample data;

[0036] Collect the credit review dialogue voices between credit reviewers and loan application users, convert the collected credit review dialogue voices into credit review dialogue texts. During the process of converting credit review dialogue voices into credit review texts, due to problems such as noise pollution, colloquial dialect features of loan application users, fuzzy voice signals, and non-standard expressions, the quality of the converted credit review dialogue texts fluctuates. Therefore, it is necessary to score the quality of the credit review dialogue texts, use the credit review dialogue texts with high scores as samples to participate in large model distillation, eliminate the credit review dialogue texts with low scores, and enhance the quality of the credit review dialogue texts with intermediate segmented scores, which is mainly used to enhance the number of samples during the transfer training process.

[0037] In the embodiment of the present invention, first, the credit review dialogue text is scored based on a large model. Since the large model has strong reasoning ability, its scoring ability is also relatively strong. The process of scoring the credit review dialogue text based on the large model is as follows:

[0038] The credit review dialogue text is input into the large model scorer. The large model scorer outputs tokens to the tokenizer, and the tokenizer outputs the score of the corresponding credit review dialogue text and the probability distributions of each corresponding score. The fusion unit uses the generation probability of the score as the weight to perform weighted fusion on the scores, and takes the weighted fused score as the score of the corresponding credit review dialogue text.

[0039] When the large model scorer adopts the LLaMA3 series of models (including the LLaMA model, the LLaMA2 model, the LLaMA3 model, and the LLaMA3.1 model), two digits correspond to one token. For example, the token corresponding to the number "85" is "5313". According to the probability distribution of the next token generated by the large model scorer, through tokenization processing by the tokenizer, the 1 - 99 scoring interval is mapped to a set of probability distributions of corresponding tokens. After probability normalization processing, a weighted fusion algorithm is used to generate the final score s of the corresponding credit review dialogue text. Among them, i represents the score, and P(i) represents the generation probability of the score i.

[0040] When the large model scorer adopts the Qwen2.5 series of models, two digits correspond to two tokens. The tokens corresponding to the number "85" are "23" and "20", that is, the number "8" and the number "5" each correspond to one token. The two - stage scoring scheme designed in the present invention is as follows: In the first stage, as shown in the direct probability weighted scoring, first, the probability distribution in the range of 1 to 9 is obtained, and the low - probability tokens are removed and recorded. In the second stage, the high - frequency tokens recorded in the first stage are added at the end of the summary evaluation report to obtain the probability in the range of 0 to 9 at this time. By multiplying the probabilities of the two stages, the high - frequency probability distribution in the 10 to 99 scoring interval is finally obtained, and then weighted fusion is performed to obtain the final score s of the corresponding credit review dialogue text. Among them, i1 and i2 are the tens and units digits of the score i respectively. P(i1) represents the probability of generating the digit i1 in the tens place, and P(i2) represents the probability of generating the digit i2 in the units place under the probability P(i1) of generating the digit i1 in the tens place. Given that the Qwen2.5 series of models (including the Qwen model, the Qwen2 model, and the Qwen2.5 model) show significant technical advantages in the Chinese context and the performance indicators are significantly better than existing models.

[0041] In another embodiment of the present invention, the credit review dialogue text is scored based on a large model. The process of scoring the credit review dialogue text is as follows:

[0042] Input the credit review dialogue text into two parallel large model evaluators. The two large model evaluators respectively output evaluation reports of the credit review dialogue text. The two evaluation reports are aggregated by a large model evaluation aggregator to output the final evaluation report of the credit review dialogue text. The large model scorer generates a score for the credit review dialogue text based on the final evaluation report.

[0043] The above large model evaluator, large model evaluation aggregator, and large model scorer all adopt general large models, and can adopt LLaMA series models, Qwen series models, Mistral, InternLM series models, including: InternLM model, InternLM2 model, InternLM2.5 model, InternLM3 model.

[0044] (2) Use the large model as the teacher model and the small model as the student model. Based on the sample data, distill the teacher model to transfer the scoring ability of the teacher model to the student model, and use the trained small model to score the quality of the credit review dialogue text.

[0045] In the embodiment of the present invention, select the score with the highest generation probability and the highest generation probability greater than the probability threshold from the sample data. Use the sample with a higher score as the positive sample and the sample with a lower score as the negative sample. Use the positive sample and the negative sample as the distillation samples and put them into the distillation dataset D. Distill the teacher model based on the distillation samples to transfer the scoring ability of the teacher model to the student model.

[0046] The large model adopts the Qwen2.5 - 72B model, and the small model adopts the Qwen2.5 - 7B model. When the large model is used in actual applications, due to high-frequency calls, the inference of the credit review large model is restricted. Through model distillation technology, transfer the knowledge and ability contained in the Qwen2.5 - 72B large model to the small model Qwen2.5 - 7B, so as to effectively reduce the consumption of computing resources while maintaining the existing performance of the model.

[0047] (3) Enhance the quality of the credit review dialogue text with intermediate scores, score the enhanced credit review dialogue text through a large model, and use the credit review dialogue text with high scores as sample data;

[0048] In the embodiment of the present invention, the method for enhancing the quality of the credit review dialogue text is specifically as follows:

[0049] Extract the questions of the credit reviewer and the answers of the loan application user in the first n rounds of conversations from the credit review dialogue text with intermediate scores, and input them into the credit review question model and the credit review answer model respectively. The credit review question model and the credit review answer model conduct conversations based on the input questions and answers respectively.

[0050] After the nth round, the credit review question model automatically generates questions based on the answers given by the credit review answer model, and the credit review answer model automatically generates answers based on the questions given by the credit review question model. After the question and answer for the current round are completed, the credit review dialogue text between the credit review question model and the credit review answer model is scored. If the score is greater than the set scoring threshold, the next round is carried out until m rounds are completed. The credit review dialogue text between the credit review question model and the credit review answer model is used as a sample. If the scores for three consecutive rounds are less than the set scoring threshold, the quality enhancement of the credit review dialogue text is stopped and the credit review dialogue text is excluded.

[0051] In the embodiment of the present invention, the credit review question model and the credit review answer model adopt the Qwen series.

[0052] The method for constructing credit review dialogue data provided by the present invention has the following beneficial technical effects:

[0053] (1) By driving the clarity and scoring screening of credit review dialogue data through a large model, the credit review dialogue text data is significantly improved. High-quality credit review dialogue data can effectively enhance the subsequent training effect of the large language model, meet the professional needs in the field of credit review dialogue, contribute to the precision, intelligence, and digital transformation of credit review outbound calls, and in addition, greatly reduce the amount of manual intervention.

[0054] (2) By distilling the scoring function of the large model into a small model, while maintaining the existing performance of the model, the consumption of computing resources is effectively reduced, and thus it can be better used for actual credit review operations.

[0055] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practice of the present application disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only considered exemplary.

[0056] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A method for constructing credit review dialogue data, characterized in that: The method is specifically as follows: (1) Collecting credit review dialogue texts, using a large model to score the credit review dialogue texts, and using credit review dialogue texts with high scores as sample data; (2) The large model is used as the teacher model and the small model is used as the student model. The teacher model is distilled based on the sample data, and the scoring ability of the teacher model is distilled to the student model. The trained small model is used to score the quality of the credit review dialogue text.

2. The method for constructing credit review dialogue data according to claim 1, characterized in that: The method further comprises: (3) Enhance the quality of the credit review dialogue texts with intermediate scores, score the enhanced credit review dialogue texts through a large model, and use the credit review dialogue texts with high scores as sample data.

3. The method for constructing credit review dialogue data according to claim 1, characterized in that: The scoring process of the credit review dialogue text based on the big model is as follows: The credit review dialogue text is input into two parallel large model evaluators. The two large model evaluators respectively output evaluation reports of the credit review dialogue text. The two evaluation reports are aggregated through the large model evaluation aggregator to output the final evaluation report of the credit review dialogue text. The large model scorer generates a score for the credit review dialogue text based on the final evaluation report.

4. The method for constructing credit review dialogue data according to claim 1, characterized in that: The scoring process of the credit review dialogue text based on the big model is as follows: The letter review dialogue text is input into the large model scorer, which outputs word units to the word segmenter. The word segmenter outputs the score corresponding to the letter review dialogue text and the distribution probabilities of its corresponding scores. The fusion unit uses the generation probability of the score as the weight, performs weighted fusion on the scores, and uses the weighted fusion score as the score of the corresponding letter review dialogue text.

5. The method for constructing credit review dialogue data as claimed in claim 2, characterized in that: The quality enhancement method of the letter review dialogue text is as follows: Extract the questions asked by the credit reviewer and the answers of the loan applicant in the first n rounds of dialogue from the credit review dialogue text with the middle score, and input them into the credit review question model and the credit review answer model respectively. The credit review question model and the credit review answer model respectively answer the questions and answers based on the input; After the nth round, the credit review question model automatically generates questions based on the answers given by the credit review answer model, and the credit review answer model automatically generates answers based on the questions given by the credit review question model. After the current round of questions and answers is completed, the credit review dialogue text between the credit review question model and the credit review answer model is scored. If the score is greater than the set scoring threshold, the next round is carried out until the mth round is completed, and the credit review dialogue text between the credit review question model and the credit review answer model is used as a sample.

6. The method for constructing credit review dialogue data according to claim 4, characterized in that: When the large model scorer uses the LLaMA3 series model, two digits correspond to one word unit, and the score s of the credit review dialogue text is calculated as follows: Among them, i represents the score output by the word segmenter, and P(i) represents the generation probability of the score i output by the word segmenter.

7. The method for constructing credit review dialogue data according to claim 4, characterized in that: When the large model scorer uses the Qwen2.5 series model, two digits correspond to two words, and the score calculation formula for the credit review dialogue text is as follows: Among them, i1 and i2 are the values ​​of the tenth and units digits of the score i respectively, P(i1) represents the probability of the tenth digit output by the word segmenter generating the number i1, and P(i2) represents the probability of the unit digit output by the word segmenter generating the number i2 under the probability P(i1) of generating the number i1 in the tenth digit.

8. The method for constructing credit review dialogue data according to claim 1, characterized in that: Select the score with the largest generation probability from the sample data, and the score with the maximum generation probability greater than the probability threshold. Take the sample with the higher score as the positive sample, and take the sample with the lower score as the negative sample. Put the positive and negative samples as distilled samples into the distillation dataset D, and distill the teacher model based on the distillation dataset.

9. The method for constructing credit review dialogue data according to claim 1, characterized in that: The large model uses the Qwen2.5-72B model, and the small model uses the Qwen2.5-7B model.

Images