Conversation corpus construction method, device and system in claim field and medium

Abstract

The invention discloses a method, device and system for constructing dialogue corpora in the claim settlement field based on a large language model and a medium, and the method comprises the steps: collecting original corpora in the claim settlement field, and carrying out the preprocessing of the original corpora; semantic clustering is carried out on the preprocessed corpus, and corpus samples are screened out; performing knowledge and intention labeling on the corpus sample to construct a training sample and an intention knowledge base; generating claim dialogue corpora by using a large language model in combination with a construction command guide type prompt; and then performing supervised fine-tuning training on the large language model based on the training sample and the claim dialogue corpus to obtain a fine-tuning large language model, thereby remarkably improving the application effect and accuracy of the LLM in the claim field.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a method, device, system and medium for constructing dialogue corpus in the claims field based on a large language model. Background Art

[0002] With the emergence of large parameter models such as ChatGPT, GPT4, Claude, and Wenxinyiyan, large language model technology is increasingly used in life and business. However, the large language models currently on the market are mainly used in general common sense dialogues and hot knowledge. Although they have strong general knowledge and intelligent decision-making and judgment capabilities, they still face many challenges in vertical fields such as claims. The main reasons include high training costs and lack of professional knowledge corpus in vertical fields.

[0003] The high training cost is mainly reflected in the following aspects: the training of large models requires huge data sets and computing resources, including data collection, data cleaning, feature extraction and other steps, which consume a lot of time and resources. In the field of natural language processing, a high-quality language model may take months or even years to complete training, which makes the vertical application cost of large models very high, limiting its application scope.

[0004] The lack of professional knowledge corpus in vertical fields is mainly reflected in the following aspects: it may be challenging to obtain sufficient quantity and quality of professional knowledge text in vertical fields, which may be due to the limited number of professional knowledge texts in the field or the difficulty in obtaining the license or permission of these texts. The lack of professional knowledge corpus in vertical fields may make it difficult for the model to understand the professional terminology, context and special language expressions in a specific field, which may affect the semantic understanding and correctness of the model in this field.

[0005] Therefore, how to efficiently generate high-quality claims domain dialogue corpus has become a key issue in the application of large language models in the claims field. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method, device, system and medium for constructing dialogue corpus in the claims field based on a large language model, which can be applied to financial technology or other related fields, aiming to improve the accuracy of the large language model in the claims field.

[0007] The technical solution of the present invention is as follows:

[0008] A method for constructing dialogue corpus in the claims field based on a large language model, comprising:

[0009] Collecting original corpus in the field of claims and preprocessing the original corpus;

[0010] Perform semantic clustering on the preprocessed corpus to screen out corpus samples;

[0011] Performing knowledge and intent annotation on the corpus samples to construct training samples and an intent knowledge base;

[0012] Use a large language model combined with command-guided prompts to generate claims dialogue corpus;

[0013] Based on the training samples and claims dialogue corpus, supervised fine-tuning training is performed on the large language model to obtain a fine-tuned large language model.

[0014] In one embodiment, the method for constructing dialogue corpus in the claims field based on a large language model further includes:

[0015] Fine-tune the large language model to predict the newly generated claims dialogue corpus, and preprocess the unrecalled corpus again as the original corpus.

[0016] In one embodiment, the step of collecting original corpus in the field of claims and preprocessing the original corpus includes:

[0017] Collect original corpus in the field of claims;

[0018] ASR translated text, long text and non-text data are screened and removed from the original corpus.

[0019] In one embodiment, the step of performing semantic clustering on the preprocessed corpus to screen out corpus samples includes:

[0020] Use the BERT model to extract text semantic feature vectors;

[0021] Cluster the text semantic feature vectors and remove text corpora with similarity greater than a preset threshold.

[0022] In one embodiment, the step of performing knowledge and intent annotation on the corpus sample to construct a training sample and an intent knowledge base includes:

[0023] Extracting and annotating the corpus text;

[0024] Construct model training samples and an intent knowledge base; the intent knowledge base includes attribute information of scene entities, category entities, and corpus samples.

[0025] In one embodiment, the step of using a large language model in combination with constructing a command-guided prompt to generate a claims dialogue corpus includes:

[0026] Construct a command-guided prompt to generate text training data;

[0027] Input prompt into a large language model to obtain the generated corpus sample;

[0028] The corpus samples are cleaned and screened to form new claims dialogue corpus.

[0029] In one embodiment, the step of performing supervised fine-tuning training on the large language model based on the training samples and the claims dialogue corpus to obtain the fine-tuned large language model includes:

[0030] Use the annotated corpus and claims dialogue corpus to build a training set;

[0031] Perform supervised fine-tuning training on the training set for the large language model to obtain a fine-tuned model;

[0032] The performance of the fine-tuned model is evaluated and iterative optimization is performed based on the evaluation results.

[0033] A device for constructing dialogue corpus in the claims domain based on a large language model, comprising:

[0034] A preprocessing module, used to collect original corpus in the field of claims and preprocess the original corpus;

[0035] The screening module is used to perform semantic clustering on the preprocessed corpus and screen out corpus samples;

[0036] Annotation module, used for performing knowledge and intent annotation on the corpus samples to construct training samples and intent knowledge base;

[0037] Corpus generation module, which is used to generate claims dialogue corpus using a large language model combined with command-guided prompts;

[0038] The supervised fine-tuning module is used to perform supervised fine-tuning training on the large language model based on training samples and claims dialogue corpus to obtain a fine-tuned large language model.

[0039] A system for constructing dialogue corpus in the claims domain based on a large language model, the system comprising at least one processor; and

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for constructing dialogue corpus in the claims field based on a large language model.

[0042] A non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the above-mentioned method for constructing dialogue corpus in the claims field based on a large language model.

[0043] Beneficial effects: The present invention discloses a method, device, system and medium for constructing dialogue corpus in the claims field based on a large language model. Compared with the prior art, the embodiments of the present invention generate high-quality dialogue corpus in the claims field through automated corpus collection, preprocessing, semantic clustering, knowledge and intent annotation, dialogue corpus generation, data cleaning and supervised fine-tuning, which significantly improves the application effect and accuracy of LLM in the claims field. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0045] Figure 1 A flow chart of a method for constructing dialogue corpus in the claims field based on a large language model provided by an embodiment of the present invention;

[0046] Figure 2 A flowchart of step S100 in the method for constructing dialogue corpus in the claims field based on a large language model provided in an embodiment of the present invention;

[0047] Figure 3 A flowchart of step S200 in the method for constructing dialogue corpus in the claims field based on a large language model provided in an embodiment of the present invention;

[0048] Figure 4 A flowchart of step S300 in the method for constructing dialogue corpus in the claims field based on a large language model provided in an embodiment of the present invention;

[0049] Figure 5 A flowchart of step S400 in the method for constructing dialogue corpus in the claims field based on a large language model provided in an embodiment of the present invention;

[0050] Figure 6 A flowchart of step S500 in the method for constructing dialogue corpus in the claims field based on a large language model provided in an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the functional modules of the device for automatically generating test data provided by an embodiment of the present invention;

[0052] Figure 8 A schematic diagram of the hardware structure of the system for automatically generating test data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0054] Based on the business knowledge dialogue data in professional fields, the present invention constructs a complete set of dialogue corpus in the claims field through manual annotation, LLMPrompt injection, data screening, etc. Based on this method, high-quality dialogue corpus in the financial business field can be generated, and the large language model can be fine-tuned in a supervised manner, tested, fed back and iteratively optimized, thereby improving the implementation and application practice of the large language model in vertical fields (i.e., the financial field).

[0055] The method for constructing dialogue corpus in the field of claims settlement based on a large language model of the present invention comprises: collecting original corpus of claims settlement process; preprocessing the original corpus, including removing meaningless text and stop words; using a BERT model (Bidirectional Encoder Representations from Transformers, a method for pre-training language representation) to perform semantic clustering on the preprocessed corpus and screen out high-quality corpus samples; annotating knowledge and intent on the screened corpus and constructing an intent knowledge base; generating training dialogue corpus using a large language model (LLM) and a constructed prompt; performing data cleaning and screening on the generated dialogue corpus to form a training data set; and performing supervised fine-tuning on the LLM to form a fine-tuned model.

[0056] In order to solve the above problems, the present invention proposes a method for constructing dialogue corpus in the claims field based on a large language model. Figure 1 , Figure 1 A flowchart of an embodiment of the method for constructing dialogue corpus in the claims field based on a large language model provided by the present invention. The method for constructing dialogue corpus in the claims field based on a large language model provided in this embodiment is applied to a system consisting of a terminal device, a network and a server, wherein the network is a medium for providing a communication link between the terminal device and the server, which may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.; the operating system on the terminal device may include a handheld device operating system (iPhone operating system, iOS system), Android system or other operating system, and the terminal device is connected to the server through the network to achieve interaction, thereby performing operations such as receiving or sending data, and specifically may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, portable computers and desktop servers, etc. Figure 1As shown, the method specifically comprises the following steps:

[0057] S100: Collect original corpus in the field of claims settlement, and pre-process the original corpus.

[0058] In this embodiment, it is necessary to first collect about 20,000 desensitized original claims process corpora recorded in the claims field. Through preliminary screening, ASR translated texts, long texts and other corpora are screened out. At the same time, multimodal data such as images and videos are cleaned up, and dirty data cleaning is performed, including: abnormal corpus processing, meaningless text corpus processing, stop word processing, etc., and 5,000 corpus samples are obtained.

[0059] In one embodiment, Figure 1 As shown, step S100 includes:

[0060] S101. Collect original corpus in the field of claims.

[0061] Optionally, the collected original corpus includes about 20,000 original claims process corpora including text, images, and videos. These corpora record various conversations, descriptions, and records in the actual claims process, covering a wealth of business scenarios and expertise.

[0062] In an optional embodiment, the data source of the original corpus can be exported from the insurance company's claims system to ensure the authenticity and representativeness of the data. When exporting the data, the original corpus is first desensitized to remove fields containing sensitive information (such as customer name, ID number, etc.) to protect personal privacy.

[0063] For example, the original corpus can collect data from multiple claims cases, including different types of insurance claims such as personal insurance, commercial insurance, auto insurance, and health insurance. It can also collect claims data from different regions and different insurance companies to ensure the diversity of regional and corporate strategies.

[0064] The data of the original corpus includes: collecting text data, such as claim applications, investigation reports, claim decisions, etc.; collecting image data, such as photos of the accident scene, photos of damaged items, etc.; collecting video data, such as surveillance videos of the accident scene, interview videos of claims personnel and customers, etc.

[0065] The present invention also collects data from each stage of the claims process, from claim application, preliminary review, on-site investigation, claim decision to claim payment, ensuring that the data at each stage has a sufficient sample size to fully understand the claims process.

[0066] In the data preprocessing stage, key features of claim cases are extracted, such as claim type, accident type, claim amount, processing time, etc., to ensure that these features are well distributed and representative in the dataset.

[0067] S102: Filter and remove ASR translated text, long text and non-text data from the original corpus.

[0068] In one embodiment, the collected original corpus is preliminarily screened and cleaned to remove corpus that is not suitable for training, which includes removing ASR translated text, long text, images, videos and other non-text data, as well as abnormal corpus processing and stop word processing, that is, removing corpus that is not suitable for training.

[0069] Among them, screening and removing corpus that is not suitable for training includes:

[0070] Remove ASR transcribed text: Identify and remove ASR (automatic speech recognition) transcribed text ending in audio formats such as MP3, as these texts may contain speech recognition errors, affecting the quality of the corpus.

[0071] Remove long text: Remove long text with a token length greater than 1024 to reduce the complexity of subsequent processing.

[0072] Clean up multimodal data: Delete non-text format data such as images and videos to ensure the pure text nature of the corpus.

[0073] Clean abnormal corpus: Remove harmful text, counterfactual text, and biased text through a combination of regular expressions and manual review.

[0074] Clean up meaningless text: Use the pre-trained BERT model to score the text and remove meaningless text with a score below a threshold.

[0075] Stop word processing: Collect and apply the industry's commonly used stop word list, perform word segmentation on each sentence, and delete stop words.

[0076] S200: semantic clustering is performed on the preprocessed corpus to screen out corpus samples.

[0077] In this embodiment, semantic clustering is performed on the selected corpus, and the semantic feature vector of each corpus is extracted using the BERT model, and then cluster analysis is performed on these vectors. The present invention can perform deduplication processing on text corpora with a similarity greater than 0.83 to ensure the diversity and uniqueness of the training data. Semantic clustering is performed on the corpus that has been initially screened and cleaned to remove text with highly overlapping semantics and reduce redundancy.

[0078] In one embodiment, Figure 3 As shown, step S200 includes:

[0079] S201. Use the BERT model to extract text semantic feature vectors.

[0080] When the BERT model extracts text semantic feature vectors, it specifically includes:

[0081] Prepare text data: Clean and preprocess the text data, such as removing special characters and punctuation marks.

[0082] Word segmentation: Use the word segmenter of the BERT model to convert the text into a series of tokens, which are the basic units that the model can understand.

[0083] Adding special tokens: The BERT model requires adding special tokens (such as [CLS] and [SEP]) at the beginning and end of the text. The output of the [CLS] token is usually used as the overall representation of the text.

[0084] Handling long texts: If the text length exceeds the maximum input length of the BERT model (usually 512 tokens), the text needs to be truncated or split.

[0085] Load BERT model: Load a pre-trained BERT model. You can use tools like Hugging Face's Transformers library to load the model.

[0086] Input text to the model: Input the processed text (including tokens, segment masks, and input IDs) into the BERT model.

[0087] Get output: The BERT model outputs the embedding vector of each token. This paper focuses on the output of [CLS] token.

[0088] Post-processing: Further process the output vector as needed, such as normalization and dimensionality reduction, and finally extract the text semantic feature vector that meets the requirements.

[0089] S202: Clustering text semantic feature vectors, and removing text corpora with similarity greater than a preset threshold.

[0090] Specifically, data cleaning and preprocessing include cleaning data, removing irrelevant or duplicate records, and ensuring data quality and consistency. Preprocessing data, such as text segmentation, part-of-speech tagging, entity recognition, etc., lays the foundation for subsequent semantic analysis and clustering.

[0091] In an optional embodiment, when semantic clustering is performed on samples, text clustering is performed using the BERT model to initially screen corpora with highly overlapping semantics and highly similar texts. Specifically, cluster analysis is performed on feature vectors, such as using K-means or DBSCAN algorithms to perform cluster analysis. Based on the clustering results, text corpora with a similarity greater than a preset threshold (such as 0.83) are removed to reduce data redundancy. The screening is performed in the following manner:

[0092] t mean =Bert(old_text)i∈{1,2,3,...,5000};

[0093]

[0094] i,j∈{1,2,3,...,5000},i≠j

[0095] t mean The semantic feature vector extracted by BERT represents the text old_text. The vector is clustered to obtain text corpora with different semantics. 1.2.3...., 5000 is 5000 corpus samples.

[0096] It represents the semantic similarity between text i and text j. After empirical verification, the text corpus with a similarity greater than 0.83 is deduplicated to obtain the text corpus to be annotated later.

[0097] In one embodiment, the step S200 may further include:

[0098] Feature extraction: Use BERT or other pre-trained language models to extract semantic feature vectors of text;

[0099] Data preprocessing: standardize the feature vector to eliminate the dimensionality effects of different dimensions, and also perform dimensionality reduction, such as using PCA (principal component analysis) or t-SNE (t-distributed stochastic neighbor embedding);

[0100] Select clustering algorithm: Select appropriate clustering algorithm according to data characteristics and requirements, such as K-means, hierarchical clustering, DBSCAN, etc.;

[0101] Determine the number of clusters (K value): If you use an algorithm such as K-means that requires specifying the number of clusters, you need to determine the appropriate K value. You can use methods such as the elbow rule and silhouette coefficient to assist in the decision;

[0102] Perform clustering: Run the clustering algorithm to cluster the feature vectors;

[0103] Evaluate clustering effect: Use indicators such as silhouette coefficient and Davies-Bouldin index to evaluate clustering effect, visualize clustering results, and check the rationality of clustering;

[0104] Screening samples: Select representative samples from each cluster. This can be samples near the cluster center, or central samples in each cluster;

[0105] Manual review (optional): Manual review of automatically screened samples to ensure their representativeness and accuracy;

[0106] Apply filtered samples: Use the filtered samples for further analysis or as training data.

[0107] When determining the number of clusters (K value), the following methods can be included:

[0108] Determining the optimal number of clusters (K value) in corpus clustering is a key step. The following are several commonly used methods:

[0109] A) Elbow Method:

[0110] This method determines the optimal K value by observing the trend of the sum of squared clustering errors (SSE) as the K value changes. As the K value increases, the SSE will gradually decrease, but when the K value approaches the optimal value, the decrease in SSE will decrease significantly, forming an "elbow". The optimal K value is usually at the elbow.

[0111] The specific steps include: calculating the SSE under different K values, drawing a relationship graph between K value and SSE, and selecting the K value corresponding to the elbow as the optimal number of clusters.

[0112] B) Silhouette Coefficient:

[0113] The silhouette coefficient measures the quality of the clustering results. It calculates the cohesion and separation of each sample point. The value range is between [-1, 1]. The closer it is to 1, the better the clustering effect.

[0114] Calculate the silhouette coefficient under different K values, and select the K value that maximizes the silhouette coefficient as the optimal number of clusters.

[0115] C) Calinski-Harabasz Criterion (CH value):

[0116] This method determines the optimal K value by comparing the intra-cluster dispersion and inter-cluster dispersion. The larger the CH value, the better the clustering effect.

[0117] Calculate the CH value under different K values, and select the K value with the largest CH value as the optimal number of clusters.

[0118] D)Davies-Bouldin Criterion(DB value):

[0119] Similar to the CH value, the smaller the DB value, the better the clustering effect. By calculating the DB value under different K values, the K value with the smallest DB value is selected as the optimal cluster number.

[0120] E) Cross-validation method:

[0121] Divide the data set into multiple parts, perform clustering on one part, and evaluate the quality of clustering on other parts. Use the silhouette coefficient or other clustering quality indicators to assist in determining the best K value.

[0122] The specific steps include: dividing the data set into K subsets, for each subset, performing clustering and calculating the clustering quality index on other subsets, and selecting the K value that gives the highest average clustering quality index.

[0123] F) Gap Statistic:

[0124] Gap Statistic determines the optimal K value by comparing the sum of the intra-cluster deviations of the original data with the sum of the intra-cluster deviations of the data under uniform distribution inference. The K value that maximizes the Gap Statistic is considered to be optimal.

[0125] The specific steps include: calculating the sum of intra-cluster deviations under different K values, generating random data and calculating the sum of its intra-cluster deviations, and selecting the K value that maximizes the Gap Statistic.

[0126] G) Canopy Algorithm:

[0127] The Canopy algorithm determines the number of initial cluster centers and cluster center points for the K-means algorithm by performing rough clustering in advance. The Canopy algorithm does not require the K value to be specified in advance and has great practical application value. The specific steps include:

[0128] Set distance thresholds T1 and T2, randomly select data points and calculate their distances to other data points, put data points with distances less than T1 into the same Canopy, and repeat this process until all data points are processed.

[0129] S300: Perform knowledge and intent annotation on the corpus sample to construct a training sample and an intent knowledge base.

[0130] In an optional embodiment, the clustered corpus is annotated with knowledge and intent to build a high-quality training sample and intent knowledge base. When annotating knowledge and intent, it is necessary to combine business experience and claims domain knowledge to extract and annotate the intent of the text. For example, the intent entity type (such as scene entity, category entity) and attribute information (such as question points, information expression, etc.) corresponding to the corpus are annotated to form a highly condensed mapping relationship between text and intent.

[0131] like Figure 4 As shown, step S300 includes:

[0132] S301, extracting and annotating the corpus text.

[0133] This embodiment extracts and annotates the text based on business experience and claims-related domain knowledge. The annotated content may include scene entities, category entities, question points, information expressions, questions, scene descriptions, etc.

[0134] S302, constructing model training samples and an intent knowledge base; the intent knowledge base includes attribute information of scene entities, category entities, and corpus samples.

[0135] In one embodiment, the format for performing knowledge and intention annotation on the filtered text is as follows:

[0136] [{"ent_type":value1,"point":value2}]

[0137] Based on business experience and claims domain knowledge, the present invention extracts and annotates the text for intent, and constructs corresponding model training samples and intent knowledge base. In the above formula, ent_type represents the entity type of the intent corresponding to the corpus, including: scene entity, category entity, etc., and point represents the attribute information corresponding to the corpus, including: question point, information expression, question, scene description, etc., in order to better condense the information in the corpus, and also form a highly condensed mapping relationship between text and intent.

[0138] For example:

[0139] text='I didn't bring my ID card'=

[0140] >[{"ent_type":'ID card',"point":'I didn't bring it'}]

[0141] This method well expresses the information characteristics of the sentence's intent and provides a better paradigm for subsequent model recognition.

[0142] Data labeling and classification include detailed labeling of data, including intent labeling, entity labeling, sentiment labeling, etc., to facilitate model understanding and learning. Classify data, such as by claim result (successful, rejected, partially paid) or claim type (vehicle damage, medical expenses, etc.).

[0143] Furthermore, the following methods are included in the annotation:

[0144] Manual annotation: Manually read corpus samples and annotate them according to predefined intent categories and entity categories. This method is suitable for small-scale data sets and can ensure the quality of annotation.

[0145] Semi-automatic annotation tools: Use annotation tools (such as Brat, Doccano, etc.) to assist manual annotation, improve efficiency, and allow multi-person collaboration to support different levels of review and proofreading processes.

[0146] Active learning: The system automatically labels a portion of the data, which is then manually reviewed and corrected. The corrected data is used to further train the model. Through iterative optimization, the system can learn more accurate labeling rules.

[0147] Remote supervised learning: Use labels in existing knowledge bases or datasets to guide the annotation of new datasets, quickly generate candidate annotations through methods such as pattern matching, and then conduct manual review.

[0148] Crowdsourcing labeling: Distributing labeling tasks to a large number of online workers and using collective wisdom to complete labeling tasks requires the design of clear guidance rules and quality control mechanisms.

[0149] Transfer learning: Use pre-trained models (such as BERT, GPT, etc.) to perform preliminary intent recognition and entity extraction on the corpus, and perform fine-tuning or manual correction based on the pre-training results to generate training samples.

[0150] Multi-task learning: Simultaneously train the model on multiple related tasks (such as intent recognition, entity recognition, sentiment analysis, etc.) to improve the generalization ability of the model.

[0151] Interactive learning: Collect feedback from the actual interaction between the model and the user, and dynamically optimize the model's labeling capabilities.

[0152] Rules and patterns: Based on domain knowledge, design rules and patterns to automatically identify and annotate specific intents and entities.

[0153] Knowledge base construction: Use the knowledge of domain experts to build a knowledge base containing information such as intent, entities, attributes, and relationships. The knowledge base can be used to guide the labeling process and can also serve as the basis for model reasoning.

[0154] Incremental labeling: During model training and application, unlabeled or uncertainly labeled samples are continuously collected for incremental labeling and learning.

[0155] Data fusion: Combine data from different sources and types to improve the accuracy and robustness of annotations through data fusion technology.

[0156] The present invention can combine the above methods to achieve better labeling effect and efficiency. At the same time, the privacy and security of the data need to be considered to ensure compliance with relevant laws and regulations.

[0157] S400. Use a large language model combined with a command-guided prompt to generate claims dialogue corpus.

[0158] In this embodiment, by constructing a prompt of specific business experience knowledge, LLM is used to generate dialogue corpus related to the claims field. Specifically, large language models (such as ChatGLM, Qwen-72B, etc.) can be used to generate new training corpus by constructing a command-guided prompt.

[0159] In one embodiment, step S400 includes:

[0160] S401. Construct a command-guided prompt to generate text training data.

[0161] This embodiment uses a large language model and a constructed command-guided prompt to generate new training corpus. For example, the following command prompt can be constructed: "You are a claims assistant. Please construct n actual corpus related to xxx based on the following business experience and knowledge."

[0162] S402: Input prompt into a large language model to obtain a generated corpus sample.

[0163] Furthermore, after prompt is used to generate new training corpus and input it into the large language model, the corresponding output can be obtained as a new training corpus sample.

[0164] S403: Clean and screen the corpus samples to form new claims dialogue corpus to ensure that it meets the training requirements.

[0165] This embodiment uses a large language model to generate text training data for the large language model by constructing a command-guided prompt. For example:

[0166] prompt='You are a claims assistant and a Chinese language master. Please construct actual corpus related to xxx based on the following business experience and knowledge. You need n pieces of data.

[0167] Business experience and knowledge are as follows:

[0168] 1.xxxxxxx

[0169] 2.xxxxxxx

[0170] 3.xxxxxxx ......

[0172] Through the general language ability of LLM, combined with prompt input, the corresponding training corpus samples will be obtained

[0173] new_text=chatglm_block(prompt)

[0174] Among them, prompt is the prompt word of the type intention required by the present invention. After being input into the large language model, the output obtained is new_text.

[0175] By cleaning and screening the new_text data and then iterating through the above step S100, a certain amount of high-quality training corpus can be obtained.

[0176] S500: Perform supervised fine-tuning training on the large language model based on the training samples and the claims dialogue corpus to obtain a fine-tuned large language model.

[0177] This embodiment adds the generated new text to the training set and performs supervised fine-tuning on the model to improve its application performance in the claims industry.

[0178] In one embodiment, if Figure 6 As shown, the step S500 includes:

[0179] S501, constructing a training set using the annotated corpus and the claims dialogue corpus;

[0180] S502, performing supervised fine-tuning training on the large language model on the training set to obtain a fine-tuned model;

[0181] S503: Evaluate the performance of the fine-tuned model and perform iterative optimization based on the evaluation results.

[0182] In this embodiment, the annotated corpus and the generated new corpus are first integrated into a training set, and then a large language model is fine-tuned in a supervised manner, using appropriate optimization algorithms (such as Adam) and loss functions (such as cross entropy loss). The performance of the fine-tuned model is then evaluated, such as using indicators such as accuracy, recall, and F1 score for evaluation. Iterative optimization is then performed based on the evaluation results, such as adjusting training parameters, increasing the number of training rounds, etc., to improve model performance. The large language model is fine-tuned in a supervised manner using the annotated corpus and the generated new corpus to improve the model's understanding and application capabilities in the claims field.

[0183] When constructing a training set, you can include dividing the data set into a training set, a validation set, and a test set. Construct training samples: Use labeled conversation data as training samples. Each sample includes the conversation text and its corresponding label (such as intent, entity, etc.).

[0184] Fine-tuning training may include: Loading a pre-trained model: Loading a pre-trained model and freezing most of its parameters. Unfreezing some layers: Unfreeze some layers of the model as needed to allow for more fine-grained adjustments during fine-tuning. Training: Perform supervised training on the model using the training set data, and adjust the model parameters to adapt to conversational data in the claims field.

[0185] In one embodiment, the step S502 further includes optimizing the model including:

[0186] Adjust the model structure or training strategy based on the results of the validation set and test set;

[0187] Use data augmentation techniques, such as synonym replacement and sentence reorganization, to increase the diversity of training samples;

[0188] Continuously collect new conversation data and perform incremental learning to adapt to changing conversation patterns and user needs.

[0189] Furthermore, after step S500, the method further includes: fine-tuning the large language model to predict the newly generated claims dialogue corpus, and preprocessing the unrecalled corpus as the original corpus again.

[0190] This embodiment uses the new text dataset generated above and the annotated training corpus to perform supervised fine-tuning training on the model. Through repeated iterative optimization and upgrading, the unrecalled corpus can also be sent to step S100 for loop iteration to obtain a fine-tuned LLM model with significantly improved performance.

[0191] Specifically, the model is fine-tuned in a supervised manner using the new_text dataset and annotated training corpus mentioned above to obtain the fine-tuned model model_peft. This model predicts new production corpora again, screens and annotates corpora with unrecalled intents again, and performs repeated iterations and optimization upgrades. The performance of the fine-tuned LLM has been significantly improved.

[0192] model_peft=lora(chatglm,new_text)

[0193] Another embodiment of the present invention provides a claim domain dialogue corpus construction device based on a large language model, such as Figure 7 As shown, the device 1 comprises:

[0194] A preprocessing module 11 is used to collect original corpus in the field of claims and preprocess the original corpus;

[0195] The screening module 12 is used to perform semantic clustering on the preprocessed corpus and screen out corpus samples;

[0196] Annotation module 13, used for performing knowledge and intent annotation on the corpus sample to construct training samples and intent knowledge base;

[0197] Corpus generation module 14, used to generate claims dialogue corpus using a large language model combined with a command-guided prompt;

[0198] The supervised fine-tuning module 15 is used to perform supervised fine-tuning training on the large language model based on the training samples and the claims dialogue corpus to obtain a fine-tuned large language model.

[0199] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which are more suitable for describing the automatic generation and execution process of test data than programs. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0200] In one embodiment, the preprocessing module 11 includes:

[0201] The collection unit is used to collect original corpus in the field of claims;

[0202] The first screening unit is used to screen and remove ASR translated text, long text and non-text data from the original corpus.

[0203] In one embodiment, the screening module 12 includes:

[0204] An extraction unit, used to extract text semantic feature vectors using the BERT model;

[0205] The second screening unit is used to cluster the text semantic feature vectors and remove text corpora with similarity greater than a preset threshold.

[0206] In one embodiment, the marking module 13 includes:

[0207] A tagging unit, used for extracting and tagging the corpus text;

[0208] The construction unit is used to construct model training samples and an intention knowledge base; the intention knowledge base includes attribute information of scene entities, category entities, and corpus samples.

[0209] In one embodiment, the corpus generation module 14 includes:

[0210] The first generating unit is used to construct a command-guided prompt and generate text training data;

[0211] The second generation unit is used to input the prompt into the large language model to obtain a generated corpus sample;

[0212] The third screening unit is used to clean and screen the corpus sample to form a new claims dialogue corpus.

[0213] In one embodiment, the supervised fine-tuning module 15 includes:

[0214] A construction unit, used to construct a training set using the annotated corpus and the claims dialogue corpus;

[0215] The fine-tuning unit is used to perform supervised fine-tuning training on the training set of the large language model to obtain a fine-tuned model;

[0216] The optimization unit is used to evaluate the performance of the fine-tuned model and perform iterative optimization based on the evaluation results.

[0217] In one embodiment, the device 1 further comprises:

[0218] The rule configuration module is used to pre-configure test data generation rules with different attributes and save them in the rule base.

[0219] In one embodiment, the iteration module is used to perform supervised fine-tuning training on the large language model based on training samples and claim dialogue corpus to obtain a fine-tuned large language model.

[0220] The claim domain dialogue corpus construction device based on a large language model of the present invention mainly includes:

[0221] Original corpus collection and preprocessing: A large amount of original corpus is collected from the production field. After preliminary screening and cleaning, irrelevant multimodal data and dirty data are removed, and 5,000 valid corpus samples are retained.

[0222] Sample sampling and initial screening: Use the BERT model to perform semantic clustering on the remaining corpus and remove duplicate texts with high similarity to ensure the diversity and quality of the training data.

[0223] Knowledge and intent annotation: Combine business experience and domain knowledge to perform detailed knowledge and intent annotation on the screened texts and build an intent knowledge base.

[0224] LLM-prompt corpus generation: Use the large language model and constructed command-guided prompts (such as "You are a claims assistant") to generate new training corpus.

[0225] Supervised fine-tuning: Add the generated new text to the training set and perform supervised fine-tuning on the model to improve its application performance in specific industries.

[0226] Another embodiment of the present invention provides a system for constructing dialogue corpus in the claims domain based on a large language model, such as Figure 8 As shown, the system 10 includes:

[0227] One or more processors 110 and memory 120, Figure 8 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0228] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0229] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for constructing a dialogue corpus in the claims field based on a large language model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for constructing a dialogue corpus in the claims field based on a large language model in the above method embodiment.

[0230] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0231] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0232] Collecting original corpus in the field of claims and preprocessing the original corpus;

[0233] Perform semantic clustering on the preprocessed corpus to screen out corpus samples;

[0234] Performing knowledge and intent annotation on the corpus samples to construct training samples and an intent knowledge base;

[0235] Use a large language model combined with command-guided prompts to generate claims dialogue corpus;

[0236] Based on the training samples and claims dialogue corpus, supervised fine-tuning training is performed on the large language model to obtain a fine-tuned large language model.

[0237] In one embodiment, the collecting of original corpus in the field of claims and preprocessing of the original corpus include:

[0238] Collect original corpus in the field of claims;

[0239] ASR translated text, long text and non-text data are screened and removed from the original corpus.

[0240] In one embodiment, the performing semantic clustering on the preprocessed corpus to screen out corpus samples includes:

[0241] Use the BERT model to extract text semantic feature vectors;

[0242] Cluster the text semantic feature vectors and remove text corpora with similarity greater than a preset threshold.

[0243] In one embodiment, the step of performing knowledge and intent annotation on the corpus sample to construct a training sample and an intent knowledge base includes:

[0244] Extracting and annotating the corpus text;

[0245] Construct model training samples and an intent knowledge base; the intent knowledge base includes attribute information of scene entities, category entities, and corpus samples.

[0246] In one embodiment, the method of using a large language model in combination with constructing a command-guided prompt to generate a claim dialogue corpus includes:

[0247] Construct a command-guided prompt to generate text training data;

[0248] Input prompt into a large language model to obtain the generated corpus sample;

[0249] The corpus samples are cleaned and screened to form new claims dialogue corpus.

[0250] In one embodiment, the supervised fine-tuning training of the large language model based on the training samples and the claims dialogue corpus to obtain the fine-tuned large language model includes:

[0251] Use the annotated corpus and claims dialogue corpus to build a training set;

[0252] Perform supervised fine-tuning training on the training set for the large language model to obtain a fine-tuned model;

[0253] The performance of the fine-tuned model is evaluated and iterative optimization is performed based on the evaluation results.

[0254] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the following steps are implemented:

[0255] Collecting original corpus in the field of claims and preprocessing the original corpus;

[0256] Perform semantic clustering on the preprocessed corpus to screen out corpus samples;

[0257] Performing knowledge and intent annotation on the corpus samples to construct training samples and an intent knowledge base;

[0258] Use a large language model combined with command-guided prompts to generate claims dialogue corpus;

[0259] Based on the training samples and claims dialogue corpus, supervised fine-tuning training is performed on the large language model to obtain a fine-tuned large language model.

[0260] In one embodiment, the collecting of original corpus in the field of claims and preprocessing of the original corpus include:

[0261] Collect original corpus in the field of claims;

[0262] ASR translated text, long text and non-text data are screened and removed from the original corpus.

[0263] In one embodiment, the performing semantic clustering on the preprocessed corpus to screen out corpus samples includes:

[0264] Use the BERT model to extract text semantic feature vectors;

[0265] Cluster the text semantic feature vectors and remove text corpora with similarity greater than a preset threshold.

[0266] In one embodiment, the step of performing knowledge and intent annotation on the corpus sample to construct a training sample and an intent knowledge base includes:

[0267] Extracting and annotating the corpus text;

[0268] Construct model training samples and an intent knowledge base; the intent knowledge base includes attribute information of scene entities, category entities, and corpus samples.

[0269] In one embodiment, the method of using a large language model in combination with constructing a command-guided prompt to generate a claim dialogue corpus includes:

[0270] Construct a command-guided prompt to generate text training data;

[0271] Input prompt into a large language model to obtain the generated corpus sample;

[0272] The corpus samples are cleaned and screened to form new claims dialogue corpus.

[0273] In one embodiment, the supervised fine-tuning training of the large language model based on the training samples and the claims dialogue corpus to obtain the fine-tuned large language model includes:

[0274] Use the annotated corpus and claims dialogue corpus to build a training set;

[0275] Perform supervised fine-tuning training on the training set for the large language model to obtain a fine-tuned model;

[0276] The performance of the fine-tuned model is evaluated and iterative optimization is performed based on the evaluation results.

[0277] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0278] The present invention generates dialogue corpus by using a large pre-trained language model, and combines model fine-tuning and prompt construction technology to improve the natural language processing capabilities in specific fields (such as claims), which specifically include:

[0279] Step 1. Pre-trained language model selection

[0280] Choose a suitable pre-trained language model, such as ChatGLM3, Qwen-72B, etc. These models have been trained on a large amount of text and have the ability to understand language and generate text.

[0281] Step 2. Dialogue corpus generation

[0282] Unconditional generation: Use a pre-trained model to generate dialogues based on randomly initialized seeds or templates. Conditional generation: Give the model a specific context or condition to generate dialogues on a specific topic or scenario.

[0283] Step 3. Dialogue data screening and cleaning

[0284] Filter the generated conversations, remove irrelevant or low-quality conversations, clean the data, and correct grammatical errors or unreasonable expressions.

[0285] Step 4. Model fine-tuning

[0286] Domain data preparation: Collect conversation data in the claims domain as a fine-tuning set.

[0287] Fine-tuning training: Use domain-specific conversation data to fine-tune the pre-trained model to make the model better adapt to the language characteristics of the claims field.

[0288] Step 5. Prompt construction technique

[0289] Design prompt templates to guide the model to generate conversations with specific intents. Prompts can contain domain-specific keywords, questions, or instructions.

[0290] Step 6. Intent annotation

[0291] The fine-tuned model can be used to perform intent recognition and entity extraction on the generated conversations. The labeling results of the model can also be manually reviewed and corrected if necessary.

[0292] Step 7. Construction of high-quality training corpus

[0293] Include intent-annotated conversations into the training corpus, and continuously optimize and update the corpus to reflect the latest conversation patterns and domain knowledge.

[0294] Step 8. Supervised model training

[0295] Use the constructed high-quality training corpus to train the supervised model. The training can include multiple natural language processing tasks such as intent recognition, sentiment analysis, entity recognition, etc.

[0296] Step 9. Effect evaluation and optimization

[0297] The model performance is evaluated through standard natural language processing evaluation indicators (such as accuracy, recall rate, F1 score, etc.), and the model is tuned according to the evaluation results, including parameter adjustment and model structure optimization.

[0298] Step 10. Application and feedback

[0299] Deploy the trained model to practical applications, such as intelligent customer service systems, and collect user feedback and actual conversation data for further learning and optimization of the model.

[0300] In summary, the present invention discloses a method, device, system and medium for constructing dialogue corpus in the field of claims based on a large language model. The method generates high-quality dialogue corpus in the field of claims through automated corpus collection, preprocessing, semantic clustering, knowledge and intent annotation, dialogue corpus generation, data cleaning and supervised fine-tuning. The automated generation of dialogue reduces the workload of manually constructing corpus and improves the efficiency of data preparation. In addition, the relevance and accuracy of dialogue data are ensured through model fine-tuning and intent annotation, and the model is better adapted to the needs of specific fields through fine-tuning and prompt technology.

[0301] In addition, by continuously collecting feedback and data, the model of the present invention can continuously learn and adapt to new dialogue scenarios.

[0302] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0303] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for constructing dialogue corpus in the claims field based on a large language model. It is characterized in that include: Collecting original corpus in the field of claims and preprocessing the original corpus; Perform semantic clustering on the preprocessed corpus to screen out corpus samples; Performing knowledge and intent annotation on the corpus samples to construct training samples and an intent knowledge base; Use a large language model combined with command-guided prompts to generate claims dialogue corpus; Based on the training samples and claims dialogue corpus, supervised fine-tuning training is performed on the large language model to obtain a fine-tuned large language model.

2. The method for constructing dialogue corpus in the claims field based on a large language model according to claim 1, It is characterized in that Also includes: Fine-tune the large language model to predict the newly generated claims dialogue corpus, and preprocess the unrecalled corpus again as the original corpus.

3. The method for constructing dialogue corpus in the claims field based on a large language model according to claim 1, It is characterized in that The step of collecting the original corpus in the field of claims and preprocessing the original corpus includes: Collect original corpus in the field of claims; ASR translated text, long text and non-text data are screened and removed from the original corpus.

4. The method for constructing dialogue corpus in the claims field based on a large language model according to claim 1, It is characterized in that The step of performing semantic clustering on the preprocessed corpus to screen out corpus samples includes: Use the BERT model to extract text semantic feature vectors; Cluster the text semantic feature vectors and remove text corpora with similarity greater than a preset threshold.

5. The method for constructing dialogue corpus in the claims field based on a large language model according to claim 1, It is characterized in that The step of performing knowledge and intent annotation on the corpus sample to construct a training sample and an intent knowledge base comprises: Extracting and annotating the corpus text; Construct model training samples and an intent knowledge base; the intent knowledge base includes attribute information of scene entities, category entities, and corpus samples.

6. The method for constructing dialogue corpus in the claims field based on a large language model according to claim 1, It is characterized in that The step of using a large language model in combination with constructing a command-guided prompt to generate a claims dialogue corpus includes: Construct a command-guided prompt to generate text training data; Input prompt into a large language model to obtain the generated corpus sample; The corpus samples are cleaned and screened to form new claims dialogue corpus.

7. The method for constructing dialogue corpus in the claims field based on a large language model according to claim 1, It is characterized in that The step of performing supervised fine-tuning training on the large language model based on the training samples and the claims dialogue corpus to obtain the fine-tuned large language model includes: Use the annotated corpus and claims dialogue corpus to build a training set; Perform supervised fine-tuning training on the training set for the large language model to obtain a fine-tuned model; The performance of the fine-tuned model is evaluated and iterative optimization is performed based on the evaluation results.

8. A device for constructing dialogue corpus in the claims domain based on a large language model. It is characterized in that include: A preprocessing module, used to collect original corpus in the field of claims and preprocess the original corpus; The screening module is used to perform semantic clustering on the preprocessed corpus and screen out corpus samples; Annotation module, used for performing knowledge and intent annotation on the corpus samples to construct training samples and intent knowledge base; Corpus generation module, which is used to generate claims dialogue corpus using a large language model combined with command-guided prompts; The supervised fine-tuning module is used to perform supervised fine-tuning training on the large language model based on training samples and claims dialogue corpus to obtain a fine-tuned large language model.

9. A claims domain dialogue corpus construction system based on a large language model. It is characterized in that The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing dialogue corpus in the claims field based on a large language model as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, It is characterized in that The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the method for constructing dialogue corpus in the claims field based on a large language model as described in any one of claims 1 to 7.

Images