Method and device for determining training data of large model in insurance field, equipment and medium

Abstract

The invention provides a training data determination method and device for an insurance field large model, equipment and a medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining at least one piece of insurance field text data and at least one piece of general text data; the general text data comprises insurance field text data and text data of other fields except the insurance field text data; filtering the insurance field text data and the general text data to obtain first training data; performing clustering processing on the first training data to obtain second training data; determining target training data based on the second training data; the target training data is used for training the insurance field large model. According to the method, wide data is acquired, and a series of processing is performed on the data, so that the memory loss of the large model is relieved, and the generalization ability of the large model is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for determining training data of a large model in the insurance field. Background Art

[0002] In the process of model training in the insurance field, if the training samples are too concentrated in a certain direction or scenario, during secondary training or fine-tuning in the vertical field, the model may lose some of its original general capabilities and become only good at processing data in the current training scenario, while its processing capabilities for other scenarios decrease, resulting in "memory loss" of the large model, thereby reducing the generalization ability of the large model. Summary of the invention

[0003] The present invention provides a method, device, equipment and medium for determining training data of a large model in the insurance field, so as to solve the defect of "memory loss" of the model caused by too concentrated training samples in the prior art, and achieve the goal of obtaining a wide range of data and performing a series of data processing to alleviate the "memory loss" of the large model and improve the generalization ability of the large model.

[0004] The present invention provides a method for determining training data of a large model in the insurance field, comprising: Acquire at least one insurance field text data and at least one general text data; the general text data is text data including the insurance field text data and other fields except the insurance field text data; Filtering the insurance field text data and the general text data to obtain first training data; Performing clustering processing on the first training data to obtain second training data; Based on the second training data, target training data is determined; the target training data is used to train a large model in the insurance field.

[0005] According to a method for determining training data for a large model in the insurance field provided by the present invention, the filtering process includes deduplication and cleaning, and the filtering process is performed on the insurance field text data and the general text data to obtain the first training data, including: determining the similarity between any two text data in the comprehensive text data; the comprehensive text data includes all the insurance field text data and all the general text data; based on the similarity, deduplication is performed on the insurance field text data and the general text data to obtain the deduplicated text data; based on the cleaning rules, the deduplication text data is cleaned to obtain the first training data; the cleaning rules are used to determine low-quality text data.

[0006] According to a method for determining training data for a large model in the insurance field provided by the present invention, clustering the first training data to obtain second training data includes: converting each text data in the first training data into vector data; performing regularization processing and dimensionality reduction processing on the vector data respectively to obtain processed vector data; randomly selecting K vector data from the processed vector data as initial clustering centers; K is an integer greater than or equal to 1; calculating the distance between each vector data except the initial clustering center and each of the initial clustering centers; based on the distance, clustering the vector data to obtain K first clustering results; the first clustering results are used to characterize the text data contained in the initial clustering centers; based on the K first clustering results and the initial clustering centers, determining the second training data.

[0007] According to a method for determining training data for a large model in the insurance field provided by the present invention, the second training data is determined based on the K first clustering results and the initial clustering centers: including: determining the clustering category corresponding to each of the text data based on the K first clustering results and the initial clustering centers; using a classification model to determine the classification category corresponding to each text data in the first training data; based on the classification category, verifying the clustering category to obtain a target clustering result; and determining the second training data based on the target clustering result.

[0008] According to a method for determining training data for a large model in the insurance field provided by the present invention, the second training data is determined based on the target clustering result, including: determining the amount of text data contained in each clustering category in the target clustering result; when the amount of text data is greater than a text data amount threshold, downsampling the text data in the clustering category to obtain the second training data.

[0009] According to a method for determining training data of a large model in the insurance field provided by the present invention, the target training data is determined based on the second training data, including: generating the user's insurance data based on the generation model and the first mapping relationship; the first mapping relationship is used to characterize the relationship between the user's age status, the user's health status and the user's insurance data; generating the user's claim data based on the generation model and the second mapping relationship; the second mapping relationship is used to characterize the relationship between the user's insurance, the user's medical condition and the user's claim data; determining the target training data based on the second training data, the insurance data and the claim data.

[0010] According to a method for determining training data for a large model in the insurance field provided by the present invention, the target training data is determined based on the second training data, the insured data and the claims data, including: fusing the second training data, the insured data and the claims data to obtain third training data; debugging the proportion of general text data in the third training data and the insurance field text data in the third training data to obtain fourth training data; adjusting the proportion of Chinese and English in the general text data in the fourth training data and the proportion of Chinese and English in the insurance field text data in the fourth training data to obtain the target training data.

[0011] The present invention also provides a training data determination device for a large model in the insurance field, comprising: An acquisition module, used to acquire at least one insurance field text data and at least one general text data; the general text data is text data including the insurance field text data and other fields except the insurance field text data; A filtering module, used for filtering the insurance field text data and the general text data to obtain first training data; A clustering module, used for performing clustering processing on the first training data to obtain second training data; A determination module is used to determine target training data based on the second training data; the target training data is used to train a large model in the insurance field.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for determining training data for a large insurance field model as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining training data for a large model in the insurance field as described in any one of the above.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for determining training data for a large model in the insurance field.

[0015] The present invention provides a method, device, equipment and medium for determining training data of a large model in the insurance field. By acquiring at least one insurance field text data and at least one general text data, the insurance field text data and the general text data are subjected to a series of processes such as filtering processing and clustering processing to obtain target training data. In this way, new field knowledge can be learned while general capabilities can be maintained. The quality of text data is improved by filtering processing, and the categories of text data are obtained by clustering processing, thereby increasing the diversity of the categories of training data, alleviating the problem of "memory loss" of the large model and improving the generalization capability of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the method for determining training data for a large model in the insurance field provided by the present invention.

[0018] Figure 2 This is the second flowchart of determining the training data of the large model in the insurance field provided by the present invention.

[0019] Figure 3 It is a structural diagram of the training data determination device for the large model in the insurance field provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Combine the following Figure 1-Figure 2 The method for determining training data for a big model in the insurance field of the present invention can be used to train any model in the insurance field, including a big model with question-answering capabilities. The executor of the method can be an electronic device, or a method for determining training data for a big model in the insurance field set in the electronic device. The device for determining training data for the big model in the insurance field can be implemented by software, hardware, or a combination of both.

[0023] Figure 1 This is one of the flow charts of the method for determining training data for a large insurance model provided by the present invention. Figure 1 As shown, the method includes the following: Step 101: Acquire at least one insurance field text data and at least one general text data.

[0024] The general text data includes the insurance field text data and text data in other fields except the insurance field text data.

[0025] Here, the method of obtaining text data in the insurance field can be to obtain it from public data sets, or to obtain it from information on the insurance field collected by the company, or to crawl insurance-related data from the website using crawler rule strategies. Among them, text data in the insurance field can include insurance product information, insurance contract information, user insurance data, user claims data, insurance company information, insurance market information, etc.

[0026] Here, general text data can be understood as all text materials on the Internet. General text data can be obtained from public data sets such as LIMA, OpenOrca, BELLE or MOSS.

[0027] Step 102: Filter the insurance field text data and the general text data to obtain first training data.

[0028] It should be noted that the training data is actually a training set containing multiple text data, including multiple insurance field text data and multiple general text data.

[0029] Furthermore, the filtering process includes deduplication and cleaning, and the filtering process of the insurance field text data and the general text data to obtain the first training data includes: determining the similarity between any two text data in the comprehensive text data; the comprehensive text data includes all the insurance field text data and all the general text data; based on the similarity, deduplication of the insurance field text data and the general text data to obtain the deduplicated text data; based on the cleaning rules, cleaning the deduplicated text data to obtain the first training data; the cleaning rules are used to determine low-quality text data.

[0030] Here, deduplication refers to removing duplicate or similar text data and retaining only one text data. Cleaning refers to removing low-quality text data.

[0031] Here, the method for determining text similarity may be any appropriate method, such as Similarity Hashing (SimHash), text matching model, cosine similarity, etc.

[0032] Exemplarily, the specific method of using SimHash to determine text similarity is: text preprocessing: extract keywords from text data; feature extraction: preset rules to extract key features from the preprocessed text, such as word frequency, TF-IDF value, etc.; weighted superposition: weight each feature, and superimpose the weighted values ​​of all features to generate a total feature vector; hash processing: hash the total feature vector to obtain a binary vector of fixed length, namely the SimHash value. Compare the SimHash values ​​corresponding to each text, where the closer the SimHash values ​​are, the more similar the text data is, and similar texts need to be deleted.

[0033] Here, the cleaning rules can be regular or filtering rules, for example, if there are many special symbols or semantically inconsistent text in the text.

[0034] Specifically, after obtaining text data in the insurance field and general text data, the similarity between any two or more text data is calculated, and the text data is deduplicated. Then, according to the cleaning rules (such as texts with many special symbols and semantically inconsistent texts), low-quality text data is filtered out to obtain the first training data.

[0035] In an embodiment of the present invention, by deduplicating text data, filtering out similar text data, reducing overtraining, filtering out low-quality text, and improving the quality of text data, the generalization ability of the model is improved, thereby alleviating the problem of "memory loss" in large models.

[0036] Step 103: cluster the first training data to obtain second training data.

[0037] Here, clustering processing refers to classifying the first training data, wherein the clustering method may be any appropriate clustering method, such as K-means, K-center, and the like.

[0038] Furthermore, the clustering process is performed on the first training data to obtain the second training data, including: converting each text data in the first training data into vector data; performing regularization and dimensionality reduction processing on the vector data respectively to obtain processed vector data; randomly selecting K vector data as initial cluster centers from the processed vector data; K is an integer greater than or equal to 1; calculating the distance between each vector data except the initial cluster center and each of the initial cluster centers; based on the distance, clustering the vector data to obtain K first clustering results; the first clustering results are used to characterize the text data contained in the initial cluster centers; based on the K first clustering results and the initial cluster centers, determining the second training data.

[0039] Here, the way of converting vector data may be any appropriate method, such as the Sentence-Bert model, the InferSent model, etc.

[0040] Here, regularization is used to prevent the model from overfitting. By introducing regularization terms, the model is subject to additional restrictions during training, making it smoother or sparser, and avoiding learning noise and details in the training data. Regularization usually includes types such as L1 regularization and L2 regularization, which penalize the parameters of the model in different ways.

[0041] Here, the dimensionality reduction process is to reduce the number of data features, convert high-dimensional data into low-dimensional space, and retain important information in the original data as much as possible. The dimensionality reduction process can be any appropriate method, such as principal component analysis, linear discriminant analysis, singular value decomposition, etc.

[0042] It should be noted that, after obtaining K first clustering results, the distance between the text data in each first clustering result and the corresponding initial clustering center is determined. When the maximum distance is less than or equal to the distance threshold, it means that the first clustering result is accurate; otherwise, the average value of the text data in each first clustering result is calculated, and the text data closest to the average value is used as the second clustering center, and the text data contained in each second clustering center is re-determined until the maximum distance between the cluster center and the text data in the clustering result is less than or equal to the distance threshold, and the clustering result is obtained.

[0043] In the implementation of the present invention, regularization processing is performed on the first training data to avoid overfitting of the model, dimensionality reduction processing is performed to reduce the complexity of the data, and clustering processing is performed to ensure the balance of various scenarios and features, thereby improving the accuracy of the large model.

[0044] Further, the second training data is determined based on the K first clustering results and the initial clustering center, including: determining the cluster category corresponding to each of the text data based on the K first clustering results and the initial clustering center; using a classification model to determine the classification category corresponding to each text data in the first training data; based on the classification category, verifying the cluster category to obtain a target clustering result; and determining the second training data based on the target clustering result.

[0045] Here, the classification model can be any suitable classification model, such as Qwen1.5 or Generative Pre-trained Transformer 4 (GPT4).

[0046] It should be noted that verification is essentially to determine whether the clustering category is the same as the classification category. If the clustering category is different from the classification category, the text data corresponding to the clustering category is removed.

[0047] In the embodiment of the present invention, the accuracy of text data classification is improved by verifying the clustering categories, and the accuracy of the model can be improved by using the text data to train the model.

[0048] Further, the determining the second training data based on the target clustering result includes: Determining the amount of text data contained in each cluster category in the target clustering result; When the amount of text data is greater than a text data amount threshold, down-sampling is performed on the text data in the cluster category to obtain the second training data.

[0049] Here, the text data volume threshold may be any appropriate value. When the text data volume corresponding to a certain clustering category is too large, it will lead to imbalance of multiple text data.

[0050] Here, the downsampling method may be any method, such as random undersampling, easy ensemble, balance cascade, etc.

[0051] In the embodiment of the present invention, by downsampling the categories with large amounts of text data, the balance and diversity of the text data are ensured.

[0052] Step 104: determine target training data based on the second training data; the target training data is used to train a large model in the insurance field.

[0053] Further, the target training data is determined based on the second training data, including: generating the user's insurance data based on the generation model and the first mapping relationship; the first mapping relationship is used to characterize the relationship between the user's age status, the user's health status and the user's insurance data; generating the user's claim data based on the generation model and the second mapping relationship; the second mapping relationship is used to characterize the relationship between the user's insurance, the user's medical condition and the user's claim data; the target training data is determined based on the second training data, the insurance data and the claim data.

[0054] Here, the generative model can be any suitable model, such as the GPT4 model.

[0055] Here, insurance data refers to the insurance recommended based on the user's age and health status. For example, if the user is 58 years old and has no major diseases, medical insurance is recommended.

[0056] The first mapping relationship may be the user's age and health status and the recommended insurance policy, for example, what type of insurance is recommended based on the user's age and health status.

[0057] Here, the claims data includes how to make claims to users based on their insurance and medical condition, for example, how to make claims to users for medical insurance they have purchased.

[0058] Specifically, the age and health status of some users in the original data and the recommended insurance strategy are fed to GPT4 to generate insurance data corresponding to different ages and different health statuses; the user's insurance status, medical condition and final claims strategy are fed to GPT4 to generate claims data corresponding to different insurances and different medical conditions.

[0059] In the embodiment of the present invention, additional insurance data and claim data are obtained through the first mapping relationship, the second mapping relationship and the generation model, the training data is expanded, the diversity of the training data is increased, and the generalization ability and accuracy of the model are improved.

[0060] Furthermore, the target training data is determined based on the second training data, the insured data and the claims data, including: fusing the second training data, the insured data and the claims data to obtain third training data; adjusting the proportion of general text data in the third training data and the proportion of insurance field text data in the third training data to obtain fourth training data; adjusting the proportion of Chinese and English in the general text data in the fourth training data and the proportion of Chinese and English in the insurance field text data in the fourth training data to obtain the target training data.

[0061] Here, the fusion processing may be to directly add the insurance data and the claim data to the second training data; or it may be to fuse the three in a weighted manner. The present invention does not limit the fusion processing.

[0062] Here, the debugging method can be to continuously change the proportion of general text data and insurance field text data to train the model, judge the performance of each model, and determine the proportion of text data corresponding to the model with the best performance as the fourth training data.

[0063] Here, the ratio of general text data to insurance field text data can be any appropriate ratio, such as 1:1, 4:1, etc.

[0064] Here, the ratio of Chinese to English can be any appropriate ratio, such as 1:1, 3:1, etc.

[0065] In order to ensure that the trained model can recognize Chinese and English, English text data is added to the training data. In order to improve the performance of the model, it is necessary to continuously adjust the ratio of Chinese and English.

[0066] Specifically, the model is trained by continuously adjusting the ratio of Chinese to English, and the performance of multiple models is compared. The text data of the Chinese to English ratio corresponding to the model with the best performance is determined as the target training data.

[0067] In an embodiment of the present invention, by adjusting the proportion of general data and insurance field text data in the training data, as well as the proportion of Chinese and English training data, the model can not only learn new field knowledge but also maintain general capabilities. At the same time, the model can recognize Chinese and English, thereby improving the model's functionality and the user experience of different users.

[0068] The following is an application scenario of a method for determining training data for a large model in the insurance field provided by the present invention.

[0069] Figure 2 This is the second flow chart of determining the training data of the insurance field big model provided by the present invention, such as Figure 2 As shown, the method includes the following: Step S201: Collect general text data and insurance field text data.

[0070] Step S202: De-duplicate the general text data and the insurance field text data according to the similarity between the text data to obtain de-duplicated text data.

[0071] Step S203: filter out low-quality texts in the deduplicated text data to obtain first training data.

[0072] Step S204: perform regularization processing and dimensionality reduction processing on the first training data to obtain processed vector data.

[0073] Step S205: cluster the data using the KCenter algorithm to divide the data into N categories.

[0074] Step S206: Use the classification model to perform classification verification on the clustered text data, and filter out text data that does not meet the classification requirements.

[0075] Step S207: down-sample the excessive text data in the cluster category to obtain second training data.

[0076] Step S208: Add the insurance data and the claim data to the second training data to obtain the third training data.

[0077] Step S209, adjusting the proportion of general text data and insurance field text data in the third training data, the proportion of Chinese and English in the general text data, and the proportion of Chinese and English in the insurance field text data to obtain target training data.

[0078] Here, after a large number of training attempts, we finally used a 1:1 ratio of general data to insurance field data for training in the secondary pre-training, with a total of 20G of training data.

[0079] In the instruction fine-tuning stage, the large model has the ability to answer questions and communicate directly with users through instruction fine-tuning training using 80,000 text data. The ratio of general data to insurance data is 4:1. The ratio of Chinese and English in general data is 1:1, and the ratio of Chinese and English in insurance field text data is 3:1.

[0080] The following is a description of the training data determination device for the big model in the insurance field provided by the present invention. The training data determination device for the big model in the insurance field described below and the training data determination method for the big model in the insurance field described above can be referenced to each other.

[0081] Figure 3 : is a schematic diagram of the structure of the training data determination device of the insurance field large model provided by the present invention, such as Figure 3 As shown, the training data determination device 300 for the insurance field big model includes: The acquisition module 301 is used to acquire at least one insurance field text data and at least one general text data; the general text data is text data including the insurance field text data and other fields except the insurance field text data; A filtering module 302, configured to filter the insurance field text data and the general text data to obtain first training data; A clustering module 303, configured to perform clustering processing on the first training data to obtain second training data; The determination module 304 is used to determine target training data based on the second training data; the target training data is used to train a large model in the insurance field.

[0082] In some embodiments, the filtering process includes deduplication and cleaning, and the filtering module 302 is specifically used to: determine the similarity between any two text data in the comprehensive text data; the comprehensive text data includes all the insurance field text data and all the general text data; based on the similarity, the insurance field text data and the general text data are deduplicated to obtain the deduplicated text data; based on the cleaning rules, the deduplicated text data is cleaned to obtain the first training data; the cleaning rules are used to determine low-quality text data.

[0083] In some embodiments, the clustering module 303 is specifically used to: convert each text data in the first training data into vector data; perform regularization processing and dimensionality reduction processing on the vector data respectively to obtain processed vector data; randomly select K vector data from the processed vector data as initial clustering centers; K is an integer greater than or equal to 1; calculate the distance between each vector data except the initial clustering center and each initial clustering center; based on the distance, cluster the vector data to obtain K first clustering results; the first clustering result is used to characterize the text data contained in the initial clustering center; based on the K first clustering results and the initial clustering center, determine the second training data.

[0084] In some embodiments, the clustering module 303 is further specifically used to: determine the clustering category corresponding to each of the text data based on the K first clustering results and the initial clustering center; determine the classification category corresponding to each text data in the first training data using a classification model; verify the clustering category based on the classification category to obtain a target clustering result; and determine the second training data based on the target clustering result.

[0085] In some embodiments, the clustering module 303 is further specifically used to: determine the amount of text data contained in each clustering category in the target clustering result; when the amount of text data is greater than a text data amount threshold, downsample the text data in the clustering category to obtain the second training data.

[0086] In some embodiments, the determination module 304 is specifically used to: generate the user's insurance data based on the generation model and the first mapping relationship; the first mapping relationship is used to characterize the relationship between the user's age status, the user's health status and the user's insurance data; generate the user's claim data based on the generation model and the second mapping relationship; the second mapping relationship is used to characterize the relationship between the user's insurance, the user's medical condition and the data claim data; determine the target training data based on the second training data, the insurance data and the claim data.

[0087] In some embodiments, the determination module 304 is further specifically used to: fuse the second training data, the insurance data and the claim data to obtain third training data; adjust the proportion of general text data in the third training data and the proportion of insurance field text data in the third training data to obtain fourth training data; adjust the proportion of Chinese and English in the general text data in the fourth training data and the proportion of Chinese and English in the insurance field text data in the fourth training data to obtain the target training data.

[0088] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the training data determination method of the insurance field large model, the method comprising: obtaining at least one insurance field text data and at least one general text data; the general text data is text data of other fields including the insurance field text data and the insurance field text data; filtering the insurance field text data and the general text data to obtain first training data; clustering the first training data to obtain second training data; determining target training data based on the second training data; the target training data is used to train the insurance field large model.

[0089] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training data determination method for the insurance field big model provided by the above-mentioned methods, and the method includes: obtaining at least one insurance field text data and at least one general text data; the general text data is text data containing the insurance field text data and other fields except the insurance field text data; filtering the insurance field text data and the general text data to obtain first training data; clustering the first training data to obtain second training data; determining target training data based on the second training data; the target training data is used to train the insurance field big model.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training data determination method for the insurance field big model provided by the above-mentioned methods, the method comprising: obtaining at least one insurance field text data and at least one general text data; the general text data is text data containing the insurance field text data and other fields except the insurance field text data; filtering the insurance field text data and the general text data to obtain first training data; clustering the first training data to obtain second training data; determining target training data based on the second training data; the target training data is used to train the insurance field big model.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining training data for a large model in the insurance field, characterized in that: include: Acquire at least one insurance field text data and at least one general text data; the general text data is text data including the insurance field text data and other fields except the insurance field text data; Filtering the insurance field text data and the general text data to obtain first training data; Performing clustering processing on the first training data to obtain second training data; Determining target training data based on the second training data; The target training data is used to train a large model in the insurance field.

2. The method for determining training data for a large insurance model according to claim 1, characterized in that: The filtering process includes deduplication and cleaning, and the filtering process is performed on the insurance field text data and the general text data to obtain the first training data, including: Determine the similarity between any two text data in the comprehensive text data; the comprehensive text data includes all the insurance field text data and all the general text data; Based on the similarity, the insurance field text data and the general text data are deduplicated to obtain deduplicated text data; Based on cleaning rules, the deduplicated text data is cleaned to obtain the first training data; the cleaning rules are used to determine low-quality text data.

3. The method for determining training data for a large insurance model according to claim 1, characterized in that: The clustering process is performed on the first training data to obtain second training data, including: Convert each text data in the first training data into vector data; Performing regularization processing and dimensionality reduction processing on the vector data respectively to obtain processed vector data; Randomly selecting K vector data from the processed vector data as initial cluster centers; K is an integer greater than or equal to 1; Calculate the distance between each vector data except the initial cluster center and each of the initial cluster centers; Based on the distance, clustering is performed on the vector data to obtain K first clustering results; the first clustering results are used to characterize the text data contained in the initial cluster center; The second training data is determined based on the K first clustering results and the initial clustering centers.

4. The method for determining training data for a large insurance model according to claim 3, characterized in that: The determining the second training data based on the K first clustering results and the initial clustering center includes: Based on the K first clustering results and the initial clustering center, determining the clustering category corresponding to each of the text data; Determine the classification category corresponding to each text data in the first training data by using a classification model; Based on the classification category, verify the clustering category to obtain a target clustering result; Based on the target clustering result, the second training data is determined.

5. The method for determining training data for a large insurance model according to claim 4, characterized in that: The determining the second training data based on the target clustering result includes: Determining the amount of text data contained in each cluster category in the target clustering result; When the amount of text data is greater than a text data amount threshold, down-sampling is performed on the text data in the cluster category to obtain the second training data.

6. The method for determining training data for a large insurance model according to any one of claims 1 to 5, characterized in that: The determining target training data based on the second training data includes: Generate the user's insurance data based on the generation model and the first mapping relationship; the first mapping relationship is used to characterize the relationship between the user's age status, the user's health status and the user's insurance data; Based on the generation model and the second mapping relationship, the user's claim data is generated; the second mapping relationship is used to characterize the relationship between the user's insurance, the user's medical condition and the user's claim data; The target training data is determined based on the second training data, the insurance data and the claim data.

7. The method for determining training data for a large insurance model according to claim 6, characterized in that: The determining the target training data based on the second training data, the insurance data and the claim data comprises: fusing the second training data, the insurance data and the claim data to obtain third training data; Debugging the proportion of general text data in the third training data and insurance field text data in the third training data to obtain fourth training data; The target training data is obtained by adjusting the proportion of Chinese and English in the general text data in the fourth training data and the proportion of Chinese and English in the insurance field text data in the fourth training data.

8. A training data determination device for a large insurance model, characterized in that: include: An acquisition module, used to acquire at least one insurance field text data and at least one general text data; the general text data is text data including the insurance field text data and other fields except the insurance field text data; A filtering module, used for filtering the insurance field text data and the general text data to obtain first training data; A clustering module, used for performing clustering processing on the first training data to obtain second training data; A determination module, configured to determine target training data based on the second training data; The target training data is used to train a large model in the insurance field.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the method for determining training data for the insurance field big model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for determining training data for the insurance field big model as described in any one of claims 1 to 7.

Images