Method, device and equipment for determining training data of insurance field large model and medium

Abstract

The application provides a training data determination method, device and equipment for an insurance field large model and a medium, and relates to the technical field of data processing. The method comprises the following steps: obtaining at least one insurance field text data and at least one general text data; the general text data comprises the insurance field text data and text data of other fields except the insurance field text data; performing filtering processing on the insurance field text data and the general text data to obtain first training data; performing clustering processing on the first training data to obtain second training data; determining target training data based on the second training data; and the target training data is used for training the insurance field large model. The application can alleviate the "memory loss" of the large model by obtaining extensive data and performing a series of processing on the data, and improve the generalization ability of the large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a training data determination method, device and equipment for a large model in the insurance field and a medium. BACKGROUND

[0002] In the process of training a model in the insurance field, if the training samples are too concentrated in a certain direction or scene, the model may lose some original general capabilities and become good at processing only the data of the current training scene in the secondary training or fine-tuning in the vertical field, while the processing capability for other scenes decreases, resulting in "memory loss" of the large model, thereby reducing the generalization capability of the large model. SUMMARY

[0003] The present application provides a training data determination method, device and equipment for a large model in the insurance field to solve the defect that the model has "memory loss" due to the excessive concentration of training samples, and to realize the alleviation of "memory loss" of the large model and the improvement of the generalization capability of the large model by obtaining extensive data and processing the data in a series of ways.

[0004] The present application provides a training data determination method for a large model in the insurance field, comprising:

[0005] obtaining at least one insurance field text data and at least one general text data; the general text data contains the insurance field text data and the text data of other fields except the insurance field text data;

[0006] filtering the insurance field text data and the general text data to obtain first training data;

[0007] clustering the first training data to obtain second training data;

[0008] determining target training data based on the second training data; the target training data is used to train a large model in the insurance field.

[0009] According to the training data determination method for a large model in the insurance field provided by the present application, the filtering process includes deduplication and cleaning, and the filtering of the insurance field text data and the general text data to obtain the first training data comprises: determining the similarity between any two text data in the comprehensive text data; the comprehensive text data includes all the insurance field text data and all the general text data; based on the similarity, the insurance field text data and the general text data are deduplicated to obtain the text data after deduplication; based on the cleaning rule, the text data after deduplication is cleaned to obtain the first training data; the cleaning rule is used to determine low-quality text data.

[0010] The method for determining training data of an insurance field large model provided by the application comprises the following steps: converting each text data in the first training data into vector data; performing regularization processing and dimension reduction processing on the vector data respectively to obtain processed vector data; randomly selecting K vector data in the processed vector data as initial clustering centers; the K is an integer greater than or equal to 1; calculating the distance between each vector data and each initial clustering center; based on the distance, the vector data is clustered to obtain K first clustering results; the first clustering result is used to represent the text data contained in the initial clustering center; and the second training data is determined based on the K first clustering results and the initial clustering centers.

[0011] The method for determining training data of an insurance field large model provided by the application comprises the following steps: converting each text data in the first training data into vector data; performing regularization processing and dimension reduction processing on the vector data respectively to obtain processed vector data; randomly selecting K vector data in the processed vector data as initial clustering centers; the K is an integer greater than or equal to 1; calculating the distance between each vector data and each initial clustering center; based on the distance, the vector data is clustered to obtain K first clustering results; the first clustering result is used to represent the text data contained in the initial clustering center; and the second training data is determined based on the K first clustering results and the initial clustering centers.

[0012] The method for determining training data of an insurance field large model provided by the application comprises the following steps: converting each text data in the first training data into vector data; performing regularization processing and dimension reduction processing on the vector data respectively to obtain processed vector data; randomly selecting K vector data in the processed vector data as initial clustering centers; the K is an integer greater than or equal to 1; calculating the distance between each vector data and each initial clustering center; based on the distance, the vector data is clustered to obtain K first clustering results; the first clustering result is used to represent the text data contained in the initial clustering center; and the second training data is determined based on the K first clustering results and the initial clustering centers.

[0013] The method for determining training data of an insurance field large model provided by the application comprises the following steps: converting each text data in the first training data into vector data; performing regularization processing and dimension reduction processing on the vector data respectively to obtain processed vector data; randomly selecting K vector data in the processed vector data as initial clustering centers; the K is an integer greater than or equal to 1; calculating the distance between each vector data and each initial clustering center; based on the distance, the vector data is clustered to obtain K first clustering results; the first clustering result is used to represent the text data contained in the initial clustering center; and the second training data is determined based on the K first clustering results and the initial clustering centers.

[0014] According to the insurance field large model training data determination method provided by the application, the target training data is determined based on the second training data, the insurance application data and the insurance claim data, and the method comprises the following steps: fusing the second training data, the insurance application data and the insurance claim data to obtain third training data; adjusting the proportion of general text data in the third training data and the proportion of insurance field text data in the third training data to obtain fourth training data; adjusting the proportion of Chinese and English in the general text data in the fourth training data and the proportion of Chinese and English in the insurance field text data in the fourth training data to obtain the target training data.

[0015] The application further provides an insurance field large model training data determination device, comprising:

[0016] An acquisition module is configured to acquire at least one insurance field text data and at least one general text data; the general text data comprises the insurance field text data and text data of other fields except the insurance field text data;

[0017] A filtering module is configured to filter the insurance field text data and the general text data to obtain first training data;

[0018] A clustering module is configured to cluster the first training data to obtain second training data;

[0019] A determination module is configured to determine target training data based on the second training data; the target training data is used to train an insurance field large model.

[0020] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the insurance field large model training data determination method according to any one of the above when executing the computer program.

[0021] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the insurance field large model training data determination method according to any one of the above.

[0022] The application further provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the insurance field large model training data determination method according to any one of the above.

[0023] The present invention provides a method, apparatus, device, and medium for determining training data for a large-scale insurance model. By acquiring at least one insurance-specific text data and at least one general text data, and performing a series of processes such as filtering and clustering on the insurance-specific text data and the general text data, the target training data is obtained. In this way, new domain knowledge can be learned while maintaining general capabilities. Furthermore, the quality of the text data is improved through filtering, and the categories of the text data are obtained through clustering, increasing the diversity of training data categories, alleviating the problem of "memory loss" in large-scale models, and improving the generalization ability of large-scale models. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is one of the flowcharts illustrating the method for determining training data for a large-scale insurance model provided by this invention.

[0026] Figure 2 This is the second schematic diagram of the process for determining training data for a large-scale insurance model provided by this invention.

[0027] Figure 3 This is a schematic diagram of the structure of the device for determining training data for a large-scale model in the insurance field provided by the present invention.

[0028] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] The following is combined Figures 1-2 The present invention describes a method for determining training data for a large insurance model, which can be used to train any model in the insurance field, including large models with question-answering capabilities. The execution subject of this method can be an electronic device, or a method for determining training data for a large insurance model set in the electronic device. The device for determining training data for a large insurance model can be implemented by software, hardware, or a combination of both.

[0031] Figure 1 is one of the flowcharts of the training data determination method of the insurance field large model provided by the present application, as shown in Figure 1 The method comprises the following steps:

[0032] Step 101, obtaining at least one insurance field text data and at least one general text data.

[0033] Among them, the general text data is the text data containing the insurance field text data and the text data of other fields except the insurance field text data.

[0034] Here, the method of obtaining insurance field text data can be obtained from public data sets, or obtained from the company's collected information related to the insurance field, or using the crawler rule strategy to crawl the insurance field related data from the website. Among them, the insurance field text data can include insurance product materials, insurance contract materials, user insurance data, user claim data, insurance company data, insurance market data, etc.

[0035] Here, the general text data can be understood as all the text materials on the Internet, and the general text data can be obtained from the public data sets LIMA, OpenOrca, BELLE or MOSS.

[0036] Step 102, filtering the insurance field text data and the general text data to obtain first training data.

[0037] It should be noted that the training data is actually a training set containing multiple text data. Among them, it contains multiple insurance field text data and multiple general text data.

[0038] Further, the filtering process includes deduplication and cleaning, and the filtering of the insurance field text data and the general text data to obtain the first training data comprises: determining the similarity between any two text data in the comprehensive text data; the comprehensive text data includes all the insurance field text data and all the general text data; based on the similarity, the insurance field text data and the general text data are deduplicated to obtain the text data after deduplication; based on the cleaning rule, the text data after deduplication is cleaned to obtain the first training data; the cleaning rule is used to determine the low-quality text data.

[0039] Here, deduplication refers to removing duplicate or similar text data, and retaining one text data. Cleaning refers to removing low-quality text data.

[0040] Here, any suitable method can be used to determine text similarity, such as similarity hashing (SimHash), text matching models, cosine similarity, etc.

[0041] For example, the specific method for determining text similarity using SimHash is as follows: Text preprocessing: Extracting keywords from the text data; Feature extraction: Extracting key features from the preprocessed text using preset rules, such as word frequency and TF-IDF value; Weighted summation: Weighting each feature and summing all the weighted values ​​to generate a total feature vector; Hash processing: Performing a hash operation on the total feature vector to obtain a fixed-length binary vector, i.e., the SimHash value. Comparing the SimHash values ​​of various texts, the closer the SimHash values ​​are, the more similar the text data is, and similar texts need to be deleted.

[0042] Here, the cleaning rules can be regular expressions or filtering rules, such as text containing many special symbols or text with unclear meaning.

[0043] Specifically, after acquiring insurance-related text data and general text data, the similarity between any two or more text data is calculated, the text data is deduplicated, and then low-quality text data is filtered out according to cleaning rules (such as texts containing many special symbols or texts with unclear meaning) to obtain the first training data.

[0044] In this embodiment of the invention, by deduplicating text data, filtering out similar text data, reducing overtraining, filtering out low-quality text, and improving the quality of text data, the generalization ability of the model is improved, and the problem of "memory loss" in large models is alleviated.

[0045] Step 103: Perform clustering processing on the first training data to obtain the second training data.

[0046] Here, clustering refers to classifying the first training data. The clustering method can be any suitable clustering method, such as K-means, K-center, etc.

[0047] Further, the step of clustering the first training data to obtain the second training data includes: converting each text data in the first training data into vector data; performing regularization and dimensionality reduction on the vector data to obtain processed vector data; randomly selecting K vector data as initial cluster centers from the processed vector data; where K is an integer greater than or equal to 1; calculating the distance between each vector data (excluding the initial cluster centers) and each of the initial cluster centers; performing clustering on the vector data based on the distances to obtain K first clustering results; the first clustering results are used to characterize the text data contained in the initial cluster centers; and determining the second training data based on the K first clustering results and the initial cluster centers.

[0048] Here, the method for transforming vector data can be any suitable method, such as the Sentence-Bert model, the InferSent model, etc.

[0049] Here, regularization is used to prevent overfitting. By introducing a regularization term, the model is subjected to additional constraints during training, making it smoother or sparser and avoiding learning noise and details from the training data. Regularization typically includes types such as L1 regularization and L2 regularization, which penalize the model's parameters in different ways.

[0050] Here, dimensionality reduction aims to reduce the number of data features, transforming high-dimensional data into a low-dimensional space while preserving as much important information as possible from the original data. The dimensionality reduction method can be any suitable approach, such as principal component analysis, linear discriminant analysis, or singular value decomposition.

[0051] It should be noted that after obtaining K first clustering results, the distance between the text data in each first clustering result and the corresponding initial cluster center is determined. If the maximum distance is less than or equal to the distance threshold, it indicates that the first clustering result is accurate; otherwise, the average value of the text data in each first clustering result is calculated, and the text data closest to the average value is taken as the second cluster center. The text data contained in each second cluster center is re-determined until the maximum distance between the cluster center in the clustering result and the text data is less than or equal to the distance threshold, thus obtaining the clustering result.

[0052] In this invention, regularization of the first training data is used to avoid overfitting of the model, dimensionality reduction is used to reduce the complexity of the data, and clustering is used to ensure the balance of various scenarios and features, thereby improving the accuracy of the large model.

[0053] Further, determining the second training data based on the K first clustering results and the initial cluster centers includes: determining the cluster category corresponding to each text data based on the K first clustering results and the initial cluster centers; determining the classification category corresponding to each text data in the first training data using a classification model; verifying the cluster category based on the classification category to obtain a target clustering result; and determining the second training data based on the target clustering result.

[0054] Here, the classification model can be any suitable classification model, such as Qwen1.5 or Generative Pre-trained Transformer 4 (GPT4), etc.

[0055] It should be noted that the verification essentially determines whether the cluster category is the same as the classification category. If the cluster category is different from the classification category, the text data corresponding to that cluster category is removed.

[0056] In this embodiment of the invention, the accuracy of text data classification is improved by verifying the cluster categories, and the accuracy of the model can be improved by using the text data to train the model.

[0057] Furthermore, determining the second training data based on the target clustering results includes:

[0058] Determine the amount of text data contained in each cluster category of the target clustering result;

[0059] If the amount of text data exceeds a threshold, the text data in the cluster category is downsampled to obtain the second training data.

[0060] Here, the text data volume threshold can be any suitable value. When the amount of text data corresponding to a certain cluster category is too large, it will lead to an imbalance of text data across different clusters.

[0061] Here, the downsampling method can be any method, such as random undersampling, easy ensemble, balance cascade, etc.

[0062] In this embodiment of the invention, downsampling is performed on categories with large amounts of text data to ensure the balance and diversity of text data.

[0063] Step 104: Based on the second training data, determine the target training data; the target training data is used to train a large model in the insurance field.

[0064] Furthermore, determining the target training data based on the second training data includes: generating the user's insurance application data based on the generative model and the first mapping relationship; the first mapping relationship is used to characterize the relationship between the user's age, health status, and the user's insurance application data; generating the user's claims data based on the generative model and the second mapping relationship; the second mapping relationship is used to characterize the relationship between the user's insurance policies, the user's medical condition, and the user's claims data; and determining the target training data based on the second training data, the insurance application data, and the claims data.

[0065] Here, the generative model can be any suitable model, such as the GPT4 model.

[0066] Here, insurance data refers to insurance recommendations based on the user's age and health status. For example, if the user is 58 years old and has no critical illnesses, medical insurance would be recommended.

[0067] The first mapping relationship could be the user's age and health status, and the recommended insurance strategy. For example, what kind of insurance should be recommended based on the user's age and health status.

[0068] Here, the claims data includes how to process claims based on the user's insurance policy and medical condition. For example, how to process claims for a user who has purchased medical insurance.

[0069] Specifically, the data includes the age and health status of some users in the original data, as well as the recommended insurance strategies. This data is then fed into GPT4 to generate insurance data corresponding to different ages and health statuses. Additionally, the data includes the user's insurance coverage, medical condition, and final claims strategy. This data is then fed into GPT4 to generate claims data corresponding to different insurance coverage and medical conditions.

[0070] In this embodiment of the invention, additional insurance data and claims data are obtained through the first mapping relationship, the second mapping relationship, and the generative model, thereby expanding the training data, increasing the diversity of the training data, and improving the generalization ability and accuracy of the model.

[0071] Furthermore, determining the target training data based on the second training data, the insurance application data, and the claims data includes: fusing the second training data, the insurance application data, and the claims data to obtain third training data; adjusting the proportion of general text data and insurance-related text data in the third training data to obtain fourth training data; and adjusting the proportion of Chinese and English in the general text data and the proportion of Chinese and English in the insurance-related text data in the fourth training data to obtain the target training data.

[0072] Here, the fusion process can be to directly add the insurance data and claims data to the second training data; or it can be to fuse the three in a weighted manner. This invention does not limit the fusion process.

[0073] Here, the debugging method can be to continuously change the proportion of general text data and insurance domain text data to train the model, judge the performance of each model, and determine the proportion of text data corresponding to the best-performing model as the fourth training data.

[0074] Here, the ratio of general text data to insurance-related text data can be any suitable ratio, such as 1:1, 4:1, etc.

[0075] Here, the ratio of Chinese to English can be any suitable ratio, such as 1:1, 3:1, etc.

[0076] To ensure that the trained model can recognize both Chinese and English, English text data was added to the training data. To improve the model's performance, the ratio of Chinese to English text needs to be continuously adjusted.

[0077] Specifically, the model is trained by continuously adjusting the ratio of Chinese to English text, the performance of multiple models is compared, and the text data with the highest ratio of Chinese to English text corresponding to the best-performing model is determined as the target training data.

[0078] In this embodiment of the invention, by adjusting the proportion of general data and insurance-related text data in the training data, as well as the proportion of Chinese and English training data, the model can learn new domain knowledge while maintaining general capabilities. At the same time, the model can recognize Chinese and English, thereby improving the model's functionality and the user experience for different users.

[0079] The following is an application scenario of the method for determining training data for a large model in the insurance field provided by this invention.

[0080] Figure 2 This is the second schematic diagram of the process for determining training data for a large-scale insurance model provided by this invention, as shown below. Figure 2 As shown, the method includes the following:

[0081] Step S201: Collect general text data and insurance-related text data.

[0082] Step S202: Based on the similarity between text data, deduplicate the general text data and the insurance field text data to obtain the deduplicated text data.

[0083] Step S203: Filter out low-quality text from the deduplicated text data to obtain the first training data.

[0084] Step S204: Perform regularization and dimensionality reduction on the first training data to obtain the processed vector data.

[0085] Step S205: Use the KCenter algorithm to cluster the data and divide it into N classes.

[0086] Step S206: Use a classification model to perform classification verification on the clustered text data and filter out text data that does not conform to the classification.

[0087] Step S207: Downsample the excessive text data in the cluster categories to obtain the second training data.

[0088] Step S208: Add the insurance data and claims data to the second training data to obtain the third training data.

[0089] Step S209: Adjust the proportions of general text data and insurance-related text data in the third training data, the proportion of Chinese and English in the general text data, and the proportion of Chinese and English in the insurance-related text data to obtain the target training data.

[0090] Here, through a large number of training attempts, we finally used general data and insurance domain data in a 1:1 ratio for training in the second pre-training stage, with a total training data of 20G.

[0091] During the instruction fine-tuning phase, the large model was trained using 80,000 text samples to acquire question-and-answer capabilities, enabling it to communicate directly with users. The ratio of general data to insurance-related data was 4:1. Within the general data, the ratio of Chinese to English was 1:1, while the ratio of Chinese to English in the insurance-related text data was 3:1.

[0092] The following describes the apparatus for determining training data for a large insurance model provided by the present invention. The apparatus for determining training data for a large insurance model described below can be referred to in correspondence with the method for determining training data for a large insurance model described above.

[0093] Figure 3 This is a schematic diagram of the structure of the training data determination device for a large-scale insurance model provided by the present invention, as shown below. Figure 3 As shown, the training data determination device 300 for large-scale insurance models includes:

[0094] The acquisition module 301 is used to acquire at least one insurance-related text data and at least one general text data; the general text data includes the insurance-related text data and text data from other fields besides the insurance-related text data.

[0095] Filtering module 302 is used to filter the insurance-related text data and the general text data to obtain the first training data;

[0096] Clustering module 303 is used to perform clustering processing on the first training data to obtain the second training data;

[0097] The determination module 304 is used to determine target training data based on the second training data; the target training data is used to train a large model in the insurance field.

[0098] In some embodiments, the filtering process includes deduplication and cleaning. The filtering module 302 is specifically used to: determine the similarity between any two text data in the comprehensive text data; the comprehensive text data includes all the insurance-related text data and all the general text data; based on the similarity, deduplicatize the insurance-related text data and the general text data to obtain deduplicated text data; based on cleaning rules, clean the deduplicated text data to obtain the first training data; the cleaning rules are used to identify low-quality text data.

[0099] In some embodiments, the clustering module 303 is specifically configured to: convert each text data in the first training data into vector data; perform regularization and dimensionality reduction processing on the vector data respectively to obtain processed vector data; randomly select K vector data as initial cluster centers from the processed vector data; where K is an integer greater than or equal to 1; calculate the distance between each vector data excluding the initial cluster centers and each of the initial cluster centers; perform clustering processing on the vector data based on the distance to obtain K first clustering results; the first clustering results are used to characterize the text data contained in the initial cluster centers; and determine the second training data based on the K first clustering results and the initial cluster centers.

[0100] In some embodiments, the clustering module 303 is further configured to: determine the clustering category corresponding to each of the text data based on the K first clustering results and the initial cluster centers; determine the classification category corresponding to each text data in the first training data using a classification model; verify the clustering category based on the classification category to obtain a target clustering result; and determine the second training data based on the target clustering result.

[0101] In some embodiments, the clustering module 303 is further configured to: determine the amount of text data contained in each cluster category in the target clustering result; and, if the amount of text data is greater than a text data threshold, downsample the text data in the cluster category to obtain the second training data.

[0102] In some embodiments, the determining module 304 is specifically configured to: generate user insurance data based on a generative model and a first mapping relationship; the first mapping relationship is used to characterize the relationship between the user's age, health status, and the user's insurance data; generate user claims data based on the generative model and a second mapping relationship; the second mapping relationship is used to characterize the relationship between the user's insurance policies, the user's medical condition, and the claims data; and determine the target training data based on the second training data, the insurance data, and the claims data.

[0103] In some embodiments, the determining module 304 is further specifically configured to: fuse the second training data, the insurance data, and the claims data to obtain third training data; adjust the proportion of general text data and insurance-related text data in the third training data to obtain fourth training data; and adjust the proportion of Chinese and English in the general text data and the proportion of Chinese and English in the insurance-related text data in the fourth training data to obtain the target training data.

[0104] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a method for determining training data for a large-scale insurance model. This method includes: acquiring at least one insurance-domain text data and at least one general text data; the general text data includes the insurance-domain text data and text data from other domains besides the insurance-domain text data; filtering the insurance-domain text data and the general text data to obtain first training data; clustering the first training data to obtain second training data; and determining target training data based on the second training data; the target training data is used to train a large-scale insurance model.

[0105] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the training data determination method for a large insurance domain model provided by the above methods. The method includes: acquiring at least one insurance domain text data and at least one general text data; the general text data includes the insurance domain text data and text data from other domains besides the insurance domain text data; filtering the insurance domain text data and the general text data to obtain first training data; clustering the first training data to obtain second training data; and determining target training data based on the second training data; the target training data is used to train a large insurance domain model.

[0107] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for determining training data for a large insurance domain model provided by the methods described above. This method includes: acquiring at least one insurance domain text data and at least one general text data; the general text data comprising the insurance domain text data and text data from other domains besides the insurance domain text data; filtering the insurance domain text data and the general text data to obtain first training data; clustering the first training data to obtain second training data; and determining target training data based on the second training data; the target training data being used to train a large insurance domain model.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining training data for a large-scale model in the insurance field, characterized in that, include: Acquire at least one insurance-related text data and at least one general text data; the general text data includes the insurance-related text data and text data from other fields besides the insurance-related text data. The insurance-related text data and the general text data are filtered to obtain the first training data; Convert each text data in the first training data into vector data; The vector data is then subjected to regularization and dimensionality reduction processing to obtain the processed vector data. K vector data points are randomly selected from the processed vector data as initial cluster centers; where K is an integer greater than or equal to 1. Calculate the distance between each vector data point other than the initial cluster centers and each of the initial cluster centers; Based on the distance, the vector data is clustered to obtain K first clustering results; the first clustering results are used to characterize the text data contained in the initial cluster centers; Determine the distance between the text data in each first clustering result and the corresponding initial cluster center. When the maximum distance is greater than the distance threshold, calculate the average value of the text data in each first clustering result. Take the text data closest to the average value as the second cluster center. Redetermine the text data contained in each second cluster center until the maximum distance between the cluster center and the text data in each clustering result is less than or equal to the distance threshold, and obtain K first clustering results. Based on the K first clustering results and the initial cluster centers, determine the clustering category corresponding to each text data; The classification model is used to determine the classification category corresponding to each text data in the first training data; Determine whether the clustering category is the same as the classification category to obtain the target clustering result; Based on the target clustering results, text data whose clustering category is different from the classification category is deleted to obtain the second training data; The generative model generates insurance data corresponding to different ages and monitoring statuses based on the user's age, monitoring status, and recommended insurance strategies included in the original data. The generation model generates claim data corresponding to different insurance policies and different medical conditions based on the user's insurance information, medical condition, and final claim settlement strategy; Based on the second training data, the insurance data, and the claims data, target training data is determined, which is used to train a large model in the insurance field.

2. The method for determining training data for a large-scale insurance model according to claim 1, characterized in that, The filtering process includes deduplication and cleaning. The filtering process on the insurance-related text data and the general text data to obtain the first training data includes: Determine the similarity between any two pieces of text data in the comprehensive text data; the comprehensive text data includes all the insurance-related text data and all the general text data. Based on the similarity, the insurance-related text data and the general text data are deduplicated to obtain the deduplicated text data. Based on the cleaning rules, the deduplicated text data is cleaned to obtain the first training data; the cleaning rules are used to identify low-quality text data.

3. The method for determining training data for a large-scale insurance model according to claim 1, characterized in that, The method further includes: Determine the amount of text data contained in each cluster category of the target clustering result; If the amount of text data exceeds a threshold, the text data in the cluster category is downsampled to obtain the second training data.

4. The method for determining training data for a large-scale insurance model according to claim 1, characterized in that, The step of determining the target training data based on the second training data, the insurance data, and the claims data includes: The second training data, the insurance data, and the claims data are merged to obtain the third training data. Adjusting the proportion of general text data and insurance-related text data in the third training data yields the fourth training data. The target training data is obtained by adjusting the proportion of Chinese and English in the general text data and the proportion of Chinese and English in the insurance field text data in the fourth training data.

5. A device for determining training data for a large-scale model in the insurance field, characterized in that, include: The acquisition module is used to acquire at least one insurance-related text data and at least one general text data; the general text data includes the insurance-related text data and text data from other fields besides the insurance-related text data. A filtering module is used to filter the insurance-related text data and the general text data to obtain the first training data; The clustering module is used to convert each text data in the first training data into vector data; And for performing regularization and dimensionality reduction processing on the vector data respectively, to obtain the processed vector data; And for randomly selecting K vector data from the processed vector data as initial cluster centers; where K is an integer greater than or equal to 1; And the distance between each vector data other than the initial cluster centers and each of the initial cluster centers; The method includes: clustering the vector data based on the distance to obtain K first clustering results; the first clustering results are used to characterize the text data contained in the initial cluster centers; and determining the distance between the text data in each first clustering result and the corresponding initial cluster center. When the maximum distance is greater than a distance threshold, the method calculates the average value of the text data in each first clustering result, takes the text data closest to the average value as the second cluster center, and redetermines the text data contained in each second cluster center until the maximum distance between the cluster center and the text data in each clustering result is less than or equal to the distance threshold, thus obtaining K first clustering results. And for determining the cluster category corresponding to each of the text data based on the K first clustering results and the initial cluster centers; And for using a classification model to determine the classification category corresponding to each text data in the first training data; And a method for determining whether the clustering category is the same as the classification category, to obtain the target clustering result; And based on the target clustering result, delete text data whose clustering category is different from the classification category to obtain second training data; The determination module is used to generate insurance data corresponding to different ages and monitoring statuses based on the user's age, monitoring status, and recommended insurance strategies included in the original data; and to generate claim data corresponding to different insurance and different medical conditions based on the user's insurance situation, medical condition, and final claim settlement strategy. And for determining target training data based on the second training data, the insurance data, and the claims data, the target training data being used to train a large model in the insurance field.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for determining training data for a large insurance model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for determining training data for a large insurance model as described in any one of claims 1 to 4.

Images