Model data processing method and device, equipment and medium

Abstract

The invention discloses a model data processing method and device, equipment and a medium, and the method comprises the steps: extracting related features according to the historical insurance policy data of an insurance customer, and constructing a data set; creating a process pool of multiple processes, performing data segmentation on the data set to obtain a plurality of segmented sub-sets, and allocating processes for each segmented sub-set; enabling each process to convert the numeric type and category type features in the allocated cutting sub-set into a format which can be identified by a model; and combining all process processing results to generate a new data set for model training. In the technical scheme of the invention, the related features are extracted through the historical insurance policy data of the insurance customer, the data set is constructed, the data types in the segmented data set are converted by using multiple processes, and finally the data are merged into the new data set which can be used for model training, so that the data processing time is shortened through multi-process parallel processing, and the data processing efficiency is improved. And the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model data processing method, device, equipment and medium. Background Art

[0002] In the insurance industry, it is usually possible to predict whether a user has the possibility of renewal or reinstatement by analyzing the user's behavior data and related features. The analysis process involves basic user information, policy information, and user behavior data. When training the model, the data set is generally divided into numerical features and categorical features, and the type of each feature in the model must be a readable specific type. This requires that when the library is used to read the data set into memory, each feature needs to be converted into a specific data type. The amount of data that needs to be processed is large, and the data types are diverse. This data processing process will consume a lot of computing time. Summary of the invention

[0003] Embodiments of the present invention provide a model data processing method, apparatus, device and medium, which aim to solve the problem that the model for predicting whether a user has the possibility of renewal or reinstatement of insurance under the prior art needs to convert each feature into a specific data type during training, resulting in excessive calculation time.

[0004] In a first aspect, an embodiment of the present invention provides a model data processing method, comprising:

[0005] Extract relevant features and construct a data set based on the insurance customer's historical policy data; create a multi-process process pool, split the data set into multiple split sub-sets, and assign a process to each of the split sub-sets; enable each process to convert the numerical and categorical features in the assigned split sub-set into a format recognizable by the model; merge the results of all process processing to generate a new data set for model training.

[0006] In a second aspect, an embodiment of the present invention further provides a model data processing device, comprising the units of the model data processing method as described above.

[0007] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to execute the steps of the above-mentioned model data processing method.

[0008] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the steps of the above-mentioned model data processing method can be implemented.

[0009] Compared with the prior art, the present invention has the following beneficial effects:

[0010] In the technical solution of the present invention, relevant features are extracted from the historical policy data of insurance customers and a data set is constructed. The data types in the segmented data sets are converted using multiple processes, and finally merged into a new data set that can be used for model training. The data processing time is shortened through multi-process parallel processing, thereby improving the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0012] Figure 1 A flow chart of the model data processing method provided by the present invention;

[0013] Figure 2 A first sub-flow chart of the model data processing method provided by the present invention;

[0014] Figure 3 A second sub-flow chart of the model data processing method provided by the present invention;

[0015] Figure 4 A third sub-flow chart of the model data processing method provided by the present invention;

[0016] Figure 5 A fourth sub-flow chart of the model data processing method provided by the present invention;

[0017] Figure 6 A schematic block diagram of units of the model data processing device provided by the present invention;

[0018] Figure 7 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0021] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0022] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0023] In order to solve the problem that the model for predicting whether a user has the possibility of renewal or reinstatement in the prior art needs to convert each feature into a specific data type during training, resulting in a long calculation time, the present invention provides a model data processing method, referring to Figures 1 to 5 , which includes:

[0024] S100, extract relevant features based on the historical policy data of insurance customers and construct a data set;

[0025] The so-called historical policy data refers to the information of all insurance policies purchased by customers in the past period of time, including the effective and expiration dates of the policies, premium information, basic customer information, etc. The so-called data set is a collection of data information used for subsequent data analysis and model training. When using LightGBM to build a model to predict whether a user has the possibility of renewal or reinstatement of insurance, it is first necessary to obtain supporting data that can reflect relevant business issues for subsequent model construction and training. For the possible target problem of predicting whether a user has renewal or reinstatement of insurance, by analyzing the customer's historical policy data, such as policy renewal, claim records, etc., it is possible to identify which customers may cancel their policies or not renew them in the future, thereby predicting the risk of customer churn and taking measures in advance. For expired policies, historical policy data can also be used to predict which customers are likely to reinstate their policies. Completing model construction through targeted data information helps improve the accuracy of model prediction efficiency and customer retention rate.

[0026] In one embodiment, referring to Figure 2 The steps of extracting relevant features and constructing a data set based on the historical policy data of insurance customers include:

[0027] S110, collecting historical policy data of insurance customers, performing feature engineering on the historical policy data, and extracting relevant features;

[0028] S120, setting the reinstatement status in the extracted relevant features as a label;

[0029] S130: construct a data set based on the relevant features and the labels.

[0030] The so-called relevant features refer to the data information related to whether the insurance customer's historical policies have been renewed or reinstated, and the sources of these data include but are not limited to the insurance company's internal database, customer management system or external data provider. The so-called label refers to a binary variable. In the embodiment of the present invention, the policy reinstatement situation is the label, and the label can be used to classify or predict the customer's historical policy data when constructing a data set. First, collect the insurance customer's historical policy data, such as customer ID, policy number, policy effective and expiration date, premium amount, claim record, age, gender, region, etc. Then, the collected data is cleaned for preprocessing, missing values, duplicate data are deleted, and the format is standardized. Then, according to business logic and data analysis, meaningful relevant features are extracted. For the embodiment of the present invention, the so-called meaningful relevant features are as follows: customer features include age, gender, region, occupation, etc., policy features include policy type, insured amount, insurance period, payment method, etc., and historical behavior features include the number of past claims, claim amount, renewal history, insurance history, etc., which are all of great reference significance for analyzing whether the customer has reinstated the insurance. After that, the reinstatement status is used as a label for subsequent modeling. If the customer re-insures after a certain time point, it is marked as 1 (reinstatement), and if the customer does not re-insure within a certain period of time, it is marked as 0 (not reinstated). The above marking results are used as labels and appended to the feature set to form the final label column. Finally, the extracted relevant features are merged with the generated labels to construct a complete data set. Make sure that each sample contains its features and the corresponding reinstatement label.

[0031] In one embodiment, referring to Figure 2 , after the step of constructing a data set based on the relevant features and the labels, the step further includes:

[0032] S140, converting the data set into a table structure data set recognizable by a model;

[0033] S150, storing the table structure data set into a memory for easy retrieval.

[0034] The so-called tabular data set refers to a data structure organized and stored in a tabular form, usually in DataFrame format, which arranges different types of features, including numerical features, categorical features, labels, etc., in rows and columns of the table to form a spreadsheet or database table. When using LightGBM to build a model to predict whether a user is likely to renew or reinstate insurance, the data set can be converted into a DataFrame through Pandas or converted into a DataFrame through Spark, and the processed data set, i.e., the tabular data set, is stored in memory to facilitate subsequent model training and prediction. It is preferred that the data set is converted into a DataFrame through Pandas, because Pandas provides a large number of built-in functions and methods, supports data selection, filtering, sorting, grouping, merging and other operations, which can make the data processing process more efficient and flexible. For historical policy data of insurance customers with multiple data types, using Pandas is more efficient. Storing the tabular data set in memory, the data stored in memory can be quickly called by the model, reducing loading time, especially when training and predicting models, because data access in memory is much faster than reading from external storage devices such as hard disks, it is crucial to improve the overall model's operating efficiency.

[0035] S200, creating a multi-process process pool, segmenting the data set to obtain multiple segmentation sub-sets, and assigning a process to each of the segmentation sub-sets;

[0036] The so-called process pool is a set of pre-created and managed processes used to execute multiple tasks in parallel. The so-called slicing subset refers to dividing the entire data set into multiple small subsets, each of which contains a part of the data set. In actual operation, the operation of creating a multi-process process pool can use the Multiprocessing library to create a multi-process process pool so that the data set can be split and assigned to multiple processes for processing. A process is assigned to each slicing subset to achieve parallel processing.

[0037] In one embodiment, referring to Figure 3 The steps of creating a multi-process process pool, segmenting the data set to obtain a plurality of segmentation sub-sets, and assigning a process to each of the segmentation sub-sets include:

[0038] S210, creating a process pool including multiple processes, and setting the number of processes in the process pool according to the number of processor cores of the computer;

[0039] S220, calculating the size of each subset according to the total size of the table structure data set and the number of processes;

[0040] S230, using the calculated size of the subset to divide the data set into the subsets of the number of processes;

[0041] S240: Allocate the segmented subsets to each process.

[0042] In order to ensure the efficiency of data processing, we first need to obtain the number of processor cores of the computer through system calls in order to set the appropriate number of processes for the process pool. Then use the Multiprocessing library to create a process pool, and set the number of processes in the process pool to the number of processor cores to make full use of computing resources. Get the total size of the data set and determine the total number of rows in the table structure data set to calculate the size of each subset. Then divide the total number of rows of the data set by the number of processes to obtain the expected size of each subset, and divide the original data set into multiple subsets based on the calculated size of each subset. It should be noted that the order and integrity of the data should be maintained during segmentation. When there is a situation where it cannot be divided evenly, it is necessary to ensure that all data can be segmented. If the total number of rows cannot be divided evenly by the number of processes, the last subset must contain the remaining data rows. Then, each segmented subset needs to be assigned to a process in the process pool to ensure that each process can independently process its assigned subset, and start the subset processing task in the process pool to monitor the execution status of the process to ensure that all processes can successfully complete their tasks while improving the efficiency of data processing.

[0043] Furthermore, adding an exception handling mechanism in each process can ensure that even if an individual process fails, it will not affect the overall data partitioning process.

[0044] S300, enabling each of the processes to convert the numerical and categorical features in the allocated slice sub-set into a format recognizable by the model;

[0045] In the embodiment of this scheme, the so-called model-recognizable format specifically refers to the type format of features that the LightGBM model can recognize. Features include numerical features and categorical features. For example, age features belong to numerical features, and region features belong to categorical features. Pandas will read categorical features as object types, but the LightGBM model cannot recognize object types, so it is necessary to convert categorical features into category types for LightGBM model recognition.

[0046] In one embodiment, referring to Figure 4 The step of enabling each of the processes to convert the numerical and categorical features in the allocated slice sub-sets into a format recognizable by the model comprises:

[0047] S310, identifying feature types of the subsets to which each process is assigned;

[0048] S320, traversing all features of the assigned subset in each of the processes, and converting numerical features and categorical features into a format recognizable by the model.

[0049] In actual operation, in each process, the feature type of the assigned subset is first identified. Identify which features are numerical, such as integers or floating-point numbers, and identify which features are categorical, such as character or categorical data. In the subsequently generated model, numerical features are used for regression analysis and numerical calculations, and categorical features are used for classification analysis. Then a mapping is established for each feature so that the subsequent conversion steps can process the features according to the mapping. After that, in each process, all features in the assigned subset are traversed. For numerical features, the astype method is used to convert them to float type so that the model can recognize them. For categorical features, the categorical features are converted to a category type format that can be recognized by the model through unique hot encoding and label encoding. It should be noted that in the process of processing features, the missing values ​​in the numerical features and categorical features must be processed to ensure that the features are on the same scale.

[0050] Furthermore, for categorical features, target encoding or frequency encoding can be used to further improve the model effect.

[0051] S400: Merge the results of all processes to generate a new data set for model training.

[0052] The so-called new data set is the data set used to load the LightGBM model for training. When all processes are completed, the output results of each process are finally merged to obtain a new data set, which can be used to train the LightGBM model to predict whether the user has the possibility of renewal or reinstatement of insurance.

[0053] In one embodiment, referring to step 5, the step of merging the results of all processes to generate a new data set for model training includes:

[0054] S410, merging the processed numerical features and categorical features into a new data set;

[0055] S420: Divide the new data set into a training set and a test set, and input the training set and the test set into the model for training.

[0056] When all processes have completed the feature conversion of their respective assigned subsets, the processing results of each process are collected, which include the converted numerical features and categorical features. The collected numerical features and categorical features processed by all processes are merged into a new data set. When merging, the number of rows of each feature must be consistent to maintain the integrity and accuracy of the data. The merged data set should include all feature columns to form a unified data structure. The merged new data set is then divided into a training set and a test set. Random sampling can be used for division to ensure that the samples of the training set and the test set are representative and to reduce sample selection bias. During the division process, stratified sampling can also be used to ensure that the distribution of category labels in the training set and the test set is consistent. Furthermore, the generated training set and test set are saved in a format such as CSV or Parquet for use in subsequent model training.

[0057] Furthermore, during model training, hyperparameter search, such as grid search or Bayesian optimization, can be used to optimize the LightGBM model to further improve model performance.

[0058] Figure 6 is a schematic block diagram of a model data processing device provided by an embodiment of the present invention. Figure 6 As shown, corresponding to the above model data processing method, the present invention also provides a model data processing device. The model data processing device includes a unit for executing the above model data processing method, and the device can be configured in a desktop computer, a tablet computer, a smart phone, and other terminals. Specifically, please refer to Figure 6 , the model data processing device comprises:

[0059] The data set construction unit 601 is used to extract relevant features and construct a data set based on the historical policy data of insurance customers; the data set segmentation unit 602 is used to create a multi-process process pool, segment the data set, and assign processes to the segmented sub-sets; the data type conversion unit 603 is used to enable each process to convert the numerical and categorical features in the assigned segmented sub-sets into a format recognizable by the model; the data result merging unit 604 is used to merge the results of all process processing to generate a new data set for model training.

[0060] The data set construction unit 601 includes: a historical policy data processing unit, which is used to collect the historical policy data of insurance customers, perform feature engineering on the historical policy data, and extract relevant features; a label setting unit, which is used to set the reinstatement status in the extracted relevant features as a label; a data set combination unit, which is used to set the reinstatement status in the extracted relevant features as a label; a data set tabularization unit, which is used to convert the data set into a table structure data set that can be recognized by the model; and a tabular data set storage unit, which is used to store the table structure data set into the memory for easy retrieval.

[0061] The data set segmentation unit 602 includes: a process pool creation unit, which is used to create a process pool including multiple processes and set the number of processes in the process pool according to the number of processor cores of the computer; a subset segmentation pre-calculation unit, which is used to calculate the size of each subset according to the total size of the table structure data set and the number of processes; a subset segmentation unit, which is used to divide the data set into subsets of the number of processes using the calculated size of the subset; and a subset allocation unit, which allocates the segmented subsets to each process.

[0062] The data type conversion unit 603 includes: a subset feature identification unit, used to identify the feature type of the subset assigned to each process; a subset feature type conversion unit, used to traverse all features of the assigned subset in each process, and convert numerical features and categorical features into a format recognizable by the model.

[0063] The data result merging unit 604 includes: a new data set combining unit, used to merge the processed numerical features and the categorical features into a new data set; a new data set dividing unit, used to divide the new data set into a training set and a test set, and input the training set and the test set into the model for training.

[0064] The above-mentioned model data processing device can be implemented in the form of a computer program. The computer program can be used in Figure 7 Runs on the computer device shown.

[0065] See also Figure 7 , Figure 7 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a terminal or a server, wherein the terminal may be an electronic device with communication functions such as a desktop computer, a tablet computer, a smart phone, etc. The server may be an independent server or a server cluster composed of multiple servers.

[0066] See also Figure 7The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0067] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a model data processing method.

[0068] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0069] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a model data processing method.

[0070] The network interface 505 is used to communicate with other devices over the network. Those skilled in the art will appreciate that Figure 7 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0071] The processor 502 is used to run a computer program 5032 stored in the memory to implement the steps of the above method.

[0072] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0073] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0074] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the steps of the above method.

[0075] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0076] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0077] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0078] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0080] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A model data processing method, characterized in that: include: Extract relevant features and construct a data set based on the historical policy data of insurance customers; Creating a multi-process process pool, segmenting the data set into multiple segmentation sub-sets, and assigning a process to each of the segmentation sub-sets; Enable each of the processes to convert the numerical and categorical features in the assigned slice sub-set into a format recognizable by the model; Combine the results of all processes to generate a new dataset for model training.

2. The model data processing method according to claim 1, characterized in that: The step of extracting relevant features and constructing a data set based on the historical policy data of insurance customers includes: Collecting historical policy data of insurance customers, performing feature engineering on the historical policy data, and extracting relevant features; Setting the reinstatement status in the extracted relevant features as a label; A data set is constructed according to the relevant features and the labels.

3. The model data processing method according to claim 2, characterized in that: The step of constructing a data set based on the relevant features and the labels further includes: Converting the data set into a table structure data set recognizable by the model; The table structure data set is stored in the memory for easy retrieval.

4. The model data processing method according to claim 3, characterized in that: The steps of creating a multi-process process pool, segmenting the data set to obtain a plurality of segmented sub-sets, and assigning a process to each of the segmented sub-sets include: Creating a process pool including multiple processes, and setting the number of processes in the process pool according to the number of processor cores of the computer; Calculate the size of each subset according to the total size of the table structure data set and the number of processes; Using the calculated size of the subset, the data set is divided into the subsets corresponding to the number of processes; The segmented subsets are allocated to various processes.

5. The model data processing method according to claim 4, characterized in that: The step of enabling each of the processes to convert the numerical and categorical features in the allocated slice sub-sets into a format recognizable by the model comprises: Performing feature type identification on the subset to which each of the processes is assigned; In each of the processes, all features of the assigned subset are traversed to convert numerical features and categorical features into a format recognizable by the model.

6. The model data processing method according to claim 5, characterized in that: The step of combining the results of all processes to generate a new data set for model training includes: Merging the processed numerical features and categorical features into a new data set; The new data set is divided into a training set and a test set, and the training set and the test set are put into the model for training.

7. A model data processing device, characterized in that: A unit comprising the model data processing method according to any one of claims 1 to 6.

8. The model data processing device according to claim 7, characterized in that: include: A data set construction unit, used to extract relevant features and construct a data set based on the historical policy data of insurance customers; A data set segmentation unit is used to create a multi-process process pool, segment the data set, and allocate processes to the segmented sub-sets; A data type conversion unit, used to enable each of the processes to convert the numerical and categorical features in the slice sub-sets allocated to it into a format recognizable by the model; The data result merging unit is used to merge the results of all processes and generate a new data set for model training.

9. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to execute the steps of the method as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 can be implemented.

Images