Data processing method, device and equipment based on data lake and storage medium
By adopting data lake-based data processing methods in insurance companies, synchronizing, preprocessing, marking and aggregating policy data, the redundancy and inefficiency caused by the decentralized storage of policy data is solved, and efficient data utilization and business-driven are achieved.
Patent Information
- Application Number
- CN202411694046.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-04-11
AI Technical Summary
Existing insurance policy data is stored decentralizedly, resulting in data redundancy and duplication, making data quality difficult to ensure, and effectively analyze and mine, resulting in inefficient data use.
Data lake-based data processing method is adopted to synchronize the original policy data of each decentralized database of the enterprise to the original data layer of the target database, and preprocess, mark, reorganize and summarize, and form policy detailed data of different tag label dimensions, and finally form service layer data.
Through the data processing method of the data lake, centralized management and efficient utilization of insurance policy data are achieved, data redundancy and duplication are reduced, data quality and usage efficiency are improved, and data driven business development is supported.
Smart Images
Figure CN119807309A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of financial technology and data processing technology, and in particular to a data processing method, device, equipment and storage medium based on a data lake. Background Art
[0002] With the development of insurance business, the amount of policy data of major insurance companies has grown rapidly. Such a large amount of policy data is often scattered in different relational databases. The policy data has no direct correlation, resulting in a large amount of data redundancy and duplication, and the data quality cannot be guaranteed. In addition, insurance companies were business-driven in the early stage, but are now gradually turning to data-driven. Therefore, it is particularly important to make good use of the company's existing policy data, conduct data analysis and mining through the existing policy data, and thus drive business development.
[0003] However, due to the current decentralized storage and management of various insurance policy data, the difficulty of data analysis is that there is no data to be analyzed. The data is too scattered, the amount of data is very large, there is no relationship, no characteristics or the characteristics are unclear, and it is difficult to apply, resulting in relatively low data utilization efficiency. Summary of the invention
[0004] The embodiments of the present application provide a data processing method, apparatus, device and storage medium based on a data lake to solve the technical problem that existing insurance policy data is difficult to reuse, resulting in relatively low data utilization efficiency.
[0005] A data processing method based on a data lake, comprising: Synchronize the original policy data of each scattered database of the enterprise to the original data layer of the target database; Based on the preset policy preprocessing rules, the original policy data of the original data layer is extracted for preprocessing to obtain target policy data; According to the data source business module and data characteristics, the target policy data in the target database is marked accordingly; According to the mark labels of the target policy data, the extracted target policy data are reorganized and spliced to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; The insurance policy detail data corresponding to different marking label dimensions formed in the data detail layer are aggregated to form service layer data in the data service layer of the target database.
[0006] A data processing system based on a data lake, the data processing system comprising a data processing device, a target database, a plurality of distributed databases of an enterprise, and at least one data demander, wherein: The data processing device synchronizes the original policy data of each dispersed database of the enterprise to the original data layer of the target database; The data processing device extracts the original policy data of the original data layer for preprocessing based on the preset policy preprocessing rules to obtain target policy data; The data processing device tags the target policy data in the target database accordingly according to the data source business module and data characteristics; The data processing device reorganizes and splices the extracted target policy data according to the mark labels of the target policy data, so as to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; The data processing device summarizes the policy detail data corresponding to the different mark label dimensions formed by the data detail layer to form service layer data in the data service layer of the target database, so that the data demander can extract the required target data from a certain level of the target database.
[0007] A data processing device based on a data lake, comprising: The synchronization module is used to synchronize the original policy data of each decentralized database of the enterprise to the original data layer of the target database; A preprocessing module, for extracting the original policy data of the original data layer for preprocessing based on a preset policy preprocessing rule to obtain target policy data; A marking module, used to mark the target policy data in the target database accordingly according to the data source business module and data characteristics; A splicing module, used for reorganizing and splicing the extracted target policy data according to the mark labels of the target policy data, so as to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; The aggregation module is used to aggregate the insurance policy detail data corresponding to different marking label dimensions formed in the data detail layer to form service layer data in the data service layer of the target database.
[0008] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method based on a data lake as described in any one of the preceding items when executing the computer program.
[0009] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the data processing method based on a data lake as described in any of the above items.
[0010] It can be seen that in this embodiment, a data processing method based on a data lake is provided, which synchronizes the original policy data of each scattered database of an enterprise to the original data layer of a target database; based on a preset policy preprocessing rule, the original policy data of the original data layer is extracted for preprocessing to obtain target policy data; according to the data source business module and the data characteristics, the target policy data in the target database is marked with a label accordingly; according to the mark label of the target policy data, the extracted target policy data is reorganized and spliced to form policy detail data corresponding to different mark label dimensions in the data detail layer of the target database; and the different mark labels formed in the data detail layer are then The policy detail data corresponding to the signature dimension is summarized to form service layer data in the data service layer of the target database, thereby forming the final policy data lake. It can be seen that different levels of data will be formed in the data lake, and all levels of data in the same data lake will be interoperable and no longer isolated. Data can be quickly connected and can be used directly. The data in the lake has labels and corresponding rules and specifications, so the corresponding data content can be found quickly. The location of the data you want and the format of the data can be quickly found in the list, which can facilitate the reuse of the data in the data lake, improve the efficiency of using policy data, and help use data to promote business development. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0012] Figure 1 This is a schematic diagram of an application environment of a data processing method based on a data lake in one embodiment of the present application; Figure 2 This is a flow chart of a data processing method based on a data lake in one embodiment of the present application; Figure 3 is another flow chart of a data processing method based on a data lake in one embodiment of the present application; Figure 4 It is a structural schematic diagram of a data processing device based on a data lake in one embodiment of the present application; Figure 5 It is a structural diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION
[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0014] like Figure 1 As shown, Figure 1 : is a processing framework diagram of a data processing method based on a data lake in an embodiment of the present application. In this method, the original policy data of each decentralized database of the enterprise is first synchronized to the original data layer of the target database; based on the preset policy preprocessing rules, the original policy data of the original data layer is extracted for preprocessing to obtain the target policy data; wherein the original policy data includes but is not limited to the following: Figure 1 The underwriting data, preservation data, claims data, renewal data, financial data, agent data and other data shown are then marked and labeled for the target policy data in the target database according to the data source business module and data characteristics; the extracted target policy data are reorganized and spliced according to the mark labels of the target policy data to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; the policy detail data corresponding to different mark label dimensions formed at the data detail layer are summarized to form service layer data at the data service layer of the target database, thereby forming a data lake. Based on the data lake, various demand parties can obtain data on demand, including but not limited to the following needs: Figure 1 The financial settlement, report issuance, machine learning, and empirical statistics requirements shown in the figure can be used by the demander to implement corresponding business based on these data to drive the business.
[0015] It should be noted that the above method may be implemented by a server, and the server may be implemented by an independent server or a server cluster composed of multiple servers, without specific limitation.
[0016] In one embodiment, if Figure 2 As shown, a data processing method based on a data lake is provided, and the method is applied to a server as an example for explanation, including the following steps: S10: Synchronize the original policy data of each decentralized database of the enterprise to the original data layer of the target database; Generally speaking, with the development of insurance business, the amount of insurance policy data of major insurance companies has grown rapidly. Such a large amount of insurance policy data is also often scattered in different relational databases, and the insurance policy data has no direct correlation. In this embodiment, the server will read the original insurance policy data of each scattered database of the enterprise, and use data synchronization data to synchronize the original insurance policy data of each scattered database of the enterprise to the original data layer of the target database. Each scattered database of the enterprise stores the corresponding original insurance policy data, which is the insurance policy data generated based on the insurance transaction, and can include insurance policy data of various dimensions, which are not specifically limited.
[0017] In one embodiment, step S10, that is, synchronizing the original policy data of each decentralized database of the enterprise to the original data layer of the target database, includes: synchronizing the original policy data of each decentralized database of the enterprise to the original data layer of the target database in real-time synchronization or T-1 simultaneous form, wherein T is a synchronization time unit.
[0018] The original data layer of the target database is designated to store the original policy data that has just been synchronized. The data stored in this original data layer is basic data, which is the data in the target database that is closest to the original policy data or the same as the original policy data.
[0019] In one embodiment, the original policy data includes underwriting data, preservation data, claims data, renewal data, financial data, and insurance agent data generated by insurance transactions. Among them, underwriting generally refers to insurance underwriting, which means that during the underwriting process of the corresponding insurance contract, the underwriter will give different underwriting conditions according to the different risk categories of the subject matter, and the underwriting data refers to the data generated during the underwriting process. Insurance preservation data refers to the data generated in the process of maintaining the validity of the insurance contract, including the process and result information of changing, supplementing or amending the content of the insurance contract. Claims data refers to various information collected and recorded when handling claims cases of insurance contracts, including but not limited to the number, amount, processing time, and payment status of claims cases. These data reflect the efficiency, service quality, and customer satisfaction of the insurance company in the claims process. Renewal data refers to the data in the insurance contract that the customer continues to pay the insurance premium after the end of the insurance period. The renewal data may include the amount and proportion of the premium that the customer continues to pay after the expiration of the insurance contract, reflecting the customer's continued willingness to purchase insurance products and satisfaction. Insurance agent data refers to the relevant data of the agent of the insurance contract, including insurance agent personal information, insurance agent work experience, transaction volume and other agent data.
[0020] It can be seen that in this embodiment, the original policy data that is synchronized and processed as described above in the embodiment of the present application covers underwriting data, preservation data, claims data, renewal data, financial data, insurance agent data and other data, so that policy data of various dimensions with subsequent reuse value form a corresponding data lake, which covers a wide range of data and improves the applicability and application scenarios of the data lake.
[0021] It should be noted that, in some other embodiments, the original insurance policy data may also include insurance policy data of other dimensions, which are not specifically limited.
[0022] S20: extracting the original policy data of the original data layer for preprocessing based on a preset policy preprocessing rule to obtain target policy data; S30: marking the target policy data in the target database accordingly according to the data source business module and data characteristics; In the steps S20-S30, after synchronizing the original policy data of each decentralized database of the enterprise to the original data layer of the target database, since the original policy data synchronized to the original data layer is still relatively basic data and is far from being usable, in this embodiment, the synchronized original policy data stored in the original data layer will remain unchanged, and based on the preset policy preprocessing rules, the original policy data of the original data layer will be extracted for preprocessing to obtain the target policy data, which is mainly used to normalize the extracted original policy data based on the preset policy preprocessing rules to obtain complete and regular target policy data. The preset policy preprocessing is related to the type of policy data, so the preset policy preprocessing is set according to the type of actual policy data, and is not specifically limited.
[0023] For example, the rules include but are not limited to: invalid age correction and elimination, user information completion, etc. The entire preprocessing process may include order capture, aggregation, cleaning and correction. Order capture is to collect data from different tables through SQL, then splice and aggregate the collected results to form a wide table, and then perform field correction, elimination and completion of the data based on the preprocessing rules and algorithms, and finally form the target policy data.
[0024] It should be understood that the original policy data is policy data generated based on insurance transactions, and may include policy data of various dimensions. Since the previous preprocessing is data normalization processing, the target policy data formed also includes policy data of various dimensions. The policy data of each dimension may have corresponding data source businesses and characteristics. In this embodiment, in order to facilitate the subsequent reuse of the data, the target policy data in the target database are marked with corresponding labels according to the data source business module and the data characteristics.
[0025] Among them, data source business refers to the marking information or related information that characterizes the source of the policy data, and data characteristics characterize the differences in the policy data, including data form type or other characteristics. For example, in this step, the source of the insurance contract and the classification of the contract will be recorded, and then the policy data will be re-marked to add corresponding marking labels. For example, Xiao Ming, who is under 18 years old in Guangdong, purchased an insurance policy, and the policy can be marked with a label such as underage or Pearl River Delta.
[0026] S40: Reorganize and splice the extracted target policy data according to the mark labels of the target policy data to form policy detail data corresponding to different mark label dimensions in the data detail layer of the target database.
[0027] After obtaining the target policy data and the reward mark label, the server will reorganize and splice the extracted target policy data according to the mark label of the target policy data, so as to form policy detail data corresponding to different mark label dimensions in the data detail layer of the target database. That is, the target database also includes a data detail layer, which stores the data after the extracted target policy data is reorganized and spliced according to the mark label of the target policy data, that is, the policy detail data corresponding to different mark label dimensions is formed.
[0028] For example, assuming that a target insurance policy data includes two field labels, namely "region" and "consumption capacity", then all the target insurance policy data with marked labels can be arranged and combined to form data dimensions of synchronized regions and different consumption capacities, and then assembled to obtain the insurance policy details data corresponding to the same marked label dimensions (synchronized regions and different consumption capacities). In this way, it is conducive to the formation of data utilization that can be used in scenarios such as thousands of people with thousands of faces.
[0029] S50: Aggregating the insurance policy detail data corresponding to different marking label dimensions formed in the data detail layer to form service layer data in the data service layer of the target database.
[0030] Furthermore, the target database also includes a data service layer. After the extracted target policy data is reorganized and spliced according to the mark labels of the target policy data to form policy detail data corresponding to different mark label dimensions in the data detail layer of the target database, the server will also extract and summarize the policy detail data corresponding to the different mark label dimensions formed in the data detail layer to form service layer data in the data service layer of the target database. It can be seen that the target database includes a multi-level data structure, including an original data layer, a data detail layer and a data service layer. Each level includes corresponding data of different level types to form a data lake structure, and ultimately forms valid data that can be directly or indirectly utilized in the data service layer, which can be used in subsequent data applications.
[0031] In other words, different levels of data will be formed in the data lake. All levels of data in the same data lake will be interoperable and no longer isolated. Data can be quickly connected and can be used directly. The data in the lake has labels and corresponding rules and specifications, so you can quickly find the corresponding data content, and you can quickly find the location of the data you want and the format of the data in the list.
[0032] It can be seen that in this embodiment, a data processing method based on a data lake is provided, which synchronizes the original policy data of each scattered database of an enterprise to the original data layer of a target database; based on a preset policy preprocessing rule, the original policy data of the original data layer is extracted for preprocessing to obtain target policy data; according to the data source business module and the data characteristics, the target policy data in the target database is marked with a label accordingly; according to the mark label of the target policy data, the extracted target policy data is reorganized and spliced to form policy detail data corresponding to different mark label dimensions in the data detail layer of the target database; and the different mark labels formed in the data detail layer are then The policy detail data corresponding to the signature dimension is summarized to form service layer data in the data service layer of the target database, thereby forming the final policy data lake. It can be seen that different levels of data will be formed in the data lake, and all levels of data in the same data lake will be interoperable and no longer isolated. Data can be quickly connected and can be used directly. The data in the lake has labels and corresponding rules and specifications, so the corresponding data content can be found quickly. The location of the data you want and the format of the data can be quickly found in the list, which can facilitate the reuse of the data in the data lake, improve the efficiency of using policy data, and help use data to promote business development.
[0033] In one embodiment, in step S30, the data characteristics include data characteristics and data common characteristics, and the target insurance policy data in the target database is marked with a label according to the data source business module and the data characteristics, including: According to the data source business module, the data's own characteristics and the data's common characteristics, the target insurance policy data in the target database are marked with tags accordingly.
[0034] In this embodiment, in order to facilitate the subsequent reuse of the data, in accordance with the data source business module and the data characteristics, the target policy data in the target database are marked with labels accordingly, and the label granularity of the marking labels is further subdivided, that is, the data characteristics include the data's own characteristics and the data's common characteristics. The data's own characteristics refer to the fact that some data have their own particularities, such as the characteristics that characterize the regional type, consumption capacity or hobbies; the data's common characteristics refer to the data's format, form, data recording form, etc.
[0035] The data source business refers to the marking information or associated information that characterizes the source of the policy data. Similarly, taking the above example, the source of the insurance contract and the classification of the contract will be used, and then the policy data will be re-marked with the corresponding source marking label and characteristic marking label. For example, Xiao Ming, who is under 18 years old in Guangdong, purchased an insurance policy, and the policy can be marked with a label such as underage or Pearl River Delta.
[0036] In one embodiment, the marking tag at least includes a service source tag, an own characteristic tag, a data type, a data region tag, and a consumption capability tag, without specific limitation.
[0037] In this embodiment, in the process of marking the target policy data in the target database according to the data source business module and the data characteristics, the fineness of the marking labels is further subdivided so that more connections can be established between the data. The data in the lake also has more detailed labels and corresponding rules and specifications, which is conducive to quickly finding the corresponding data content in the subsequent time, and is also conducive to the fineness of splicing and reorganization. It can be more convenient to reuse the data in the data lake, improve the efficiency of using policy data, and is conducive to using data to promote business development.
[0038] In one embodiment, if Figure 3 As shown, after the service layer data is formed in the data service layer of the target database, the following steps are also included: S60: receiving a data request sent by a data demander, the data request being used to request target data in a certain level in the target database, the data demander including any one or more of the following demanders: a financial settlement demander, a data mining demander, a model training demander, and a data statistics demander; S70: According to the data request, target data is extracted from a certain level in the target database and fed back to the data demander.
[0039] In this example, see Figure 3 as well as Figure 1 As shown, service layer data is formed in the data service layer of the target database, that is, the target database forms the data obtained by processing in the manner of the present application. For the target database, it can be connected to multiple different data demanders / data demand systems. The server can receive a data request sent by a data demander, and the data request is used to request target data in a certain level in the target database. The certain level includes but is not limited to the initial data layer, the data detail layer, and the data service layer, and each level includes corresponding data of different levels. Exemplarily, as an example, the data demander includes any one or more of the following demanders: financial settlement demanders, data mining demanders, model training demanders, and data statistics demanders.
[0040] It should be understood that different data demanders have different requirements for data usage, and the data lake formed based on the embodiment of the present application includes data at multiple different levels, and the data has labels and corresponding rules and specifications. In addition to being able to quickly find the corresponding data content, you can also find the corresponding data of the demander according to the needs. For example, data used for financial settlement, report data, machine learning, data mining, and empirical statistics, the above different demand data are different, and the required data can be quickly and conveniently extracted through the data lake.
[0041] It can be seen that in this embodiment, since the target database forms a data lake, all users with data needs can come here to obtain data, and there is no need to look for data sources everywhere. There will be different data layers here, and you can choose data of different levels according to your needs, and the data are all labeled, so you can quickly find the data you need. The data is commonly classified and processed so that the subsequent data demanders can use it quickly and directly, without having to perform ordinary dimensional data processing themselves. In addition, since the data in the data lake is cleaned and corrected, the accuracy of the data quality is more reliable. It can be said that the data lake obtained by processing through the embodiment of the present application is equivalent to a unified platform to realize multiple data uses, which can solve the data usage needs of various enterprises in one stop.
[0042] In one embodiment, after step S70, that is, according to the data request, the target data is extracted from a certain level in the target database and fed back to the data demander, and the following steps are also included: S80: Aggregate all target data extracted from a certain level in the target database to form application layer data in the data application layer of the target database.
[0043] In this embodiment, application layer data is also formed in the data service layer of the target database, that is, the target database forms another type of data obtained by processing in the manner of this application, and each layer includes corresponding data of different levels, including original data layer, data detail layer, data service layer and application layer data. Since it is formed, when it is called again later, the required data can be directly extracted from the application layer data according to the application scenario of the data demander, which is more convenient.
[0044] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0045] In one embodiment, a data processing system is provided, the data processing system comprising a data processing device, a target database, a plurality of distributed databases of an enterprise, and at least one data demander, wherein: The data processing device synchronizes the original policy data of each dispersed database of the enterprise to the original data layer of the target database; The data processing device extracts the original policy data of the original data layer for preprocessing based on the preset policy preprocessing rules to obtain target policy data; The data processing device tags the target policy data in the target database accordingly according to the data source business module and data characteristics; The data processing device reorganizes and splices the extracted target policy data according to the mark labels of the target policy data, so as to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; The data processing device summarizes the policy detail data corresponding to the different mark label dimensions formed by the data detail layer to form service layer data in the data service layer of the target database, so that the data demander can extract the required target data from a certain level of the target database.
[0046] In one embodiment, the data characteristics include data characteristics and data common characteristics. The data processing device tags the target insurance policy data in the target database accordingly according to the data source business module and the data characteristics, including: The data processing device marks the target policy data in the target database accordingly according to the data source business module, the data's own characteristics and the data's common characteristics.
[0047] In one embodiment, the marking tag includes at least a business source tag, an own characteristic tag, a data type, a data region tag, and a consumption capability tag.
[0048] In one embodiment, after the data service layer of the target database forms the service layer data, the method further includes: The data processing device receives a data request sent by a data demander, wherein the data request is used to request target data in a certain level in the target database, and the data demander includes any one or more of the following demanders: a financial settlement demander, a data mining demander, a model training demander, and a data statistics demander; The data processing device extracts target data from a certain level in the target database according to the data request and feeds back the target data to the data demander.
[0049] In one embodiment, after the data processing device aggregates the insurance policy detail data corresponding to different marking label dimensions formed at the data detail layer to form service layer data at the data service layer of the target database, it further includes: The data processing device aggregates all target data extracted from a certain level in the target database to form application layer data in a data application layer of the target database.
[0050] In one embodiment, the original insurance policy data includes underwriting data, preservation data, claims data, renewal data, and insurance agent data.
[0051] It can be seen that in this embodiment, a data processing system based on a data lake is provided. Different levels of data will be formed in the data lake. All levels of data in the same data lake will be interoperable and no longer isolated. Connections can be quickly established between data and they can be used directly. The data in the lake has labels and corresponding rules and specifications, so the corresponding data content can be found very quickly. The location of the data you want and the format of the data can be quickly found in the list, which can facilitate the reuse of the data in the data lake, improve the efficiency of using policy data, and is conducive to using data to promote business development.
[0052] In one embodiment, a data processing device based on a data lake is provided, and the data processing device based on a data lake corresponds one-to-one to the data processing method based on a data lake in the above embodiment. Figure 4 As shown, the data processing device based on the data lake includes a synchronization module 101, a preprocessing module 102, a marking module 103, a splicing module 104 and a summary module 105. The functional modules are described in detail as follows: The synchronization module 101 is used to synchronize the original policy data of each dispersed database of the enterprise to the original data layer of the target database; The preprocessing module 102 is used to extract the original policy data of the original data layer for preprocessing based on the preset policy preprocessing rules to obtain target policy data; The marking module 103 is used to mark the target policy data in the target database according to the data source business module and data characteristics; The splicing module 104 is used to reorganize and splice the extracted target policy data according to the mark labels of the target policy data, so as to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; The aggregation module 105 is used to aggregate the insurance policy detail data corresponding to different marking label dimensions formed in the data detail layer to form service layer data in the data service layer of the target database.
[0053] In one embodiment, the marking module 103 is specifically used for: According to the data source business module, the data's own characteristics and the data's common characteristics, the target insurance policy data in the target database are marked with tags accordingly.
[0054] In one embodiment, the marking tag includes at least a business source tag, an own characteristic tag, a data type, a data region tag, and a consumption capability tag.
[0055] In one embodiment, the data lake-based data processing device further includes a receiving module; A receiving module, configured to receive a data request sent by a data demander, wherein the data request is used to request target data in a certain level in the target database, and the data demander includes any one or more of the following demanders: a financial settlement demander, a data mining demander, a model training demander, and a data statistics demander; The sending module is used to extract the target data from a certain level in the target database according to the data request and feed it back to the data demander.
[0056] In one embodiment, the aggregation module 105 is further configured to: All target data extracted from a certain level in the target database are aggregated to form application layer data in a data application layer of the target database.
[0057] In one embodiment, the original insurance policy data includes underwriting data, preservation data, claims data, renewal data, and insurance agent data.
[0058] In this embodiment, a data processing device based on a data lake is provided. Different levels of data will be formed in the data lake. Data of all levels in the same data lake will be interoperable and no longer isolated. Data can be quickly connected and can be used directly. The data in the lake has labels and corresponding rules and specifications, so the corresponding data content can be found very quickly. The location of the data you want and the format of the data can be quickly found in the list, which can facilitate the reuse of the data in the data lake, improve the efficiency of using policy data, and is conducive to using data to promote business development.
[0059] For the specific definition of the data processing device based on the data lake, please refer to the definition of the data processing device method based on the data lake above, which will not be repeated here. Each module in the above-mentioned data processing device based on the data lake can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0060] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, it realizes the functions of a data processing device based on a data lake, or when it is executed, it realizes the steps of a data processing method based on a data lake.
[0061] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the function of a data processing device based on a data lake is realized, or when the processor executes the computer program, the steps of a data processing method based on a data lake are realized to avoid repetition.
[0062] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the functions of a data processing device based on a data lake, or implements the steps of a data processing method based on a data lake. To avoid repetition, it is not repeated here.
[0063] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0064] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0065] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A data processing method based on a data lake, characterized in that: include: Synchronize the original policy data of each scattered database of the enterprise to the original data layer of the target database; Based on the preset policy preprocessing rules, the original policy data of the original data layer is extracted for preprocessing to obtain target policy data; According to the data source business module and data characteristics, the target policy data in the target database is marked accordingly; According to the mark labels of the target policy data, the extracted target policy data are reorganized and spliced to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; The insurance policy detail data corresponding to different marking label dimensions formed in the data detail layer are aggregated to form service layer data in the data service layer of the target database.
2. The data processing method based on data lake according to claim 1 is characterized in that: The data characteristics include data characteristics and data common characteristics. The target insurance policy data in the target database is marked with a label according to the data source business module and the data characteristics, including: According to the data source business module, the data's own characteristics and the data's common characteristics, the target insurance policy data in the target database are marked with tags accordingly.
3. The data processing method based on data lake according to claim 2 is characterized in that: The marking label at least includes a business source label, an own characteristic label, a data type, a data region label and a consumption capability label.
4. The data processing method based on a data lake according to any one of claims 1 to 3, characterized in that: After the data service layer of the target database forms the service layer data, the method further includes: Receiving a data request sent by a data demander, the data request is used to request target data in a certain level in the target database, the data demander includes any one or more of the following demanders: a financial settlement demander, a data mining demander, a model training demander, and a data statistics demander; According to the data request, target data is extracted from a certain level in the target database and fed back to the data demander.
5. The data processing method based on data lake according to claim 4 is characterized in that: The step of aggregating the insurance policy detail data corresponding to different marking label dimensions formed at the data detail layer to form service layer data at the data service layer of the target database includes: All target data extracted from a certain level in the target database are aggregated to form application layer data in a data application layer of the target database.
6. The data processing method based on a data lake according to any one of claims 1 to 3, characterized in that: The original policy data includes underwriting data, preservation data, claims data, renewal data, and insurance agent data.
7. A data processing system based on a data lake, characterized in that: The data processing system includes a data processing device, a target database, a plurality of distributed databases of an enterprise, and at least one data demander, wherein: The data processing device synchronizes the original policy data of each dispersed database of the enterprise to the original data layer of the target database; The data processing device extracts the original policy data of the original data layer for preprocessing based on the preset policy preprocessing rules to obtain target policy data; The data processing device tags the target policy data in the target database accordingly according to the data source business module and data characteristics; The data processing device reorganizes and splices the extracted target policy data according to the mark labels of the target policy data, so as to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; The data processing device summarizes the policy detail data corresponding to the different mark label dimensions formed by the data detail layer to form service layer data in the data service layer of the target database, so that the data demander can extract the required target data from a certain level of the target database.
8. A data processing device based on a data lake, characterized in that: include: The synchronization module is used to synchronize the original policy data of each decentralized database of the enterprise to the original data layer of the target database; A preprocessing module, for extracting the original policy data of the original data layer for preprocessing based on a preset policy preprocessing rule to obtain target policy data; A marking module, used to mark the target policy data in the target database accordingly according to the data source business module and data characteristics; A splicing module, used for reorganizing and splicing the extracted target policy data according to the mark labels of the target policy data, so as to form policy detail data corresponding to different mark label dimensions at the data detail layer of the target database; The aggregation module is used to aggregate the insurance policy detail data corresponding to different marking label dimensions formed in the data detail layer to form service layer data in the data service layer of the target database.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method based on the data lake is implemented as described in any one of claims 1 to 6.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method based on a data lake is implemented as described in any one of claims 1 to 6.