Big data integration method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a big data integration method and device, equipment and a storage medium, and the method comprises the steps: determining to-be-processed data from to-be-integrated data; wherein the to-be-processed data comprises first data, second data and third data; the first data is determined according to timestamps corresponding to all specified service stages in the to-be-integrated data; the second data is determined according to a service main key corresponding to each specified service behavior in the to-be-integrated data; the third data is determined according to a predetermined common universal dimension; performing data conversion on the determined to-be-processed data according to a preset data conversion process; the data conversion process at least comprises one of code value standardization processing, data desensitization and data length standardization processing; and loading the converted data to a target database.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a big data integration method, apparatus, device and storage medium. Background Art

[0002] Currently, data integration in the insurance industry typically begins with building a distributed cluster and constructing a data foundation, such as a data warehouse or data lake. Incremental markers are then used to regularly identify data changes. Standard ETL (Extract, Transform, and Translate) or system integration are then used to extract data from multiple raw data sources, transform it, and load it into the target database to achieve data integration. This data integration approach can address data processing and response issues, improve statistical analysis efficiency, and optimize business process logic.

[0003] However, the insurance business involves complex links such as insurance application, revision, and termination, which will interfere with the screening logic based on conventional incremental identifiers such as business primary keys and timestamps, resulting in inaccurate data changes identified by incremental identifiers, making big data integration less effective. Summary of the Invention

[0004] The embodiments of the present disclosure provide a big data integration method, apparatus, device and storage medium to solve the problem in related technologies that data changes are identified by relying on incremental identification, while the insurance business has complex links such as insurance application, revision, and final insurance, which will interfere with the screening logic based on conventional incremental identification, resulting in inaccurate data changes identified by relying on incremental identification, resulting in poor big data integration effect.

[0005] In a first aspect, an embodiment of the present disclosure provides a big data integration method, the method comprising: Determine data to be processed from the data to be integrated; wherein the data to be processed includes first data, second data, and third data; wherein the first data is determined based on timestamps corresponding to each specified business stage in the data to be integrated; the second data is determined based on business primary keys corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a pre-determined common universal dimension; Performing data conversion on the determined data to be processed according to a preset data conversion process; the data conversion process includes at least one of the following: code value standardization, data desensitization, and data length standardization; Load the transformed data into the target database.

[0006] In a second aspect, an embodiment of the present disclosure provides a big data integration device, the device comprising: An extraction module is configured to determine data to be processed from the data to be integrated; wherein the data to be processed includes first data, second data, and third data; the first data is determined based on timestamps corresponding to each specified business stage in the data to be integrated; the second data is determined based on business primary keys corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a predetermined common dimension. A conversion module, configured to perform data conversion on the determined data to be processed according to a preset data conversion process; the data conversion process includes at least one of the following: code value standardization, data desensitization, and data length standardization; The loading module is used to load the converted data into the target database.

[0007] In a third aspect, an embodiment of the present disclosure provides a big data integration device, comprising: a processor; and a memory configured to store computer-executable instructions, which, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the method described in the first aspect.

[0010] The at least one technical solution provided by the embodiments of the present invention can achieve the following technical effects: In an embodiment of the present invention, the data to be processed can be first determined from the data to be integrated, wherein the data to be processed may include first data, second data, and third data. The first data is determined based on the timestamp corresponding to each specified business stage in the data to be integrated; the second data is determined based on the business primary key corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a predetermined common universal dimension. Then, the determined data to be processed can be converted according to a preset data conversion process; the data conversion process may include at least one of the following: code value standardization, data desensitization, and data length standardization. After the conversion is completed, the converted data can be loaded into the target database.

[0011] The embodiment of the present invention can first determine the data to be processed from the data to be integrated, and determine it in a multi-dimensional way, that is, according to the timestamp corresponding to each specified business stage and the business primary key corresponding to each specified business behavior, which changes the method of relying solely on conventional incremental identification, comprehensively considers the business stage and behavior information, and can cope with the interference of complex links in the insurance business on the conventional incremental identification screening logic, so that the determined data is more accurate. In addition, the third data is determined based on the pre-determined public general dimension to provide a unified standard and basis for data processing. Subsequently, according to the preset data conversion process, the determined data to be processed is subjected to data conversion operations such as code value standardization, data desensitization, and data length standardization to further standardize the data format. Finally, the converted data is loaded into the target database to complete the data integration, which effectively solves the problem of inaccurate recognition of data changes and poor big data integration effect by relying on incremental identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] To more clearly illustrate one or more embodiments of the present disclosure or technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments described in the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. Figure 1 A flowchart of a big data integration method provided by one embodiment of the present invention; Figure 2 A schematic diagram of a scenario of a big data integration method provided by one embodiment of the present invention; Figure 3 A schematic diagram of the module composition of a big data integration device 300 provided in one embodiment of the present invention; Figure 4 A schematic diagram of the hardware structure of a big data integration device provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure and to make the purposes, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with specific embodiments of the present disclosure and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0014] The technical solutions provided by various embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0015] See Figure 1 , Figure 1 A flow chart of a big data integration method provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method includes the following steps: Step 102: Determine the data to be processed from the data to be integrated; wherein the data to be processed includes first data, second data, and third data; wherein the first data is determined based on the timestamp corresponding to each specified business stage in the data to be integrated; the second data is determined based on the business primary key corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a predetermined common dimension.

[0016] Step 104: Perform data conversion on the determined data to be processed according to a preset data conversion process; the data conversion process includes at least one of the following: code value standardization, data desensitization, and data length standardization.

[0017] Step 106: Load the converted data into the target database.

[0018] In the embodiment of the present invention, the data to be processed may be determined from the data to be integrated, wherein the data to be processed may include first data, second data, and third data.

[0019] The first data may be determined based on the timestamps corresponding to each designated business stage in the data to be integrated. The designated business stages may include the underwriting business stage, the insurance approval business stage, and the insurance termination business stage. When determining the first data, the first data may be determined from the data to be integrated based on the timestamps corresponding to each designated business stage.

[0020] In one example, the process of determining the first data based on the timestamp corresponding to the underwriting business data extraction stage can be: if the underwriting business occurs within the integration period, the policy records are filtered by the underwriting time; if the data changes due to correction, migration, etc. within the integration period, the policy change records are filtered by the change timestamp.

[0021] The process of determining the first data based on the timestamp corresponding to the correction business data extraction stage can be: if the correction business occurs within the integration period, the policy track records are filtered by the approval time; if the data changes due to correction, migration, etc. within the integration period, the policy track change records are filtered by the change timestamp.

[0022] The process of determining the first data according to the timestamp corresponding to the terminal insurance business data extraction stage may be: terminal insurance business occurs within the integration period, and the policy records are filtered according to the terminal insurance time.

[0023] In an embodiment of the present invention, the second data can be determined based on the business primary key corresponding to each specified business activity in the data to be integrated. Specifically, the specified business activities may include claims settlement, premium collection and payment, handling fee collection and payment, complaint processing, and return visit processing. To determine the second data from the data to be integrated, the policy records can be back-filled based on the business primary key corresponding to each specified business activity. The second data can then be determined based on the back-filled policy records.

[0024] In one example, when the designated business behavior is a claims business behavior, the process of backfilling the policy records according to the business primary key corresponding to each designated business behavior can be: when a claims business occurs within the integration period, the policy records are backfilled with the policy primary key corresponding to the relevant claims business primary key, and the policy records that have already been backfilled will not be backfilled repeatedly.

[0025] When the designated business behavior is a premium collection and payment business behavior, the process of backfilling the policy records according to the business primary key corresponding to each designated business behavior can be: when premium collection and payment business occurs within the integration period, the policy record is backfilled using the policy primary key corresponding to the premium collection and payment business primary key.

[0026] When the designated business behavior is a fee collection and payment business behavior, the process of backfilling the policy records according to the business primary key corresponding to each designated business behavior can be: when a fee collection and payment business occurs within the integration period, the policy record is backfilled using the policy primary key corresponding to the fee collection and payment business primary key.

[0027] When the designated business behavior is a complaint business behavior, the process of backfilling the policy records according to the business primary key corresponding to each designated business behavior can be as follows: when a complaint business occurs within the integration cycle, the policy record is backfilled with the policy primary key corresponding to the complaint business primary key, and the policy record that has been backfilled will not be backfilled repeatedly.

[0028] When the designated business behavior is a return business behavior, the process of replenishing the policy records according to the business primary key corresponding to each designated business behavior can be as follows: when a return business occurs within the integration cycle, the policy records are replenished with the policy primary key corresponding to the return business primary key, and the policy records that have been replenished will not be replenished repeatedly.

[0029] In an embodiment of the present invention, the third data may be determined based on a pre-determined common universal dimension. Among them, the common universal dimension may be determined based on the commonality of the business process. The insurance business includes multiple links such as underwriting, revision, and claims settlement. Although the business rules of each link are different, there are common data requirements. When constructing different subject data tables, in order to achieve consistent data processing, common procedural data is extracted from the business process. The basic customer information (name, contact information, etc.) and basic insurance product information (type of insurance, insured amount, etc.) in the policy information will be involved in business links such as underwriting, claims settlement, and revision. The extraction and processing rules for these common data are part of the common logic. By sorting out the business process, the processing rules of these common data can be found, the common logic can be determined, and unified assignment can be achieved to improve data processing efficiency and consistency.

[0030] As can be seen from the above content, the data extraction step in the embodiment of the present invention can connect to multiple upstream data sources to identify and extract incremental data. It can mainly include incremental data identification, common dimension data merging and extraction, and incremental data extraction. Among them, incremental data identification includes multi-stage business primary key merging, extraction, and replenishment; multi-stage can refer to splitting the business primary key merging, extraction, and replenishment actions according to different data sources and business scenarios; common dimension data merging and extraction extracts the general procedural data required for the data processing process through common logic; the incremental data extraction part extracts all data records to be summarized from the data source according to the target database based on the incremental data screening logic such as business primary key and timestamp.

[0031] Among them, in the part of multi-stage business primary key merging, extraction and replenishment, taking the insurance policy data integration as an example, when any business link in the insurance business life cycle (including but not limited to underwriting, revision, termination, claims, financial accounting behavior, customer service behavior, etc.) occurs within the target integration cycle, it may trigger the insurance policy data integration, aiming to achieve panoramic business backtracking through data.

[0032] Taking monthly integration as an example, business key extraction includes the underwriting business data extraction stage, the revision business data extraction stage, and the final insurance business data extraction stage. Business key restoration includes the claim business behavior restoration stage, the premium collection and payment business behavior restoration stage, the handling fee collection and payment business behavior restoration stage, the complaint business behavior restoration stage, and the follow-up business behavior restoration stage. Since these contents have been detailed in the above embodiment, they will not be repeated in this embodiment of the present invention.

[0033] After determining the data to be processed, data conversion can be performed on the determined data to be processed according to a preset data conversion process, wherein the data conversion process can include at least one of the following: code value standardization, data desensitization, and data length standardization.

[0034] In an embodiment of the present invention, when performing data conversion, a series of conversion, calculation, and process flow processing can be performed to process all the original incremental data extracted from the data source into the final state required for the report or market, which mainly includes business code conversion, privacy data desensitization, and data interception. Among them, business code conversion maps the original values ​​of fields with predefined value ranges to target code values ​​according to standard specifications; privacy data desensitization is responsible for desensitizing and transforming personal sensitive information fields; data interception intercepts the overlong field values ​​of the converted data according to the requirements of the target database to ensure that the data is smoothly loaded into the target database.

[0035] After the data conversion is completed, the converted data can be loaded into the target database.

[0036] In one embodiment of the present invention, data loading is responsible for determining the final data version and loading it into the target database. This process includes change data identification and data synchronization. During change data identification, a full-field summary comparison of row records can be performed on the converted incremental data against the existing data in the target database. This is used to filter out incremental data extracted due to timestamp changes caused by non-target model field updates, determine the target incremental data version, and avoid synchronizing invalid duplicate data. During data synchronization, the target incremental data is loaded into the target database to complete data integration.

[0037] In one example, if Figure 2 The hash value calculation (Cal hash value) shown in the figure calculates a hash value (h1, h2, etc.) for each data record to be synchronized (such as record_1, record_2, etc.). For record_1, the calculated hash value h1 is marked as "√", indicating that it passes the verification rule. For record_3, the calculated hash value h2 is equal to the hash value h2 calculated for record_2, indicating a duplicate record. Therefore, the hash value h2 calculated for record_3 is marked as "×" and not integrated. The hash value calculation here is a key step in identifying changed data. By comparing the hash values ​​of existing and incremental data records (i.e., comparing all-field summaries), we determine whether data attributes have truly changed. If attributes have changed, such as in record_1 and record_2, they can be synchronized. If attributes have not changed, such as in record_3, they are not synchronized.

[0038] After the transformed data is loaded into the target database, quality monitoring of the data loaded into the target database may be performed.

[0039] In one embodiment of the present invention, when quality monitoring is performed on data loaded into a target database, data quality verification can be performed first using predefined verification rules. The quality verification can include at least one of the following: the presence of null values, inconsistency of data between tables, and then, anomaly detection can be performed on the data that has passed the data quality verification based on predefined anomaly monitoring indicators and anomaly detection models.

[0040] Specifically, data quality monitoring can be used to implement data integration quality assurance mechanisms to discover data anomalies at the source of data or during processing. It can include functions such as data verification, indicator modeling, anomaly detection, and result distribution. Among them, data verification can batch execute verification rules on the full amount of loaded data to verify simple data quality issues; indicator modeling manages data anomaly monitoring indicators configured according to business rules and various calculation models built based on statistical models, machine learning, time series prediction and other methods; anomaly detection can perform batch calculations based on the data to be tested and predefined indicators and training models to form a set of test results; and result distribution can distribute the test result set to designated data management users for result confirmation and to promote subsequent source data governance, system transformation and business process compliance.

[0041] In addition, in the embodiment of the present invention, when quality monitoring is performed on the data loaded into the target database, it can also be combined with the ETL process, and the quality management of the data loaded into the target database can be performed according to certain rules through automated and intelligent means. First, data problems with simple deterministic logic such as null values ​​and inconsistencies between tables are discovered through standardized verification rules; second, business data problems such as value range exceeding the limit, violation of business rationality, and code distribution abnormalities are further identified using abnormal monitoring indicators; third, various calculation models are constructed based on various statistical, machine learning, time series prediction and other methods, and potential data abnormal fluctuations are detected in combination with historical data and custom thresholds. In practice, by combining with the ETL process, batch verification, feature engineering and learning modeling can be performed on the data integration results, and an abnormal detection indicator set can be output to present the data integration situation in a panoramic manner, maximize the exposure of potential data problems, and build a data distribution channel with upstream business systems and relevant competent departments to achieve integrated closed-loop management of data processes and business processes, so that data quality standardization can feed back to the standardization of business processes.

[0042] In an embodiment of the present invention, the above-mentioned data extraction step, data conversion step, and loading step can also be scheduled. When executing, they can be executed through a scheduling execution module. Specifically, the scheduling execution module can cascade the various functional components to realize the overall data integration process, which mainly includes task identification, process orchestration, data source management, task execution, and logging functions. Among them, task identification can scan the interface table through a timer program and place the tasks to be executed into the task pool; process orchestration can define the order of task execution according to business scenarios or data processing requirements; data source management can be configured to adapt to different source database types; logging can record the task execution status and error information of the data extraction, conversion, and loading processes in real time.

[0043] In an embodiment of the present invention, the data to be processed can be first determined from the data to be integrated, wherein the data to be processed may include first data, second data, and third data. The first data is determined based on the timestamp corresponding to each specified business stage in the data to be integrated; the second data is determined based on the business primary key corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a predetermined common universal dimension. Then, the determined data to be processed can be converted according to a preset data conversion process; the data conversion process may include at least one of the following: code value standardization, data desensitization, and data length standardization. After the conversion is completed, the converted data can be loaded into the target database.

[0044] The embodiment of the present invention can first determine the data to be processed from the data to be integrated, and determine it in a multi-dimensional way, that is, according to the timestamp corresponding to each specified business stage and the business primary key corresponding to each specified business behavior, which changes the method of relying solely on conventional incremental identification, comprehensively considers the business stage and behavior information, and can cope with the interference of complex links in the insurance business on the conventional incremental identification screening logic, so that the determined data is more accurate. In addition, the third data is determined based on the pre-determined public general dimension to provide a unified standard and basis for data processing. Subsequently, according to the preset data conversion process, the determined data to be processed is subjected to data conversion operations such as code value standardization, data desensitization, and data length standardization to further standardize the data format. Finally, the converted data is loaded into the target database to complete the data integration, which effectively solves the problem of inaccurate recognition of data changes and poor big data integration effect by relying on incremental identification.

[0045] Corresponding to the above-mentioned big data integration method, an embodiment of the present invention further provides a big data integration device, Figure 3 A schematic diagram of the module composition of the big data integration device 300 provided in an embodiment of the present invention is shown in FIG. Figure 3 As shown, the big data integration device 300 includes: Extraction module 301 is configured to determine data to be processed from the data to be integrated; wherein the data to be processed includes first data, second data, and third data; the first data is determined based on timestamps corresponding to each specified business stage in the data to be integrated; the second data is determined based on business primary keys corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on pre-determined common dimensions; The conversion module 302 is configured to perform data conversion on the determined data to be processed according to a preset data conversion process; the data conversion process includes at least one of the following: code value standardization, data desensitization, and data length standardization; The loading module 303 is used to load the converted data into the target database.

[0046] Optionally, the designated business stage includes an underwriting business stage, an insurance revision business stage, and a final insurance business stage; the extraction module 301 is used to: Determining the first data from the data to be integrated; The extraction module 301 is further configured to: The first data is determined from the data to be integrated according to the timestamps corresponding to the designated business stages.

[0047] Optionally, the designated business behavior includes: claim settlement business behavior, premium collection and payment business behavior, fee collection and payment business behavior, complaint business behavior, and return visit business behavior; the extraction module 301 is used to: Determining the second data from the data to be integrated; The extraction module 301 is further configured to: Backfill the policy records based on the business primary key corresponding to each designated business behavior; The second data is determined based on the policy record after the cover is made.

[0048] Optionally, the loading module 303 is used to: Calculating full-field hash values ​​for the converted incremental data and the data in the target database; Determine the incremental data having a different hash value from the data in the target database as the target incremental data; The target incremental data is loaded into the target database.

[0049] Optionally, the method further includes ( Figure 3 (not shown): The monitoring module 304 is configured to monitor the quality of the data loaded into the target database after the converted data is loaded into the target database.

[0050] Optionally, the monitoring module 304 is configured to: Perform data quality verification using predefined verification rules; the quality verification includes at least one of the following: the presence of null values, inconsistent data between tables; Based on predefined anomaly monitoring indicators and anomaly detection models, anomaly detection is performed on data that has passed data quality verification.

[0051] In an embodiment of the present invention, the data to be processed can be first determined from the data to be integrated, wherein the data to be processed may include first data, second data, and third data. The first data is determined based on the timestamp corresponding to each specified business stage in the data to be integrated; the second data is determined based on the business primary key corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a predetermined common universal dimension. Then, the determined data to be processed can be converted according to a preset data conversion process; the data conversion process may include at least one of the following: code value standardization, data desensitization, and data length standardization. After the conversion is completed, the converted data can be loaded into the target database.

[0052] The embodiment of the present invention can first determine the data to be processed from the data to be integrated, and determine it in a multi-dimensional way, that is, according to the timestamp corresponding to each specified business stage and the business primary key corresponding to each specified business behavior, which changes the method of relying solely on conventional incremental identification, comprehensively considers the business stage and behavior information, and can cope with the interference of complex links in the insurance business on the conventional incremental identification screening logic, so that the determined data is more accurate. In addition, the third data is determined based on the pre-determined public general dimension to provide a unified standard and basis for data processing. Subsequently, according to the preset data conversion process, the determined data to be processed is subjected to data conversion operations such as code value standardization, data desensitization, and data length standardization to further standardize the data format. Finally, the converted data is loaded into the target database to complete the data integration, which effectively solves the problem of inaccurate recognition of data changes and poor big data integration effect by relying on incremental identification.

[0053] Corresponding to the above-mentioned big data integration method, an embodiment of the present invention further provides a big data integration device, Figure 4 A schematic diagram of the hardware structure of a big data integration device provided in one embodiment of the present invention.

[0054] The big data integration device may be the terminal device or server for integrating big data provided in the above embodiments.

[0055] Big data integration devices can vary significantly depending on their configuration or performance. They may include one or more processors 401 and memory 402. Memory 402 may store one or more applications or data. Memory 402 may be either transient or persistent. Applications stored in memory 402 may include one or more modules (not shown), each of which may include a series of computer-executable instructions for the big data integration device. Furthermore, processor 401 may be configured to communicate with memory 402 to execute the series of computer-executable instructions in memory 402 on the big data integration device. Big data integration devices may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input / output interfaces 405, and one or more keyboards 406.

[0056] Specifically in this embodiment, the big data integration device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the big data integration device, and is configured to be executed by one or more processors.

[0057] In an embodiment of the present invention, the data to be processed can be first determined from the data to be integrated, wherein the data to be processed may include first data, second data, and third data. The first data is determined based on the timestamp corresponding to each specified business stage in the data to be integrated; the second data is determined based on the business primary key corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a predetermined common universal dimension. Then, the determined data to be processed can be converted according to a preset data conversion process; the data conversion process may include at least one of the following: code value standardization, data desensitization, and data length standardization. After the conversion is completed, the converted data can be loaded into the target database.

[0058] The embodiment of the present invention can first determine the data to be processed from the data to be integrated, and determine it in a multi-dimensional way, that is, according to the timestamp corresponding to each specified business stage and the business primary key corresponding to each specified business behavior, which changes the method of relying solely on conventional incremental identification, comprehensively considers the business stage and behavior information, and can cope with the interference of complex links in the insurance business on the conventional incremental identification screening logic, so that the determined data is more accurate. In addition, the third data is determined based on the pre-determined public general dimension to provide a unified standard and basis for data processing. Subsequently, according to the preset data conversion process, the determined data to be processed is subjected to data conversion operations such as code value standardization, data desensitization, and data length standardization to further standardize the data format. Finally, the converted data is loaded into the target database to complete the data integration, which effectively solves the problem of inaccurate recognition of data changes and poor big data integration effect by relying on incremental identification.

[0059] Another embodiment of the present disclosure further provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the above process is implemented.

[0060] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned big data integration method embodiment and achieve the same effects and functions, which will not be repeated here.

[0061] Another embodiment of the present disclosure further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above process is implemented.

[0062] The computer program product in the embodiment of the present disclosure can implement each process of the above-mentioned big data integration method embodiment and achieve the same effects and functions, which will not be repeated here.

[0063] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0064] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using physical hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by the customer's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly performed using software called a "logic compiler." This is similar to the software compilers used during program development. Before compilation, the original code must be written in a specific programming language, called a Hardware Description Language (HDL). There are many types of HDL, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that simply by programming a method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0065] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller purely in computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the means for implementing the various functions included therein can also be considered as structures within the hardware component. Alternatively, the means for implementing the various functions can be considered both a software module implementing the method and a structure within the hardware component.

[0066] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0067] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0068] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system, or computer program product. Thus, one or more embodiments of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0072] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0073] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0074] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0075] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. A big data integration method, characterized in that: The method comprises: Determine data to be processed from the data to be integrated; wherein the data to be processed includes first data, second data, and third data; the first data is determined based on timestamps corresponding to each specified business stage in the data to be integrated; the second data is determined based on business primary keys corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a pre-determined common universal dimension; Performing data conversion on the determined data to be processed according to a preset data conversion process; the data conversion process includes at least one of the following: code value standardization, data desensitization, and data length standardization; Load the transformed data into the target database.

2. The method according to claim 1, characterized in that The designated business stages include the underwriting business stage, the insurance revision business stage, and the final insurance business stage; and determining the data to be processed from the data to be integrated includes: Determining the first data from the data to be integrated; The determining the first data from the data to be integrated includes: The first data is determined from the data to be integrated according to the timestamps corresponding to the designated business stages.

3. The method according to claim 1, characterized in that The designated business behaviors include: claims settlement business behaviors, premium collection and payment business behaviors, fee collection and payment business behaviors, complaint business behaviors, and return visit business behaviors; the data to be processed is determined from the data to be integrated, including: Determining the second data from the data to be integrated; The determining the second data from the data to be integrated includes: Backfill the policy records based on the business primary key corresponding to each designated business behavior; The second data is determined based on the policy record after the cover is made.

4. The method according to claim 1, wherein The step of loading the converted data into the target database includes: Calculating full-field hash values ​​for the converted incremental data and the data in the target database; Determine the incremental data having a different hash value from the data in the target database as the target incremental data; The target incremental data is loaded into the target database.

5. The method according to claim 1, wherein After loading the converted data into the target database, the method further includes: Performing quality control on the data loaded into the target database.

6. The method according to claim 5, characterized in that The quality monitoring of the data loaded into the target database includes: Perform data quality verification using predefined verification rules; the quality verification includes at least one of the following: the presence of null values, inconsistent data between tables; Based on predefined anomaly monitoring indicators and anomaly detection models, anomaly detection is performed on data that has passed data quality verification.

7. A big data integration device, characterized in that: The device comprises: An extraction module is configured to determine data to be processed from the data to be integrated; wherein the data to be processed includes first data, second data, and third data; the first data is determined based on timestamps corresponding to each specified business stage in the data to be integrated; the second data is determined based on business primary keys corresponding to each specified business behavior in the data to be integrated; and the third data is determined based on a predetermined common dimension. A conversion module, configured to perform data conversion on the determined data to be processed according to a preset data conversion process; the data conversion process includes at least one of the following: code value standardization, data desensitization, and data length standardization; The loading module is used to load the converted data into the target database.

8. A big data integration device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images