A multi-source data normalization processing method, device, system and storage medium

Abstract

The application provides a multi-source data normalization processing method, device and system and a storage medium, and relates to the technical field of big data. The multi-source data normalization processing method comprises the following steps: based on a big data platform, obtaining change record information of multi-source data of different data sources, and writing the change record information into a message queue; determining a processor corresponding to the change record information in the message queue, and performing normalization processing on the multi-source data corresponding to the change record information through the processor to obtain standard data information; and storing the standard data information in a storage medium corresponding to the standard data information. The application realizes aggregation of multi-source data of different systems and data structures, solves the problems of unclear data definition, non-standard format and non-uniform specification to adapt to different application scenarios, enables different businesses to perform data acquisition and business processing in a normalized manner, and provides convenience for data processing of staff of financial institutions.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and more specifically, to a method, apparatus, system, and storage medium for the normalization of multi-source data. Background Technology

[0002] Financial institutions or their related departments, such as banking systems, have different systems, application platforms, and corresponding databases that store a large amount of data related to business, users, and regulations. The data storage standards differ between different systems, application platforms, and corresponding databases.

[0003] As the volume of business in the banking system grows, the existing methods of data processing, governance, and management have become somewhat limited due to the historical lack of unified standards, constraints, and development specifications for business data. These limitations manifest in many ways, such as issues related to data standardization, data aggregation, and risk control compliance.

[0004] Therefore, due to differences in data storage standards and specifications within the current banking system, financial institutions, or their related departments, data acquisition, data aggregation, risk control, compliance, and other aspects are significantly affected, causing inconvenience for data processing by financial institution staff. Summary of the Invention

[0005] In view of this, and to address the aforementioned technical problems, the present invention provides a method for multi-source data normalization processing, comprising:

[0006] Based on the big data platform, change record information corresponding to multi-source data from different data sources is obtained, and the change record information is written into a message queue;

[0007] The processor corresponding to the change record information in the message queue is determined, and the multi-source data corresponding to the change record information is normalized by the processor to obtain standard data information;

[0008] The standard data information is stored in a storage medium corresponding to the standard data information, so that the standard data information in the storage medium can be accessed through the big data platform.

[0009] Preferably, the step of acquiring change record information corresponding to multi-source data from different data sources based on a big data platform and writing the change record information into a message queue includes:

[0010] The monitoring method is determined based on the database type of the source database where the multi-source data is located, and the source database is monitored using the monitoring method.

[0011] Collect message change logs generated by the source database when data changes occur, and write the message change logs into the message queue.

[0012] Preferably, the step of acquiring change record information corresponding to multi-source data from different data sources based on a big data platform and writing the change record information into a message queue includes:

[0013] Receive push format information from a third-party system that stores the multi-source data, based on the message push format corresponding to the big data platform;

[0014] The push format information is combined into a push data set;

[0015] Write the push format information from the data set into the corresponding message queue.

[0016] Preferably, the big data platform acquires change record information corresponding to multi-source data from different data sources and writes the change record information into a message queue, including:

[0017] According to a preset time interval, retrieve change records from the data source corresponding to the multi-source data;

[0018] Write the change record to the message queue.

[0019] Preferably, the step of determining the processor corresponding to the change record information in the message queue includes:

[0020] Based on the servers deployed on the big data platform, the processor corresponding to the change record information in the message queue is determined by the message type and routing rules of the servers.

[0021] Preferably, before acquiring change record information corresponding to multi-source data from different data sources based on a big data platform and writing the change record information into a message queue, the method further includes:

[0022] Construct a preset standard big data storage protocol; wherein, the preset standard big data storage protocol includes field names, field types, field formats, and field constraint specifications;

[0023] The step of normalizing the multi-source data corresponding to the change record information through the processor to obtain standard data information includes:

[0024] According to a preset standard big data storage protocol, key information from the multi-source data is extracted;

[0025] The key information is converted into a standard format corresponding to the field format in the preset standard big data storage protocol to obtain normalized data, and the normalized data is used as the standard data information.

[0026] Preferably, before extracting key information from the multi-source data according to a preset standard big data storage protocol, the method further includes:

[0027] The validity of the multi-source data corresponding to the message queue is verified.

[0028] After converting the key information into a standard format corresponding to the preset standard big data storage protocol to obtain normalized data, and using the normalized data as the standard data information, the method further includes:

[0029] The standard data information is reviewed and verified so that, after the review and verification are passed, the standard data information is stored in the storage medium.

[0030] Preferably, storing the standard data information in a storage medium corresponding to the standard data information includes:

[0031] Obtain the functional category corresponding to the standard data information;

[0032] The target storage medium into which the standard data information is to be stored is determined based on the functional category.

[0033] The standard data information is stored in the corresponding target storage medium.

[0034] Furthermore, to address the aforementioned problems, the present invention also provides a multi-source data normalization processing apparatus, comprising:

[0035] The acquisition module is used to acquire change record information corresponding to multi-source data from different data sources based on the big data platform, and write the change record information into a message queue;

[0036] The processing module is used to determine the processor corresponding to the change record information in the message queue, and to perform normalization processing on the multi-source data corresponding to the change record information through the processor to obtain standard data information;

[0037] The storage module is used to store the standard data information in a storage medium corresponding to the standard data information, so that the standard data information in the storage medium can be accessed through the big data platform.

[0038] In addition, to solve the above problems, the present invention also provides a multi-source data normalization processing system, including a memory and a processor. The memory stores a multi-source data normalization processing program, and the processor runs the multi-source data normalization processing program to enable the multi-source data normalization processing system to perform the multi-source data normalization processing method as described above.

[0039] In addition, to solve the above problems, the present invention also provides a computer-readable storage medium storing a multi-source data normalization processing program, which, when executed by a processor, implements the multi-source data normalization processing method as described above.

[0040] In addition, to solve the above problems, the present invention also provides a computer-readable storage medium storing a multi-source data normalization processing program, which, when executed by a processor, implements the multi-source data normalization processing method as described above.

[0041] This invention provides a method, apparatus, system, and storage medium for multi-source data normalization processing, wherein the method includes:

[0042] Based on the big data platform, change record information corresponding to multi-source data from different data sources is obtained, and the change record information is written into a message queue;

[0043] The processor corresponding to the change record information in the message queue is determined, and the multi-source data corresponding to the change record information is normalized by the processor to obtain standard data information;

[0044] The standard data information is stored in a storage medium corresponding to the standard data information, so that the standard data information in the storage medium can be accessed through the big data platform.

[0045] This invention utilizes a big data platform to construct a message queue for change records of multi-source data from different data sources. A processor then standardizes the multi-source data and stores it in the corresponding storage medium. This invention aggregates multi-source data from different systems and data structures, internally solving the problems of scattered data storage and management; it addresses issues of unclear data definitions, non-standard formats, and inconsistent specifications; and the output data has the potential to interact with third parties. Furthermore, depending on the data type and requirements, storage methods can be flexibly selected to adapt to different application scenarios. Whether it's data acquisition, data aggregation, or risk control, compliance, and other business operations, data acquisition and business processing can be standardized, providing convenience for data processing for financial institution staff. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the hardware operating environment involved in an embodiment of the multi-source data normalization processing method of the present invention;

[0047] Figure 2 This is a flowchart illustrating the first embodiment of the multi-source data normalization processing method of the present invention;

[0048] Figure 3 This is a flowchart illustrating the refinement of step S100 (steps S110-S120) in the second embodiment of the multi-source data normalization processing method of the present invention.

[0049] Figure 4 This is a flowchart illustrating the refinement of step S100 (steps S130-S150) in the second embodiment of the multi-source data normalization processing method of the present invention.

[0050] Figure 5 This is a flowchart illustrating the refinement of step S100 (steps S160-S170) in the second embodiment of the multi-source data normalization processing method of the present invention.

[0051] Figure 6 This is a detailed flowchart of step S200 in the third embodiment of the multi-source data normalization processing method of the present invention;

[0052] Figure 7 This is a detailed flowchart of step S300 in the fourth embodiment of the multi-source data normalization processing method of the present invention;

[0053] Figure 8 This is a schematic diagram of the module connections of the multi-source data normalization processing device of the present invention;

[0054] Figure 9 This is a schematic diagram of the overall workflow of the multi-source data normalization processing method of the present invention;

[0055] Figure 10 This is a schematic diagram of the workflow from router to processor in the multi-source data normalization processing method of the present invention.

[0056] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0057] The embodiments of the present invention are described in detail below, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0058] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0059] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] like Figure 1 The diagram shown is a structural schematic of the hardware operating environment of the terminal involved in an embodiment of the present invention.

[0062] The multi-source data normalization processing system of this invention can be a PC, or a mobile terminal device such as a smartphone, tablet, or portable computer. This multi-source data normalization processing system may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen, an input unit such as a keyboard, or a remote control; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a stable memory, such as a disk storage device. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001. Optionally, the multi-source data normalization processing system may also include RF (Radio Frequency) circuitry, audio circuitry, a Wi-Fi module, etc. In addition, this multi-source data normalization processing system can also be configured with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated here.

[0063] Those skilled in the art will understand that Figure 1The multi-source data normalization processing system shown is not intended to limit it and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Figure 1 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a data interface control program, a network connection program, and a multi-source data normalization processing program.

[0064] In summary, this invention aggregates multi-source data from different systems and data structures, internally solving the problems of scattered data storage and management; it also addresses issues of unclear data definitions, non-standard formats, and inconsistent specifications, while the output data has the potential to interact with third parties; furthermore, depending on the data type and requirements, storage methods can be flexibly selected to adapt to different application scenarios. Regardless of data acquisition, data aggregation, or risk control, compliance, and other business operations, data acquisition and business processing can be standardized, providing convenience for data processing for financial institution staff.

[0065] Example 1:

[0066] Reference Figure 2 The first embodiment of the present invention provides a method for normalizing multi-source data, comprising:

[0067] Step S100: Based on the big data platform, obtain change record information corresponding to multi-source data from different data sources, and write the change record information into a message queue;

[0068] As mentioned above, different financial institutions, or different departments within the same financial institution, such as Department A of Bank A and Department B of Bank A, may have different storage standards and different data entry standards for the same data. In actual business operations, the differences in standards and specifications may lead to the inability to apply data uniformly or may have a certain impact on normal business operations.

[0069] For example, if customer M has applied for loans in systems A and B respectively, then both systems A and B will store customer information. In this case, system A's customer gender will be "sex" (0 = male, 1 = female) and date of birth (2000-01-01), while system B's customer gender will be "sexy" (1 = male, 2 = female) and date of birth (01 / 01 / 2000). This phenomenon causes a lot of meaningless time to be spent on data format conversion and compatibility when the systems interact.

[0070] For example, for customer M, the account manager needs to count how many loans he / she has in total, what they are, sort them by time, and paginate them. Due to the scattered storage of data, aggregation is difficult. At the same time, if a new business system is added and the new system is reconnected, the complexity of aggregation will increase.

[0071] For example, customer M may act as a guarantor for company N1 in system A, and simultaneously act as a guarantor for other companies in systems B, C, ..., X. If a unified big data system exists, it can analyze user behavior to determine whether customer M is at risk, whether they are on a blacklist, and identify fraudulent users, thus preventing the loss of some company assets.

[0072] As mentioned above, the systems and platforms used by different financial institutions, or different departments within the same financial institution, are all data sources. These data sources store a large amount of multi-source data.

[0073] The big data platform used in this embodiment is designed for retrieving, processing, storing, and further applying multi-source data from different data sources.

[0074] The change record information mentioned above refers to the information related to changes that occur when different data sources perform business processes such as adding new data, changing existing data, modifying and organizing data. This includes information such as the name of the data source, the time of change, the size of the change, the number of bytes, and the content of the change.

[0075] The above-mentioned message queue (MQ) is a queue file used to store change record information, which contains multiple change record information.

[0076] Step S200: Determine the processor corresponding to the change record information in the message queue, and use the processor to perform normalization processing on the multi-source data corresponding to the change record information to obtain standard data information;

[0077] As described above, the big data platform employs one or more servers for deployment, requiring partitioning during data processing. The processor, specifically the message processor, handles the multi-source data linked to by change record information; in this embodiment, normalization processing is employed.

[0078] The aforementioned standardization process can include data auditing, verification, standardized format conversion, cleaning, and expansion, among other things.

[0079] Standardization processing involves adjusting the format of changed data obtained from different data sources to obtain standard data information with the same format, specifications, and standards, and then storing the processed standard data information.

[0080] Step S300: Store the standard data information in a storage medium corresponding to the standard data information, so that the standard data information in the storage medium can be accessed through the big data platform.

[0081] The aforementioned storage media can be the storage device of the deployed server, or various storage media such as hard drives, optical discs, or even cloud storage media.

[0082] As described above, after storage, the big data platform can perform data querying and data analysis based on user commands.

[0083] For example, in data querying, big data platforms can provide a unified data query interface, through which users can obtain data in a standardized and uniform format, thereby enabling further data display.

[0084] For example, data analysis and big data platforms can aggregate collected and standardized data to achieve big data analysis functions.

[0085] This embodiment utilizes a big data platform to construct a message queue for change records of multi-source data from different data sources. A processor then standardizes the multi-source data and stores it in the corresponding storage medium. This embodiment aggregates multi-source data from different systems and data structures, internally solving the problems of scattered data storage and management; it addresses issues of unclear data definitions, non-standard formats, and inconsistent specifications. The output data also has the potential to interact with third parties. Furthermore, depending on the data type and requirements, storage methods can be flexibly selected to adapt to different application scenarios. Whether it's data acquisition, data aggregation, or risk control, compliance, and other business operations, data acquisition and business processing can be standardized, providing convenience for data processing for financial institution staff.

[0086] Example 2:

[0087] Reference Figures 3-5 The second embodiment of the present invention provides a method for normalizing multi-source data, based on the above embodiment 1.

[0088] Based on the above, step S100, based on the big data platform, obtains change record information corresponding to multi-source data from different data sources and writes the change record information into a message queue. This is the data message queue acquisition step, which can also be equivalent to the multi-source data collection step from multiple data sources.

[0089] The above-mentioned data collection from data sources can include two methods and three approaches.

[0090] The two methods are passive data collection and active data collection.

[0091] Three methods are available: passive data collection (database message change log collection and third-party system push collection) and active data collection (timed JOB retrieval collection). Specifically:

[0092] 1. Method for collecting database message change logs in passive collection mode:

[0093] Step S100 includes:

[0094] refer to Figure 3 Step S110: Determine the monitoring method based on the database type of the source database where the multi-source data is located, and monitor the source database using the monitoring method.

[0095] The different data sources mentioned above can be monitored using different methods to obtain data update and change messages. Database types can include, but are not limited to, Oracle and MySQL databases.

[0096] For example, this may include, but is not limited to, the following situations:

[0097] (1) For Oracle databases, you can listen through Oracle Golden Gate (OGG);

[0098] (2) For MySQL databases, binlog can be used for monitoring.

[0099] Step S120: Collect the message change log generated by the source database when data changes occur, and write the message change log into the message queue.

[0100] Therefore, in this method, the corresponding listening method is selected according to the database type of different data sources, and the corresponding message change log is obtained through the listening method and stored in the message queue.

[0101] 2. Third-party system push data collection method:

[0102] refer to Figure 4 Step S100 may further include:

[0103] Step S130: Receive push format information from a third-party system storing the multi-source data, based on the message push format corresponding to the big data platform;

[0104] The aforementioned third-party system can be a data source other than the big data platform. In the corresponding system, the system pushes the relevant data according to the message push format pre-set by the big data platform.

[0105] Step S140: Combine the push format information into a push data set;

[0106] Step S150: Write the push format information in the data set into the corresponding message queue.

[0107] On the big data platform side, push information in the format that conforms to the message push format is actively pushed by the third-party system, and a certain amount of push information in multiple formats is combined to form a push data set, and the set is written into the message queue.

[0108] 3. Timed JOB fetching method:

[0109] refer to Figure 5 Step S100, based on a big data platform, involves acquiring change record information corresponding to multi-source data from different data sources and writing the change record information into a message queue. It may also include:

[0110] Step S160: According to a preset time interval, retrieve the change records from the data source corresponding to the multi-source data;

[0111] Step S170: Write the change record into the message queue.

[0112] Methods 1 and 2 mentioned above are both methods for passively collecting data through push on big data platforms. Method 3 provides an active data collection method, namely, a timed JOB pull data collection method.

[0113] A timed job is a pull method that uses a pre-defined time interval, which can be a point in time or a period of time, to pull change records and write them to the corresponding message queue at each corresponding time interval.

[0114] In this embodiment, data is collected and retrieved using three methods: active collection and passive collection. The passive collection method uses database message change log collection and third-party system push collection, while the active collection method uses timed JOB pull collection. All three methods can coexist in the big data platform and can be flexibly processed according to different data formats, system platforms, specifications, and actual situations, allowing for flexible data collection or reception.

[0115] Example 3:

[0116] Reference Figure 6The third embodiment of the present invention provides a method for normalizing multi-source data. Based on the above embodiment 1, step S200, determining the processor corresponding to the change record information in the message queue, includes:

[0117] Step S210: Based on the server deployed on the big data platform, determine the processor corresponding to the change record information in the message queue through the message type and routing rules of the server.

[0118] As mentioned above, a big data platform can be deployed with one or more servers.

[0119] As described above, after data retrieval and collection using the aforementioned methods, a large number of messages are stored in the message queue, i.e., MQ. At this point, the data is processed by partitioning the data through multiple deployed servers.

[0120] As mentioned above, the router used in the server where the big data platform is located processes the messages.

[0121] At this point, the router determines and locates the corresponding message handler in the following way:

[0122] 1. Message type;

[0123] 2. Custom routing rules.

[0124] Before step S100, which involves acquiring change record information corresponding to multi-source data from different data sources based on a big data platform and writing the change record information into a message queue, the method further includes:

[0125] Step S400: Construct a preset standard big data storage protocol; wherein, the preset standard big data storage protocol includes field names, field types, field formats, and field constraint specifications;

[0126] As mentioned above, the pre-defined standard big data storage protocol established on the big data platform can include, but is not limited to, field names, field types, field formats, and field constraint specifications. This pre-defined standard big data storage protocol can be used for further data standardization processing based on a unified pre-defined standard big data storage protocol.

[0127] For example, a pre-defined standard big data storage protocol may have the following specifications:

[0128]

[0129]

[0130] For example, a pre-defined standard big data storage protocol can also include the following specifications:

[0131]

[0132] In step S200, the processor performs normalization processing on the multi-source data corresponding to the change record information to obtain standard data information, including:

[0133] Step S220: Extract key information from the multi-source data according to a preset standard big data storage protocol;

[0134] A message processor, once determined, can perform various data operations on messages, i.e., multi-source data. These operations may include, but are not limited to, validation, transformation, cleaning, and expansion of multi-source data.

[0135] The key information mentioned above can be information stored in multi-source data that can be extracted and standardized. It can be information with a certain order or an unordered information sequence.

[0136] As mentioned above, during the extraction process, as long as the data contains information such as field names, field types, field formats, and field constraint specifications that conform to the preset standard big data storage protocol, extraction can be performed.

[0137] Step S230: Convert the key information into a standard format corresponding to the field format in the preset standard big data storage protocol to obtain normalized data, and use the normalized data as the standard data information.

[0138] The standard format mentioned above refers to the unified format corresponding to the field format in the preset standard big data storage protocol.

[0139] By formatting and standardizing the extracted key information according to a standard format, standardized data in a unified format is obtained and used as standard data information.

[0140] Furthermore, before extracting key information from the multi-source data according to a preset standard big data storage protocol, step S220 further includes:

[0141] Step S240: Verify the validity of the multi-source data corresponding to the message queue.

[0142] Before extracting key information, a validity verification step is added to ensure more efficient data standardization processing on the big data platform. This step verifies the validity of multi-source data. If the data is valid, the next step of key information collection can proceed; otherwise, no further processing is performed. This indirectly reduces unnecessary processes on the system and improves data processing efficiency.

[0143] Step S230, after converting the key information into a standard format corresponding to the preset standard big data storage protocol to obtain normalized data, and using the normalized data as the standard data information, further includes:

[0144] Step S250: Review and verify the standard data information so that, after the review and verification are passed, the standard data information is stored in the storage medium.

[0145] After obtaining the standardized data, to ensure its accuracy, this embodiment adds a data cleaning step. That is, the obtained standard data information is reviewed and verified.

[0146] In addition, extended steps can be added:

[0147] Define other operations during data processing, such as notifying third-party applications of message changes, issuing warning messages, etc.

[0148] Example 4:

[0149] Reference Figure 7 The fourth embodiment of the present invention provides a method for normalizing multi-source data. Based on the above embodiment 1, step S300, storing the standard data information in a storage medium corresponding to the standard data information, includes:

[0150] Step S310: Obtain the functional category corresponding to the standard data information;

[0151] After the data is standardized, standard data information is obtained. Next, this standard data information needs to be stored. To ensure targeted storage for different situations, functional categories need to be distinguished during storage.

[0152] Function categories can be defined by the functions and requirements of the data.

[0153] Step S320: Determine the target storage medium into which the standard data information is to be stored based on the functional category;

[0154] Step S330: Store the standard data information into the corresponding target storage medium.

[0155] In this embodiment, the processor stores the processed data, i.e., standard data information, in different corresponding storage media according to function and requirements. These different corresponding storage media are called target storage media.

[0156] The target storage medium can be a data format, database, etc., that corresponds to the standard data information functions and requirements.

[0157] For example, it can be divided into the following situations:

[0158] 1. Big data analytics can be accessed using Elasticsearch;

[0159] 2. Data with high concurrency can use Redis;

[0160] 3. High-availability data can use relational databases such as MySQL and Oracle.

[0161] Furthermore, after storage on the corresponding target storage medium, the data within the target storage medium can be further utilized. This may include, but is not limited to, the following:

[0162] 1. Data Query: Provides a unified query interface and standardizes data return;

[0163] 2. Data Analysis: Provides aggregated big data analysis functions.

[0164] In summary, the data storage method adopted in this embodiment allows for flexible selection of storage methods based on different data types and requirements, adapting to various application scenarios. The big data platform hides the complexity of multiple business systems, enabling unified external interaction and improving R&D efficiency. For big data analysis, the search function provides aggregation capabilities, supporting more complex scenarios. At the same time, it supports data customization functions for different scenarios.

[0165] In addition, refer to Figure 8 This embodiment also provides a multi-source data normalization processing device, including:

[0166] The acquisition module 10 is used to acquire change record information corresponding to multi-source data from different data sources based on a big data platform, and write the change record information into a message queue.

[0167] Processing module 20 is used to determine the processor corresponding to the change record information in the message queue, and to perform normalization processing on the multi-source data corresponding to the change record information through the processor to obtain standard data information;

[0168] The storage module 30 is used to store the standard data information in a storage medium corresponding to the standard data information, so that the standard data information in the storage medium can be retrieved through the big data platform.

[0169] Furthermore, this embodiment also provides a multi-source data normalization processing system, including a memory and a processor. The memory stores a multi-source data normalization processing program, and the processor runs the multi-source data normalization processing program to enable the multi-source data normalization processing system to perform the multi-source data normalization processing method as described above.

[0170] In addition, this embodiment also provides a computer-readable storage medium storing a multi-source data normalization processing program, which, when executed by a processor, implements the multi-source data normalization processing method as described above.

[0171] In summary, for reference Figure 9 A schematic diagram illustrating the overall workflow of multi-source data normalization processing methods; Figure 10 This is a schematic diagram illustrating the workflow from router to processor. This embodiment utilizes a big data platform to construct a message queue for change records of multi-source data from different data sources. The processor then standardizes the multi-source data and stores it in the corresponding storage medium. This invention aggregates multi-source data from different systems and data structures, internally solving the problems of scattered data storage and management; it addresses issues of unclear data definitions, non-standard formats, and inconsistent specifications. Simultaneously, the output data has the potential to interact with third parties. Furthermore, depending on the data type and requirements, storage methods can be flexibly selected to adapt to different application scenarios. Whether it's data acquisition, data aggregation, or risk control, compliance, and other business operations, data acquisition and business processing can be standardized, providing convenience for data processing for financial institution staff.

[0172] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention. The above are only preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for normalizing multi-source data, characterized in that, include: Construct a preset standard big data storage protocol; wherein the preset standard big data storage protocol includes field names, field types, field formats, and field constraint specifications; Based on the big data platform, change record information corresponding to multi-source data from different data sources is obtained, and the change record information is written into a message queue; Determine the processor corresponding to the change record information in the message queue, and According to a preset standard big data storage protocol, key information of the multi-source data is extracted; the key information is converted into a standard format corresponding to the field format in the preset standard big data storage protocol to obtain normalized data, and the normalized data is used as standard data information. Obtain the functional category corresponding to the standard data information; determine the target storage medium to which the standard data information should be stored based on the functional category; store the standard data information into the corresponding target storage medium so that the standard data information in the storage medium can be retrieved through the big data platform.

2. The multi-source data normalization processing method as described in claim 1, characterized in that, The process of acquiring change record information corresponding to multi-source data from different data sources based on a big data platform and writing the change record information into a message queue includes: The monitoring method is determined based on the database type of the source database where the multi-source data is located, and the source database is monitored using the monitoring method. Collect message change logs generated by the source database when data changes occur, and write the message change logs into the message queue.

3. The multi-source data normalization processing method as described in claim 1, characterized in that, The process of acquiring change record information corresponding to multi-source data from different data sources based on a big data platform and writing the change record information into a message queue includes: Receive push format information from a third-party system that stores the multi-source data, based on the message push format corresponding to the big data platform; The push format information is combined into a push data set; Write the push format information from the data set into the corresponding message queue.

4. The multi-source data normalization processing method as described in claim 1, characterized in that, The process of acquiring change record information corresponding to multi-source data from different data sources based on a big data platform and writing the change record information into a message queue includes: According to a preset time interval, retrieve change records from the data source corresponding to the multi-source data; Write the change record to the message queue.

5. The multi-source data normalization processing method as described in claim 1, characterized in that, The step of determining the processor corresponding to the change record information in the message queue includes: Based on the servers deployed on the big data platform, the processor corresponding to the change record information in the message queue is determined by the message type and routing rules of the servers.

6. The multi-source data normalization processing method as described in claim 1, characterized in that, Before extracting key information from the multi-source data according to a preset standard big data storage protocol, the process also includes: The validity of the multi-source data corresponding to the message queue is verified. After converting the key information into a standard format corresponding to the preset standard big data storage protocol to obtain normalized data, and using the normalized data as the standard data information, the method further includes: The standard data information is reviewed and verified so that, after the review and verification are passed, the standard data information is stored in the storage medium.

7. A multi-source data normalization processing device, characterized in that, include: The acquisition module is used to construct a preset standard big data storage protocol; wherein the preset standard big data storage protocol includes field names, field types, field formats and field constraint specifications; based on the big data platform, it acquires change record information corresponding to multi-source data from different data sources and writes the change record information into a message queue; The processing module is used to determine the processor corresponding to the change record information in the message queue, and extract the key information of the multi-source data according to the preset standard big data storage protocol; convert the key information into a standard format corresponding to the field format in the preset standard big data storage protocol to obtain normalized data, and use the normalized data as standard data information; The storage module is used to obtain the functional category corresponding to the standard data information; determine the target storage medium to which the standard data information is to be stored according to the functional category; and store the standard data information into the corresponding target storage medium so that the standard data information in the storage medium can be called through the big data platform.

8. A multi-source data normalization processing system, characterized in that, The system includes a memory and a processor. The memory stores a multi-source data normalization processing program, and the processor runs the multi-source data normalization processing program to enable the multi-source data normalization processing system to perform the multi-source data normalization processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-source data normalization processing program, which, when executed by a processor, implements the multi-source data normalization processing method as described in any one of claims 1-6.

Images