A big data comparison method and system in the financial field

Abstract

The present application relates to the technical field of computer, especially relates to a kind of big data comparison method and system in financial field, the method comprises: obtaining first data source data and second data source data, pre-processing by preset data unloading rule;According to data type and content block storage, obtain the preset number of peer block area, assign the same identification to each pair of peer block area;The file block content of first data source data and second data source data with the same identification is cut into field, and key word is extracted;According to preset mapping rule, all key words in the file block of first data source data and second data source data with the same identification are compared, and the corresponding target library table unit is determined;According to preset comparison rule, the data content of corresponding target library table unit is compared;Obtain comparison result.The method is used to solve the problems caused by massive data comparison of new and old systems in system reconstruction process.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a big data comparison method and system in the financial field. Background Art

[0002] Because banking involves money and accounts, data accuracy is paramount, placing high demands on system security and stability. Consequently, banking software systems are characterized by high system complexity and low tolerance for functional upgrades.

[0003] In recent years, with increasing demands for user experience, new business development, and independent research and development of core technologies, the banking industry has undertaken large-scale business process transformation and software system reconstruction and upgrades. For major transformation projects such as system reconstruction and hardware and software replacement, comparing basic data before and after the transformation is essential to ensuring stable system operation and product quality. In actual operations, banking data is characterized by large volumes, inconsistent formats between new and old data, and complex mapping rules. Manually comparing large amounts of data is inefficient within a limited implementation cycle, and item-by-item comparisons are prone to errors. In particular, for scenarios involving massive data comparisons between old and new systems during system reconstruction, a universal process solution is urgently needed given the potential for different data sources, format differences, and complex comparison and mapping rules. Summary of the Invention

[0004] To address the deficiencies of the existing technology, the present invention proposes a big data comparison method and system in the financial field, which is used in scenarios where massive data from old and new systems are compared during system reconstruction, to solve the data comparison difficulties caused by different data sources, different formats, and complex comparison and mapping rules.

[0005] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0006] The first aspect of the present invention discloses a big data comparison method in the financial field, comprising the following steps:

[0007] Obtain data from the first data source and data from the second data source, and pre-process them respectively according to preset data unloading rules;

[0008] The first data source data and the second data source data are stored in blocks according to data type and content, generating a preset number of file blocks of the first data source data and file blocks of the second data source data, wherein the file blocks of the first data source data and the file blocks of the second data source data are matched and equal to each other, and the same identifier is assigned to each pair of equal file blocks of the first data source data and the second data source data;

[0009] Cut the file block contents of the first data source data and the second data source data having the same identifier into fields respectively, and extract keywords;

[0010] According to a preset mapping rule, all keywords in the file blocks of the first data source data and the second data source data having the same identifier are compared to determine the corresponding target library table unit;

[0011] Compare the data contents of the corresponding target library table units according to the preset comparison rules, obtain the comparison results of the first data source data and the second data source data, and save them;

[0012] The comparison results are displayed in a visual window.

[0013] Furthermore, pre-processing is performed using preset data unloading rules to eliminate data format differences and precision differences.

[0014] Furthermore, the preset mapping rule includes: constructing a mapping association word library, setting mapping association words to associate corresponding keywords in the first data source and the second data source.

[0015] Furthermore, comparing the data contents of the corresponding target library table units according to a preset comparison rule includes: comparing the fields of the corresponding target library table units one by one.

[0016] Furthermore, the keywords include one or more of customer number, customer identity information, data amount type and currency symbol.

[0017] Furthermore, the comparison rule configuration file is set to an XML format.

[0018] Furthermore, displaying the comparison results through a visual window includes identifying and marking the difference fields, and clearly displaying them on a browser interface.

[0019] The second aspect of the present invention discloses a big data comparison system in the financial field, comprising:

[0020] A preprocessing module, configured to obtain data from the first data source and data from the second data source, and preprocess them respectively according to preset data unloading rules;

[0021] a segmentation module that stores the first data source data and the second data source data in blocks according to data type and content, generates a preset number of file blocks of the first data source data and file blocks of the second data source data, wherein the file blocks of the first data source data and the file blocks of the second data source data are matched and equivalent to each other, and assigns the same identifier to each pair of equivalent file blocks of the first data source data and the second data source data;

[0022] a processing module, configured to divide the file block contents of the first data source data and the second data source data having the same identifier into fields respectively, and obtain keywords;

[0023] A mapping module, configured to compare all keywords in file blocks of the first data source data and the second data source data having the same identifier according to a preset mapping rule, and determine corresponding target library table units;

[0024] A comparison module is used to compare the data content of the corresponding target library table unit according to a preset comparison rule, obtain the comparison result of the first data source data and the second data source data, and save it;

[0025] The comparison results are displayed in a visual window.

[0026] According to a third aspect of the present invention, a computer-readable storage medium is disclosed, wherein a computer program is stored on the storage medium, and the computer program implements the above method when executed by a processor.

[0027] A fourth aspect of the present invention discloses an electronic device, comprising a processor and a memory;

[0028] The memory is used to store operation instructions;

[0029] The processor is used to execute the above method by calling the operation instruction.

[0030] In a fifth aspect of the present invention, a computer program product is disclosed, comprising a computer program and / or instructions, which implement the steps of the above method when executed by a processor.

[0031] The beneficial effects of the present invention are:

[0032] The method and system for big data comparison in the financial field described in the present invention compares all keywords in file blocks of first data source data and second data source data with the same identifier according to preset mapping rules, and locks the corresponding target library table units; then, according to the preset comparison rules, compares the data contents of the corresponding target library table units; it is possible to complete a full comparison of new and old system data, quickly and accurately capture difference data during the comparison process, realize rapid error correction, solve the large-scale data comparison needs brought about by large-scale transformations such as system reconstruction and software and hardware replacement, and improve personnel efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a big data comparison method in the financial field provided by an embodiment of the present invention.

[0034] Figure 2A schematic diagram of the structure of a big data comparison system in the financial field provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] The first aspect of the present invention relates to a process of steps as follows Figure 1 The big data comparison method in the financial field shown includes the following steps:

[0038] Step S1: Obtain data from a first data source and data from a second data source, and pre-process them respectively according to preset data unloading rules;

[0039] Step S2: The first data source data and the second data source data are stored in blocks according to data type and content, generating a preset number of file blocks of the first data source data and file blocks of the second data source data. The file blocks of the first data source data and the file blocks of the second data source data are matched and equivalent to each other, and the same identifier is assigned to each pair of equivalent file blocks of the first data source data and the second data source data;

[0040] Step S3: Cut the file block contents of the first data source data and the second data source data having the same identifier into fields respectively, and obtain keywords;

[0041] Step S4: According to a preset mapping rule, all keywords in the file blocks of the first data source data and the second data source data having the same identifier are compared to determine the corresponding target library table unit;

[0042] Step S5: Compare the data contents of the corresponding target library table units according to the preset comparison rules, obtain the comparison results of the first data source data and the second data source data, and save them;

[0043] Step S6: Display the comparison results through a visualization window.

[0044] In this embodiment, the first data source data and the second data source data come from different data sources respectively, and the data sources may optionally include mysql, oracle, gaussDB, etc. The first data source data is old system data, and the second data source data is new system data. There are differences in data format, data storage format or precision between the first data source data and the second data source data. Pre-processing is performed respectively through preset data unloading rules, including eliminating data format differences, storage format differences and precision differences.

[0045] In this embodiment, a pre-configured script is used to write the data to be compared into HDFS. HDFS is a mature and highly fault-tolerant distributed system that can provide high-throughput data access and is suitable for large-scale data access.

[0046] For example, take the amount data as an example. The original system data is of MySQL DOUBLE type, and the new system data is of Oracle DECIMAL(5,2). To prevent the MySQL data from being 1.00000000000001 when unloading the data, TRUNCATE(price,2) is used during the data unloading operation. Another example is the remark field used for data source information in the original system data. The original system data is in XML format.

[0047] <site> <name> GOOGLE< / name> <url> <! [CDATA[https: / / www.google.com]]>< / url> < / site> ,

[0048] The new system data is in json format.

[0049] {"site":{"name":"GOOGLE","url":"https: / / www.google.com"}}, you need to compare the site.url in the new and old system data to see if they are consistent, that is, you need to parse the data on both sides and extract the corresponding https: / / www.google.com.

[0050] In this embodiment, the preset mapping rules include: constructing a mapping association word library, and setting mapping association words to associate corresponding keywords in the first data source and the second data source.

[0051] In this embodiment, the keywords preferably include customer number, customer identity information, data amount type, and currency symbol.

[0052] The comparing the data contents of the corresponding target library table units according to the preset comparison rules includes: comparing the fields of the corresponding target library table units one by one.

[0053] In financial systems involving foreign exchange transactions, the old system's currency fields use currency abbreviations, while the new system uses numeric codes. This requires conversion before comparison. For example, the value of the RMB in the old system is CNY, while in the new system it is 156. To compare, the old system's CNY is first mapped to 156 according to a configured mapping table, and then compared to the new system's value.

[0054] For some more complex scenarios, such as customer number conversion, it is impossible to describe the mapping rules in the configuration file by enumeration, so you need to develop the relevant conversion methods yourself to complete the comparison in the program. For example, the customer code length in the old system is 10 digits, and the customer code length in the new system is 12 digits. The extra two leading digits are used for database routing. When comparing, the data is truncated in the new system and only the last 10 digits are compared. For the data comparison in the above scenario, the design of the comparison rules needs to have good scalability and be able to support new business data comparison requirements with minimal changes. Therefore, the comparison rule configuration file adopts the XML format with good readability, strong scalability, and easy Java processing.

[0055] Specifically, the comparison rules are as follows:

[0056] <RECORD filetype_A="split"filetype_B="split"split_character_A="|@|"split_character_B="|@|">

[0057] <item name_A="TradeID"index_A="1"name_B="TradeID"index_B="2"is_key="trye"DateType="string"ifCompare="true" / >

[0058]

[0059] The first line indicates that each line of text in File A and File B is split using the |@| delimiter. The TradeID is the first field after the split in Text A and the second field after the split in Text B. Both fields are of string type. The data with the same TradeID in Text A and B are compared together using the is_key keyword.

[0060] The method for comparing big data in the financial field described in the present invention compares all keywords in file blocks of first data source data and second data source data with the same identifier according to preset mapping rules, and locks the corresponding target library table units; then, according to the preset comparison rules, compares the data contents of the corresponding target library table units; and can complete the full comparison of new and old system data. During the comparison process, the difference data can be quickly and accurately captured to achieve rapid error correction, solve the large-scale data comparison needs brought about by large-scale transformations such as system reconstruction and software and hardware replacement, and improve personnel efficiency.

[0061] Another aspect of the present invention relates to a big data comparison system in the financial field, the structure of which is as follows: Figure 2 Shown, including:

[0062] A preprocessing module, configured to obtain data from the first data source and data from the second data source, and preprocess them respectively according to preset data unloading rules;

[0063] a segmentation module that stores the first data source data and the second data source data in blocks according to data type and content, generates a preset number of file blocks of the first data source data and file blocks of the second data source data, wherein the file blocks of the first data source data and the file blocks of the second data source data are matched and equivalent to each other, and assigns the same identifier to each pair of equivalent file blocks of the first data source data and the second data source data;

[0064] a processing module, configured to divide the file block contents of the first data source data and the second data source data having the same identifier into fields respectively, and obtain keywords;

[0065] A mapping module, configured to compare all keywords in file blocks of the first data source data and the second data source data having the same identifier according to a preset mapping rule, and determine corresponding target library table units;

[0066] A comparison module is used to compare the data content of the corresponding target library table unit according to a preset comparison rule, obtain the comparison result of the first data source data and the second data source data, and save it;

[0067] Display module, which displays the comparison results through a visual window.

[0068] By using this system, the above-mentioned calculation and processing method can be executed and the corresponding technical effects can be achieved.

[0069] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all steps of the method in the above embodiment. The computer-readable storage medium stores a computer program that implements all steps of the method in the above embodiment when executed by a processor.

[0070] An embodiment of the present invention also provides an electronic device for executing the above-mentioned method. As an implementation device of the method, the electronic device has at least a processor and a memory, and in particular, the memory stores the data and related computer programs required for executing the method, and the processor calls the data and programs in the memory to execute all the steps of the implementation method and obtains the corresponding technical effect.

[0071] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges, and the bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and, therefore, will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter can be the same component, namely a transceiver, which provides a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory can be used to store data used by the processor when performing operations.

[0072] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) employed may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device. The memory may be one or more of a cache, flash memory, a hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, and may store the aforementioned data and information. It may also store programs for executing the relevant information, and the processor may execute the programs stored in the memory to implement information storage or processing. The input unit is used to provide input to the processor, and may, for example, be a keypad or touch input device. The power supply is used to provide power to the electronic device. The display is used to display objects such as images and text, and may, for example, be an LCD display. The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, similar to the case of a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone, thereby implementing common telecommunications functions. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to a central processing unit, enabling local recording via the microphone and playback of stored audio via the speaker.

[0073] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or boxes.

[0075] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system that is implemented in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The present invention is described in detail below. ...

[0077] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention are intended to be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A big data comparison method in the financial field, characterized in that: The following steps are involved: Obtain data from the first data source and data from the second data source, and pre-process them respectively according to preset data unloading rules; The first data source data and the second data source data are stored in blocks according to data type and content, generating a preset number of file blocks of the first data source data and file blocks of the second data source data, wherein the file blocks of the first data source data and the file blocks of the second data source data are matched and equal to each other, and the same identifier is assigned to each pair of equal file blocks of the first data source data and the second data source data; Cut the file block contents of the first data source data and the second data source data having the same identifier into fields respectively, and extract keywords; According to a preset mapping rule, all keywords in the file blocks of the first data source data and the second data source data having the same identifier are compared to determine the corresponding target library table unit; Compare the data contents of the corresponding target library table units according to the preset comparison rules, obtain the comparison results of the first data source data and the second data source data, and save them; The comparison results are displayed in a visual window.

2. The method according to claim 1, wherein Pre-processing is performed using preset data unloading rules, including eliminating data format differences and accuracy differences.

3. The method according to claim 1 or 2, wherein: The preset mapping rules include: constructing a mapping association word library, setting mapping association words to associate corresponding keywords in the first data source and the second data source.

4. The method according to claim 1 or 2, wherein: The comparing the data contents of the corresponding target library table units according to the preset comparison rules includes: comparing the fields of the corresponding target library table units one by one.

5. The method according to claim 1, wherein The keywords include one or more of customer number, customer identity information, data amount type and currency symbol.

6. The method according to claim 1, wherein The display of the comparison results through the visualization window includes identifying and marking the difference fields, and clearly displaying them on the browser interface.

7. A big data comparison system in the financial field, characterized by: include: A preprocessing module, configured to obtain data from the first data source and data from the second data source, and preprocess them respectively according to preset data unloading rules; a segmentation module that stores the first data source data and the second data source data in blocks according to data type and content, generates a preset number of file blocks of the first data source data and file blocks of the second data source data, wherein the file blocks of the first data source data and the file blocks of the second data source data are matched and equivalent to each other, and assigns the same identifier to each pair of equivalent file blocks of the first data source data and the second data source data; a processing module, configured to divide the file block contents of the first data source data and the second data source data having the same identifier into fields respectively, and obtain keywords; A mapping module, configured to compare all keywords in file blocks of the first data source data and the second data source data having the same identifier according to a preset mapping rule, and determine corresponding target library table units; A comparison module is used to compare the data content of the corresponding target library table unit according to a preset comparison rule, obtain the comparison result of the first data source data and the second data source data, and save it; Display module, which displays the comparison results through a visual window.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which implements the method according to any one of claims 1 to 6 when executed by a processor.

9. An electronic device, characterized in that: including processor and memory; The memory is used to store operation instructions; The processor is configured to execute the method according to any one of claims 1 to 6 by calling the operation instruction.

10. A computer program product comprising a computer program and / or instructions, characterized in that When the computer program and / or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images