A data processing method and related equipment

Abstract

The present disclosure provides a data processing method and related equipment, which can determine the database and data table for storing target transaction data according to the generation time of the target transaction data and the pre-divided different frequency time intervals, and can store data according to the uneven distribution characteristics of the access frequency and timeliness requirements of the data table in the time dimension, and can adapt to the big data and high-concurrency reading and writing requirements of transaction data tables in actual applications.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method and related equipment. Background Art

[0002] With the continuous development and expansion of banking business, the transaction data generated by related business systems is also increasing. Among them, the problems of large data volume, high concurrency and high reading and writing frequency have always restricted the further development of banking business.

[0003] Currently, the access frequency and timeliness requirements for data tables are unevenly distributed, that is, there are more operations on recent transaction data and higher timeliness requirements, while there are fewer operations on long-term transaction data and lower timeliness requirements. Based on this uneven distribution feature, there is currently no solution that can effectively solve the big data and high-concurrency reading and writing requirements of transaction data tables in practical applications. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data processing method and related devices that overcome the above problems or at least partially solve the above problems. The technical solutions are as follows:

[0005] A data processing method, comprising:

[0006] Obtain target transaction data generated by the target business system;

[0007] When the generation time of the target transaction data is within a preset high-frequency transaction time interval, the target transaction data is stored in a preset online storage database partition table, wherein the preset high-frequency transaction time interval is from the current time to a first preset time, and the first preset time is before the current time;

[0008] When the generation time of the target transaction data is within a preset intermediate frequency transaction time interval, the target transaction data is stored in a preset near-line storage database partition table, and the preset intermediate frequency transaction time interval is from the first preset time to the second preset time, and the second preset time is before the first preset time;

[0009] When the generation time of the target transaction data is within a preset low-frequency transaction time interval, the target transaction data is stored in a preset offline storage database, wherein the preset low-frequency transaction time interval is before the second preset time.

[0010] Optionally, the method further includes:

[0011] When the generation time of the first data stored in the preset online storage database partition table is within the first preset time, the first data is stored from the preset online storage database partition table to the preset near-line storage database partition table.

[0012] Optionally, the method further includes:

[0013] When the generation time of the second data stored in the preset near-line storage database partition table is within the second preset time, the second data is stored from the preset near-line storage database partition table to the preset offline storage database.

[0014] Optionally, the method further includes:

[0015] Before 0:00 on the day, preset pseudo transaction data is inserted into the preset online storage database partition table to trigger a preset day partition script to create a day partition data table corresponding to the day in the online storage database partition table.

[0016] Optionally, the method further includes:

[0017] At the beginning of the month, a monthly partition data table corresponding to the month is created in the preset near-line storage database partition table.

[0018] Optionally, the method further includes:

[0019] When the generation time of the critical data stored in the preset online storage database partition table and the second preset time interval are less than the preset number of days, the critical data is backed up to the preset nearline storage database partition table by a configuration item filtering method.

[0020] Optionally, after backing up the critical data to the preset near-line storage database partition table by the configuration item filtering method, the method further includes:

[0021] The critical data in the preset online storage database partition table is compared with the critical data in the preset near-line storage database partition table at regular intervals. If the critical data are inconsistent, the critical data in the preset near-line storage database partition table is overwritten according to the critical data in the preset online storage database partition table.

[0022] A data processing device comprises: a target transaction data obtaining unit, a first storage unit, a second storage unit and a third storage unit,

[0023] The target transaction data obtaining unit is used to obtain the target transaction data generated by the target business system;

[0024] The first storage unit is used to store the target transaction data in a preset online storage database partition table when the generation time of the target transaction data is within a preset high-frequency transaction time interval, wherein the preset high-frequency transaction time interval is from the current time to a first preset time, and the first preset time is before the current time;

[0025] The second storage unit is used to store the target transaction data in a preset near-line storage database partition table when the generation time of the target transaction data is within a preset intermediate frequency transaction time interval, and the preset intermediate frequency transaction time interval is from the first preset time to the second preset time, and the second preset time is before the first preset time;

[0026] The third storage unit is used to store the target transaction data in a preset offline storage database when the generation time of the target transaction data is within a preset low-frequency transaction time interval, wherein the preset low-frequency transaction time interval is before the second preset time.

[0027] A computer-readable storage medium stores a program, wherein the program, when executed by a processor, implements any of the data processing methods described above.

[0028] An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute any of the data processing methods described above.

[0029] By means of the above technical scheme, a data processing method and related equipment provided by the present disclosure can obtain the target transaction data generated by the target business system; when the generation time of the target transaction data is in a preset high-frequency transaction time interval, the target transaction data is stored in a preset online storage database partition table, wherein the preset high-frequency transaction time interval is from the current time to the first preset time, and the first preset time is before the current time; when the generation time of the target transaction data is in a preset medium-frequency transaction time interval, the target transaction data is stored in a preset near-line storage database partition table, and the preset medium-frequency transaction time interval is from the first preset time to the second preset time, and the second preset time is before the first preset time; when the generation time of the target transaction data is in a preset low-frequency transaction time interval, the target transaction data is stored in a preset offline storage database, wherein the preset low-frequency transaction time interval is before the second preset time. The present disclosure determines the data table for storing the target transaction data according to the different frequency time intervals pre-divided according to the generation time of the target transaction data, and can perform data storage in the time dimension for the uneven distribution characteristics of the access frequency and timeliness requirements of the data table, and can adapt to the large data and high concurrency reading and writing requirements of transaction data tables in practical applications.

[0030] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present disclosure. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0032] Figure 1 A flowchart showing an implementation of a data processing method provided by an embodiment of the present disclosure is shown;

[0033] Figure 2 A schematic diagram showing the structure of a data processing device provided by an embodiment of the present disclosure is shown;

[0034] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0036] like Figure 1 As shown, a flow chart of an implementation of a data processing method provided by an embodiment of the present disclosure may include:

[0037] S100: Obtain target transaction data generated by a target business system.

[0038] The target business system may be a banking business system that generates transaction data during the processing of business transactions.

[0039] S200. When the generation time of the target transaction data is within a preset high-frequency transaction time interval, the target transaction data is stored in a preset online storage database partition table, wherein the preset high-frequency transaction time interval is from the current time to a first preset time, and the first preset time is before the current time.

[0040] The data stored in the preset online storage database is always kept in an "online" state for users to read at will, meeting the computing platform's requirements for data access speed.

[0041] The disclosed embodiment can determine the target transaction data generated in the preset high-frequency transaction time interval as transaction data involving high-frequency query, high-frequency update, and high-frequency write, and store it in a preset online storage database partition table. The preset online storage database partition table can be partitioned by day.

[0042] S300. When the generation time of the target transaction data is within the preset intermediate frequency transaction time interval, the target transaction data is stored in a preset near-line storage database partition table. The preset intermediate frequency transaction time interval is from the first preset time to the second preset time, and the second preset time is before the first preset time.

[0043] The disclosed embodiment can determine the target transaction data generated in the preset intermediate frequency transaction time interval as transaction data involving intermediate frequency query, intermediate frequency update, and intermediate frequency write, and store it in the preset near-line storage database partition table. The preset near-line storage database partition table can be partitioned by month.

[0044] S400. When the generation time of the target transaction data is within a preset low-frequency transaction time interval, the target transaction data is stored in a preset offline storage database, wherein the preset low-frequency transaction time interval is before a second preset time.

[0045] The preset offline storage database may be a non-relational database, for example, a NoSQL offline storage database.

[0046] The disclosed embodiment may determine the target transaction data whose generation time is within a preset low-frequency transaction time interval as transaction data no longer accessed by the user, and store the target transaction data in a preset offline storage database.

[0047] Specifically, after obtaining the target transaction data, the embodiment of the present disclosure can first determine whether the generation time of the target transaction data is within a preset high-frequency transaction time interval. If so, the target transaction data is stored in a preset online storage database partition table. If not, it is determined whether the generation time of the target transaction data is within a preset medium-frequency transaction time interval. If so, the target transaction data is stored in a preset near-line storage database partition table. If not, it is determined that the generation time of the target transaction data is within a preset low-frequency transaction time interval, and the target transaction data is stored in a preset offline storage database.

[0048] Optionally, a preset first number may be spaced between the current time and the first preset time. A preset second number may be spaced between the first preset time and the second preset time. Optionally, the preset first number is greater than 2, and the preset second number is greater than the preset first number.

[0049] Optionally, the preset online storage database partition table, the preset nearline storage database partition table and the preset offline storage database are deployed in different servers. The disclosed embodiment realizes storage separation of transaction data with different access frequencies through partitioning, thereby alleviating the pressure on the server where the preset online storage database partition table is located.

[0050] The present disclosure provides a data processing method, which can obtain target transaction data generated by a target business system; when the generation time of the target transaction data is in a preset high-frequency transaction time interval, the target transaction data is stored in a preset online storage database partition table, wherein the preset high-frequency transaction time interval is from the current time to the first preset time, and the first preset time is before the current time; when the generation time of the target transaction data is in a preset medium-frequency transaction time interval, the target transaction data is stored in a preset near-line storage database partition table, and the preset medium-frequency transaction time interval is from the first preset time to the second preset time, and the second preset time is before the first preset time; when the generation time of the target transaction data is in a preset low-frequency transaction time interval, the target transaction data is stored in a preset offline storage database, wherein the preset low-frequency transaction time interval is before the second preset time. The present disclosure determines the data table where the target transaction data is stored according to the different frequency time intervals pre-divided according to the generation time of the target transaction data, and can perform data storage in the time dimension for the uneven distribution characteristics of the access frequency and timeliness requirements of the data table, and can adapt to the large data and high-concurrency reading and writing requirements of transaction data tables in practical applications.

[0051] Optionally, in an embodiment of the present disclosure, when the generation time of the first data stored in the preset online storage database partition table is within a first preset time, the first data may be stored from the preset online storage database partition table to the preset nearline storage database partition table.

[0052] It can be understood that when the generation time of the first data is within the first preset time, the first data changes from transaction data involving high-frequency query, high-frequency update, and high-frequency write to transaction data involving medium-frequency query, medium-frequency update, and medium-frequency write. Therefore, the embodiment of the present disclosure can transfer the first data in the preset online storage database partition table whose generation time is not within the preset high-frequency transaction time interval to the preset near-line storage database partition table. After the first data is transferred to the preset near-line storage database partition table, the first data in the preset online storage database partition table is deleted.

[0053] Optionally, in an embodiment of the present disclosure, when the generation time of the second data stored in the preset nearline storage database partition table is within a second preset time, the second data may be stored from the preset nearline storage database partition table to the preset offline storage database.

[0054] It is understandable that, when the generation time of the second data is in the second preset event, the second data changes from transaction data involving intermediate frequency query, intermediate frequency update, and intermediate frequency write to transaction data that the user no longer accesses. Therefore, the embodiment of the present disclosure can transfer the second data in the preset near-line storage data partition table whose generation time is not in the preset intermediate frequency transaction time interval to the preset offline storage database. After the second data is transferred to the preset offline storage database, the second data in the preset near-line storage database partition table is deleted.

[0055] Optionally, the disclosed embodiment may insert preset pseudo transaction data into a preset online storage database partition table before 0:00 on the same day to trigger a preset day partition script to create a day partition data table corresponding to the same day in the online storage database partition table.

[0056] The preset pseudo transaction data is false transaction data that complies with the field definition rules of the transaction data table. In practical applications, since transaction data tables mostly use time as a partition field, the disclosed embodiment can create a daily partition data table corresponding to the day by creating pseudo transaction data corresponding to the day. The disclosed embodiment can delete the preset pseudo transaction data after the daily partition data table is stably applied.

[0057] Optionally, the embodiment of the present disclosure may create a monthly partition data table corresponding to the current month in a preset near-line storage database partition table at the beginning of the current month.

[0058] Optionally, in an embodiment of the present disclosure, when the generation time of critical data stored in the preset online storage database partition table and the second preset time interval are less than the preset number of days, the critical data can be backed up to the preset nearline storage database partition table through a configuration item filtering method.

[0059] It is understandable that there is critical data in the transaction data stored in the preset online storage database partition table whose generation time is about to fail to meet the preset high-frequency transaction time interval. The disclosed embodiment can import the critical data in the preset online storage database partition table into the near-line storage database partition table through the configuration item filtering method every day using SQL*LOADER loop, set data overlap windows in the near-line storage database partition table and the online storage database partition table, realize loop backup, and ensure data security.

[0060] Among them, the configuration item filtering method includes configuring the rows and columns to be filtered through scripts or scheduling jobs. Specifically, for the filtering of row data: due to the characteristics of relatively stable historical data and long full-table backup time, the embodiments of the present disclosure adopt an incremental backup method (i.e., backing up part of the row data daily) to back up transaction data tables. For example, in a data table, a certain time field is PAY_DATE. By setting PAY_DATE as the filtering field through a scheduling job and setting the time range (t0 to t1) and rules, the row data where t0 <= PAY_DATE < t1 can be filtered out. For the filtering of column data: when the transaction data table to be backed up is too large, it needs to be split by columns for backup, or when some unimportant field values need to be removed, the column data of the transaction data table can be filtered during backup. Write the fields to be backed up into a script configuration file, set the filtering rules, and only back up the fields in the backup configuration file.

[0061] Optionally, after backing up the critical data to the preset near-line storage database partition table through the configuration item filtering method, the embodiments of the present disclosure can periodically compare the critical data in the preset online storage database partition table with the critical data in the preset near-line storage database partition table. If the critical data is inconsistent, the critical data in the preset near-line storage database partition table is overwritten according to the critical data in the preset online storage database partition table.

[0062] Specifically, the embodiments of the present disclosure can compare the critical data in the preset online storage database partition table with the critical data in the preset near-line storage database partition table daily through a pre-edited batch task.

[0063] The embodiments of the present disclosure perform storage separation, cyclic backup and verification, partition creation and cleaning for transaction data tables with high concurrency of big data, distribute the concurrency pressure to different servers, tables and partitions, reduce the data for rebuilding partition indexes and locks, improve the read and write efficiency of the database, and enhance the user experience.

[0064] Although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order. In certain circumstances, multitasking and parallel processing may be advantageous.

[0065] It should be understood that the various steps recited in the method embodiments of the present disclosure can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0066] Corresponding to the above method embodiments, the embodiments of the present disclosure also provide a data processing device, the structure of which is as Figure 2As shown, it may include: a target transaction data obtaining unit 100 , a first storage unit 200 , a second storage unit 300 and a third storage unit 400 .

[0067] The target transaction data obtaining unit 100 is used to obtain target transaction data generated by a target business system.

[0068] The first storage unit 200 is used to store the target transaction data in a preset online storage database partition table when the generation time of the target transaction data is in a preset high-frequency transaction time interval, wherein the preset high-frequency transaction time interval is from the current time to a first preset time, and the first preset time is before the current time.

[0069] The second storage unit 300 is used to store the target transaction data in a preset near-line storage database partition table when the generation time of the target transaction data is within a preset intermediate frequency transaction time interval, and the preset intermediate frequency transaction time interval is from the first preset time to the second preset time, and the second preset time is before the first preset time.

[0070] The third storage unit 400 is used to store the target transaction data in a preset offline storage database when the generation time of the target transaction data is within a preset low-frequency transaction time interval, wherein the preset low-frequency transaction time interval is before the second preset time.

[0071] Optionally, the data processing device may further include: a fourth storage unit.

[0072] The fourth storage unit is configured to store the first data from the preset online storage database partition table to the preset near-line storage database partition table when the generation time of the first data stored in the preset online storage database partition table is within the first preset time.

[0073] Optionally, the data processing device may further include: a fifth storage unit.

[0074] The fifth storage unit is configured to store the second data from the preset near-line storage database partition table to the preset offline storage database when the generation time of the second data stored in the preset near-line storage database partition table is within the second preset time.

[0075] Optionally, the data processing device may further include: a day partition creation unit.

[0076] The daily partition creation unit is used to insert preset pseudo transaction data into the preset online storage database partition table before 0:00 on the same day, so as to trigger a preset daily partition script to create a daily partition data table corresponding to the same day in the online storage database partition table.

[0077] Optionally, the data processing device may further include: a monthly partition creation unit.

[0078] The monthly partition creation unit is used to create a monthly partition data table corresponding to the current month in the preset near-line storage database partition table at the beginning of the current month.

[0079] Optionally, the data processing device may further include: a critical data backup unit.

[0080] The critical data backup unit is used to back up the critical data to the preset near-line storage database partition table through a configuration item filtering method when the generation time of the critical data stored in the preset online storage database partition table and the second preset time interval are less than the preset number of days.

[0081] Optionally, the data processing device may further include: a critical data comparison unit.

[0082] The critical data comparison unit is used to periodically compare the critical data in the preset online storage database partition table with the critical data in the preset near-line storage database partition table after the critical data backup unit backs up the critical data to the preset near-line storage database partition table through the configuration item filtering method. If the critical data are inconsistent, the critical data in the preset near-line storage database partition table is overwritten according to the critical data in the preset online storage database partition table.

[0083] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0084] The present disclosure provides a data processing device, which can obtain target transaction data generated by a target business system; when the generation time of the target transaction data is in a preset high-frequency transaction time interval, the target transaction data is stored in a preset online storage database partition table, wherein the preset high-frequency transaction time interval is from the current time to the first preset time, and the first preset time is before the current time; when the generation time of the target transaction data is in a preset medium-frequency transaction time interval, the target transaction data is stored in a preset near-line storage database partition table, and the preset medium-frequency transaction time interval is from the first preset time to the second preset time, and the second preset time is before the first preset time; when the generation time of the target transaction data is in a preset low-frequency transaction time interval, the target transaction data is stored in a preset offline storage database, wherein the preset low-frequency transaction time interval is before the second preset time. The present disclosure determines the data table where the target transaction data is stored according to the different frequency time intervals pre-divided according to the generation time of the target transaction data, and can perform data storage in the time dimension for the uneven distribution characteristics of the access frequency and timeliness requirements of the data table, and can adapt to the large data and high-concurrency reading and writing requirements of transaction data tables in practical applications.

[0085] The data processing device includes a processor and a memory. The target transaction data acquisition unit 100, the first storage unit 200, the second storage unit 300 and the third storage unit 400 are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

[0086] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set. By adjusting the kernel parameters, the data table for storing the target transaction data can be determined according to the generation time of the target transaction data and the pre-divided different frequency time intervals. The data can be stored in the time dimension according to the uneven distribution characteristics of the access frequency and timeliness requirements of the data table, and can adapt to the big data and high-concurrency reading and writing requirements of transaction data tables in actual applications.

[0087] An embodiment of the present disclosure provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the data processing method is implemented.

[0088] An embodiment of the present disclosure provides a processor, which is used to run a program, wherein the data processing method is executed when the program is run.

[0089] like Figure 3As shown, the embodiment of the present disclosure provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003; wherein the processor 1001 and the memory 1002 communicate with each other through the bus 1003; the processor 1001 is used to call the program instructions in the memory 1002 to execute the above data processing method. The electronic device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0090] The present disclosure also provides a computer program product, which, when executed on an electronic device, is suitable for executing a program initialized with the steps of the data processing method.

[0091] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] In a typical configuration, an electronic device includes one or more processors (CPU), a memory, and a bus. The electronic device may also include an input / output interface, a network interface, and the like.

[0093] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0094] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0095] In the description of the present disclosure, it should be understood that the terms "up", "down", "front", "back", "left" and "right" etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the positions or elements referred to must have specific directions, be constructed and operate in specific directions. Therefore, they should not be understood as limitations of the present disclosure.

[0096] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0097] It will be appreciated by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] The above are only embodiments of the present disclosure and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A data processing method, characterized in that: include: Obtain target transaction data generated by the target business system; When the generation time of the target transaction data is within a preset high-frequency transaction time interval, the target transaction data is stored in a preset online storage database partition table, wherein the preset high-frequency transaction time interval is from the current time to a first preset time, and the first preset time is before the current time; When the generation time of the target transaction data is within a preset intermediate frequency transaction time interval, the target transaction data is stored in a preset near-line storage database partition table, the preset intermediate frequency transaction time interval is from the first preset time to a second preset time, and the second preset time is before the first preset time; When the generation time of the target transaction data is within a preset low-frequency transaction time interval, the target transaction data is stored in a preset offline storage database, wherein the preset low-frequency transaction time interval is before the second preset time; wherein the current time is separated from the first preset time by a preset first number of days, and the first preset time is separated from the second preset time by a preset second number of days; When the generation time of the first data stored in the preset online storage database partition table is within the first preset time, storing the first data from the preset online storage database partition table to the preset near-line storage database partition table; When the generation time of the second data stored in the preset near-line storage database partition table is within the second preset time, storing the second data from the preset near-line storage database partition table to the preset offline storage database; Before 0:00 on the same day, insert preset pseudo transaction data into the preset online storage database partition table to trigger a preset daily partition script to create a daily partition data table corresponding to the same day in the online storage database partition table, wherein the preset pseudo transaction data is false transaction data that complies with the field definition rules of the transaction data table, and the transaction data table uses time as a partition field, and creates a daily partition data table corresponding to the same day by creating pseudo transaction data corresponding to the same day; When the generation time of the critical data stored in the preset online storage database partition table and the second preset time interval are less than the preset number of days, the configuration item filtering method is used to use SQL*LOADER to cyclically import the critical data into the near-line storage database partition table, and a data overlapping window is set in the near-line storage database partition table and the online storage database partition table to achieve cyclic backup.

2. The method according to claim 1, characterized in that Also includes: At the beginning of the month, a monthly partition data table corresponding to the month is created in the preset near-line storage database partition table.

3. The method according to claim 1, characterized in that After backing up the critical data to the preset near-line storage database partition table through the configuration item filtering method, the method further includes: The critical data in the preset online storage database partition table is compared with the critical data in the preset near-line storage database partition table at regular intervals. If the critical data are inconsistent, the critical data in the preset near-line storage database partition table is overwritten according to the critical data in the preset online storage database partition table.

4. A data processing device, characterized in that: include: a target transaction data obtaining unit, a first storage unit, a second storage unit and a third storage unit, The target transaction data obtaining unit is used to obtain the target transaction data generated by the target business system; The first storage unit is used to store the target transaction data in a preset online storage database partition table when the generation time of the target transaction data is within a preset high-frequency transaction time interval, wherein the preset high-frequency transaction time interval is from the current time to a first preset time, and the first preset time is before the current time; The second storage unit is used to store the target transaction data in a preset near-line storage database partition table when the generation time of the target transaction data is within a preset intermediate frequency transaction time interval, the preset intermediate frequency transaction time interval is from the first preset time to a second preset time, and the second preset time is before the first preset time; The third storage unit is used to store the target transaction data in a preset offline storage database when the generation time of the target transaction data is within a preset low-frequency transaction time interval, wherein the preset low-frequency transaction time interval is before the second preset time; wherein the current time is separated from the first preset time by a preset first number of days, and the first preset time is separated from the second preset time by a preset second number of days; When the generation time of the first data stored in the preset online storage database partition table is within the first preset time, storing the first data from the preset online storage database partition table to the preset near-line storage database partition table; When the generation time of the second data stored in the preset near-line storage database partition table is within the second preset time, storing the second data from the preset near-line storage database partition table to the preset offline storage database; Before 0:00 on the same day, insert preset pseudo transaction data into the preset online storage database partition table to trigger a preset daily partition script to create a daily partition data table corresponding to the same day in the online storage database partition table, wherein the preset pseudo transaction data is false transaction data that complies with the field definition rules of the transaction data table, and the transaction data table uses time as a partition field, and creates a daily partition data table corresponding to the same day by creating pseudo transaction data corresponding to the same day; When the generation time of the critical data stored in the preset online storage database partition table and the second preset time interval are less than the preset number of days, the configuration item filtering method is used to use SQL*LOADER to cyclically import the critical data into the near-line storage database partition table, and a data overlapping window is set in the near-line storage database partition table and the online storage database partition table to achieve cyclic backup.

5. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 3 is implemented.

6. An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein: The processor and the memory communicate with each other via the bus; The processor is used to call the program instructions in the memory to execute the data processing method according to any one of claims 1 to 3.

Images