Data processing method and device, storage medium and electronic equipment

Abstract

The present disclosure relates to a data processing method, device, storage medium and electronic equipment. The data processing method comprises: obtaining a current reading data table and a current writing data table corresponding to a current time node; in response to a data query operation on the current reading data table, reading target data in the current reading data table; in the case where there is new data, storing the new data into the current writing data table to obtain a writing data table after the new data; determining the current reading data table as a writing data table corresponding to a next time node of the current time node, and determining the writing data table after the new data as a reading data table corresponding to the next time node of the current time node. The present disclosure can automatically control the read-write separation, improve the data processing efficiency, avoid the data reading operation of the application while writing data, and solve the problems of high data processing cost and long data processing delay in the whole link.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to data processing methods, apparatus, storage media and electronic devices. Background Technology

[0002] With the development of information technology, data is growing explosively, and business development requires the empowerment of big data, necessitating batch processing of massive amounts of data. Current technologies cannot simultaneously write massive amounts of data into a database while simultaneously reading data from it. Due to the large data volume, read and write times are long, resulting in prolonged read-write interlocking issues, leading to high data processing costs and extended processing times. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, this disclosure provides a data processing method, apparatus, storage medium, and electronic device.

[0004] According to one aspect of this disclosure, a data processing method is provided, comprising:

[0005] Retrieve the currently read data table and the currently written data table corresponding to the current time point;

[0006] In response to a data query operation on the currently read data table, the target data in the currently read data table is read.

[0007] If new data is added, the new data is stored in the currently written data table to obtain the written data table after the new data is added.

[0008] The currently read data table is determined as the write data table corresponding to the next time node of the current time node, and the write data table after adding new data is determined as the read data table corresponding to the next time node of the current time node.

[0009] In some possible implementations, the amount of data currently read from the data table is greater than the amount of data currently written to the data table; the newly added data includes newly added data corresponding to historical time nodes and newly added data corresponding to the previous time node of the current time node; the historical time nodes are those that are chronologically preceding the current time node and adjacent to the previous time node of the current time node.

[0010] The step of storing the newly added data into the currently written data table to obtain the written data table after the addition of new data includes:

[0011] The newly added data of the node corresponding to the historical time node and the newly added data of the node corresponding to the previous time node of the current time node are stored in the current write data table to obtain the write data table after the newly added data. The amount of data in the write data table after the newly added data is greater than the amount of data in the current read data table.

[0012] In some possible implementations, the method further includes:

[0013] At the current time point, the real-time data corresponding to the current time point is stored in the real-time data module;

[0014] In response to a data query operation corresponding to the current time node, real-time data corresponding to the current time node is obtained from the real-time data module.

[0015] In some possible implementations, before obtaining the currently read data table and the currently written data table corresponding to the current time node, the method further includes:

[0016] At the starting time node, create the initial first data table and the initial second data table;

[0017] Obtain the complete historical data of the previous time node of the starting time node;

[0018] Write the historical full data of the previous time node of the starting time node into the initial first data table and the initial second data table to obtain the first data table and the second data table respectively.

[0019] The first data table is determined as the write data table corresponding to the next time node after the starting time node, and the second data table is determined as the read data table corresponding to the next time node after the starting time node.

[0020] In some possible implementations, the amount of data currently read from the data table is equal to the amount of data currently written to the data table; the newly added data includes the node newly added data at the starting time node;

[0021] When new data is added, the new data is stored in the currently written data table to obtain the written data table after the new data is added, including:

[0022] If new data is added, the new data of the node at the starting time node is stored in the current write data table to obtain the write data table after the new data is added; the starting time node is the previous time node of the current time node.

[0023] In some possible implementations, before obtaining the historical full data of the previous time node of the starting time node, the method further includes:

[0024] Obtain the business data table involved in the business operation of the previous time node of the starting time node in the business system;

[0025] The business data table is cleaned to obtain the historical full data of the previous time node of the starting time node.

[0026] In some possible implementations, the method further includes:

[0027] Obtain the historical full data corresponding to two adjacent time points;

[0028] Data mapping processing is performed on the historical full data of the previous time node of the two adjacent time nodes to obtain the first data mapping result;

[0029] Data mapping processing is performed on the historical full data of the next time node after the two adjacent time nodes to obtain the second data mapping result;

[0030] If the first data mapping result is inconsistent with the second data mapping result, the node increment data of the later time node relative to the previous time node is determined.

[0031] In some possible implementations, the historical full data corresponding to each time point includes user data of multiple users, and the user data of each user includes field data corresponding to multiple user fields.

[0032] The step of performing data mapping processing on the historical full data of the previous time node of the two adjacent time nodes to obtain the first data mapping result includes:

[0033] Data concatenation is performed on the field data corresponding to each user field in the historical full data of the previous time node to obtain the first user concatenated data.

[0034] Perform data mapping processing on the concatenated data of the first user to obtain the first data mapping result corresponding to each user;

[0035] The step of performing data mapping processing on the historical full data of the next time node after the two adjacent time nodes to obtain the second data mapping result includes:

[0036] Data concatenation is performed on the field data corresponding to each user field in the historical full data of the next time node to obtain the second user concatenated data.

[0037] The concatenated data of the second user is processed by data mapping to obtain the second data mapping result corresponding to each user;

[0038] The step of determining the node increment data of the later time node relative to the earlier time node when the first data mapping result and the second data mapping result are inconsistent includes:

[0039] If the first data mapping result corresponding to at least one user is inconsistent with the second data mapping result corresponding to the at least one user, the user data of the at least one user at the later time node shall be determined as the node newly added data at the later time node.

[0040] According to a second aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising:

[0041] The data table acquisition module is used to acquire the currently read data table and the currently written data table corresponding to the current time node.

[0042] The data reading module is used to read the target data in the currently read data table in response to a data query operation on the currently read data table;

[0043] The data writing module is used to store the newly added data into the currently written data table when there is new data, so as to obtain the written data table after the new data is added.

[0044] The read / write data table replacement module is used to determine the currently read data table as the write data table corresponding to the next time node of the current time node, and to determine the write data table after adding new data as the read data table corresponding to the next time node of the current time node.

[0045] According to a third aspect of this disclosure, an electronic device is provided, including at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements a data processing method as described in any one of the first aspects by executing the instructions stored in the memory.

[0046] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the data processing method as described in any of the first aspects.

[0047] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0048] Implementing this disclosure will have the following beneficial effects:

[0049] The process involves retrieving the currently read and write data tables corresponding to the current time point. First, the read and write data tables for the current time point are determined. In response to a data query operation on the currently read data table, the target data is read from the table. If new data is added, it is stored in the currently write data table, resulting in a new write data table. This separation of the data table for reading and writing data avoids read / write locks caused by reading and writing data to the same table, improving data processing efficiency. The currently read data table is designated as the write data table for the next time point, and the write data table after adding new data is designated as the read data table for the next time point. Each time point's write data table is different from the write data tables of its adjacent time points to ensure the read data tables are updated, resulting in a more comprehensive data read in the next iteration.

[0050] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0052] Figure 1 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown;

[0053] Figure 2 A flowchart illustrating a real-time data storage method according to an embodiment of the present disclosure is shown;

[0054] Figure 3 A flowchart illustrating a data table initialization method according to an embodiment of the present disclosure is shown.

[0055] Figure 4 A flowchart illustrating a method for determining historical full data according to an embodiment of this disclosure is shown.

[0056] Figure 5 A flowchart illustrating a method for determining incremental node data according to an embodiment of the present disclosure is shown.

[0057] Figure 6 A flowchart illustrating a method for determining a first data mapping result according to an embodiment of the present disclosure is shown.

[0058] Figure 7 A flowchart illustrating a method for determining a second data mapping result according to an embodiment of the present disclosure is shown.

[0059] Figure 8 A flowchart illustrating a data read / write method according to an embodiment of the present disclosure is shown.

[0060] Figure 9 This diagram illustrates the structure of a data processing apparatus according to an embodiment of the present disclosure.

[0061] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0062] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0063] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0064] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0065] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0066] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0067] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0068] The application environment disclosed herein includes a data processing system, which includes risk control applications, a risk control application library, a data warehouse, and a business system.

[0069] Figure 1 This diagram illustrates a flow chart of a data processing method according to an embodiment of the present disclosure, such as... Figure 1 As shown, the above method includes:

[0070] S101. Obtain the currently read data table and the currently written data table corresponding to the current time node;

[0071] Time points are defined as intervals of one day, half a day, or one hour. The currently read data table represents the data table that has only undergone read operations at the current time point, while the currently written data table represents the data table that has only undergone write operations at the current time point.

[0072] If the previous time node is not the starting time node, the current read data table corresponding to the current time node can be the write data table corresponding to the previous time node, and the current write data table corresponding to the current time node can be the read data table corresponding to the previous time node.

[0073] In some embodiments, the risk control application library determines that the current read data table corresponding to the current time node is data table D and the current write data table is data table X. The current read data table D can be used to store the indicator data required for risk control, and the current write data table X can be used to store the indicator data required for risk control.

[0074] S102. In response to a data query operation on the currently read data table, read the target data from the currently read data table;

[0075] When there is a data reading task, the target data is read from the currently read data table. The target data can be all the data in the currently read data table.

[0076] In some embodiments, if the risk control application needs to read indicator data from the risk control application library at the current time point, the indicator data is read from the currently read data table D using a preset interface to obtain the target data.

[0077] S103. If new data is added, store the new data in the current write data table to obtain the write data table after the new data is added.

[0078] The newly added data represents the incremental data relative to the historical full data corresponding to the previous time point, as written to the current data table. If new data exists, it will be written to the current data table.

[0079] In some embodiments, incremental data includes user data that has been updated or added. For example, if user A has at least one updated or added user data field at the current time point compared to the previous time point, the user data corresponding to user A is determined to be newly added data at the current time point.

[0080] In some embodiments, the historical full data corresponding to each time point includes all user data up to each time point, and the user data includes field data corresponding to each user field.

[0081] In some embodiments, at the current time point, if there is new data in the data warehouse that needs to be written to the risk control application library, the new data is written to the currently written data table X, which is staggered from the currently read data table D.

[0082] S104. Determine the currently read data table as the write data table corresponding to the next time node of the current time node, and determine the write data table after adding new data as the read data table corresponding to the next time node of the current time node.

[0083] At the next time point after the current time point, the currently read data table will be treated as the write data table, and the currently written data table will be treated as the read data table. At adjacent time points, the operations on the two data tables will differ.

[0084] In some embodiments, the currently read data table D is used as the write data table for the next time node, and the currently written data table X is used as the read data table for the next time node. That is, at the next time node, in response to a data query operation on the read data table, the indicator data in the read data table X is read, and if there is newly added data, the newly added data is stored in the write data table D. The read and write operations for subsequent time nodes are repeated in this manner.

[0085] The above technical solution uses two data tables to automatically manage read and write separation, avoiding simultaneous data writing and data reading operations by applications, thus solving problems such as high data processing costs and data processing latency in the entire chain.

[0086] In some embodiments, the amount of data currently read from the data table is greater than the amount of data currently written to the data table; the newly added data includes newly added data corresponding to the nodes of historical time nodes and newly added data corresponding to the nodes of the previous time node of the current time node; the historical time node is one that is in chronological order before the current time node and adjacent to the previous time node of the current time node.

[0087] If new data is added, the new data is stored in the currently written data table, resulting in the written data table after the new data is added, including:

[0088] S1031. Store the newly added data of the node corresponding to the historical time node and the newly added data of the node corresponding to the previous time node of the current time node into the current write data table to obtain the write data table after adding data. The amount of data in the write data table after adding data is greater than the amount of data in the current read data table.

[0089] If the current time node is not the next time node after the starting time node, and the amount of data currently read from the data table corresponding to the current time node is greater than the amount of data currently written to the data table corresponding to the current time node, the newly added data includes the newly added data corresponding to the nodes of the two time nodes above the current time node. The two time nodes above the current time node are the previous time node of the current time node and the historical time node. The historical time node is the time node that is before the current time node in time order and is adjacent to the previous time node of the current time node. That is, the historical time node is the time node before the previous time node of the previous time node of the current time node.

[0090] The process of storing newly added data into the current write data table involves storing the newly added data of the nodes corresponding to the historical time nodes and the newly added data of the nodes corresponding to the previous time node into the current write data table, resulting in a write data table with the newly added data. At this point, the data in the write data table with the newly added data includes more newly added data of the nodes corresponding to the previous time node than the data in the currently read data table.

[0091] In some embodiments, if the starting time node is May 4th and the current time node is May 7th, then, if there is new data added to the node on May 5th relative to May 4th and new data added to the node on May 6th relative to May 5th, the new data added to the node on May 5th and the new data added to the node on May 6th are written to the current write data table.

[0092] In some embodiments, if there is no corresponding node with newly added data in the previous time node of the current time node, but there is corresponding node with newly added data in the historical time node, then the node with newly added data corresponding to the historical time node is written into the current write data table to obtain the write data table after adding data. At this time, the amount of data in the write data table after adding data is the same as the amount of data in the current read data table.

[0093] In some embodiments, if there is corresponding new data for the node in the previous time node of the current time node, but no corresponding new data for the node in the historical time node, then the new data for the node corresponding to the previous time node of the current time node is written into the current write data table to obtain the write data table after the new data is added. At this time, the amount of data in the write data table after the new data is added is the same as the amount of data in the current read data table.

[0094] The above technical solution writes the data added two time nodes before the current time node into a current write data table that is different from the current read data table, thereby increasing the integrity of the data read at the next time node.

[0095] Please see Figure 2 In some embodiments, the method further includes:

[0096] S201. At the current time point, store the real-time data corresponding to the current time point into the real-time data module;

[0097] S202. In response to a data query operation corresponding to the current time node, retrieve the real-time data corresponding to the current time node from the real-time data module.

[0098] The real-time data corresponding to the current time node can be the real-time data between the current time node and historical time nodes. Historical time nodes are those that precede the current time node in chronological order and are adjacent to the previous time node. The real-time data corresponding to the current time node is stored in the real-time data module. When reading target data from the current data table, the real-time data corresponding to the current time node is retrieved from the real-time data module. This allows for the deduplication and merging of the target data and real-time data to obtain the full data up to the current time node. The target data can be the full historical data corresponding to a historical time node. The real-time data module can clear historical data at preset time intervals to reduce data volume and improve system efficiency.

[0099] In some embodiments, data is acquired and stored in a real-time data module. If the starting time node is May 4th and the current time node is May 7th, at the current time node, the currently read data table includes the historical full data corresponding to May 5th. The target data is read from the currently read data table, and the real-time data corresponding to the current time node is obtained from the real-time data module, that is, the real-time data from May 5th to the current time node May 7th. Based on the target data and the real-time data, the historical full data corresponding to the current time node can be obtained.

[0100] In some embodiments, the historical full data corresponding to the current time node is all user data involved up to the previous time node, and the user data includes multiple user fields and the field data corresponding to each user field.

[0101] The above technical solution, through a real-time data module, compensates for the incomplete data in the currently read data table at the current time point, ensuring the integrity and real-time nature of the acquired data.

[0102] Please see Figure 3 In some embodiments, before obtaining the currently read data table and the currently written data table corresponding to the current time node, the method further includes:

[0103] S301. At the start time node, create the initial first data table and the initial second data table;

[0104] S302. Obtain the complete historical data of the previous time node of the starting time node;

[0105] S303. Write the historical full data of the previous time node of the starting time node into the initial first data table and the initial second data table to obtain the first data table and the second data table respectively.

[0106] S304. The first data table is determined as the write data table corresponding to the next time node after the start time node, and the second data table is determined as the read data table corresponding to the next time node after the start time node.

[0107] At the starting time node, create an initial first data table and an initial second data table with the same structure. Obtain the full historical data of the previous time node of the starting time node. Write the full historical data corresponding to the previous time node of the starting time node into the initial first data table and the initial second data table to obtain the first data table corresponding to the initial first data table and the second data table corresponding to the initial second data table, thus completing the initialization of the two data tables.

[0108] Write data table that takes either the first data table or the second data table as the next time node after the starting time node, and read data table that takes the other data table from the first data table and the second data table as the next time node after the starting time node.

[0109] In some embodiments, the historical full data of the previous time node of the starting time node is all user data up to the previous time node of the starting time node, and the user data includes multiple user fields and field data of each user field.

[0110] In some embodiments, if the starting time node is May 4th, an initial first data table and an initial second data table are created on May 4th. The historical full data corresponding to May 3rd is written into the initial first data table to obtain the first data table. The historical full data corresponding to May 3rd is written into the initial second data table to obtain the second data table. The first data table is used as the write data table corresponding to May 5th, and the second data table is used as the read data table corresponding to May 5th.

[0111] The above technical solution initializes the read and write data tables at the starting time node by writing all the historical data of the previous time node to both data tables at once. This ensures that only new data needs to be written later, increasing data integrity and reducing the amount of data written later.

[0112] In some embodiments, the amount of data currently read from the data table is equal to the amount of data currently written to the data table; newly added data includes newly added data for nodes at the start time node;

[0113] If new data is added, the new data is stored in the currently written data table, resulting in the written data table after the new data is added, including:

[0114] S1032. In the case of newly added data, store the newly added data of the node at the starting time node into the current write data table to obtain the write data table after the newly added data; the starting time node is the previous time node of the current time node.

[0115] When the current time node is the next time node after the starting time node, the newly added data is the node newly added data of the starting time node. The node newly added data of the starting time node is the incremental data of the historical full data of the starting time node relative to the historical full data of the previous time node of the starting time node. If there is newly added data, the node newly added data of the starting time node will be stored in the currently written data table.

[0116] In some embodiments, data is acquired and stored in the real-time data module. If the starting time node is May 4th and the current time node is May 5th, at the current time node, the currently read data table stores the historical full data corresponding to May 3rd, and the currently written data table stores the historical full data corresponding to May 3rd. The target data is read from the currently read data table. If there is new data for the node on May 4th relative to May 3rd, the new data for the node on May 4th is written to the currently written data table.

[0117] The above technical solution creates and initializes two initial data tables at the starting time node. Both data tables contain the complete historical data of the previous time node of the starting time node. Therefore, it is not necessary to write the newly added data of the previous time node of the starting time node again, which improves the efficiency of data writing while ensuring the data is relatively complete.

[0118] Please see Figure 4 In some embodiments, before obtaining the historical full data of the time point preceding the starting time point, the method further includes:

[0119] S401. Obtain the business data table involved in the business operation of the previous time node of the start time node in the business system;

[0120] S402. Clean the business data table to obtain the historical full data of the previous time node of the starting time node.

[0121] Before obtaining the historical full data of the previous time point of the starting time point, the data involved in the business execution up to the previous time point of the starting time point is synchronized to the data warehouse through a preset automated script. The data warehouse cleans all the data corresponding to the previous time point of the starting time point to obtain the historical full data corresponding to the previous time point of the starting time point, and stores the historical full data of the previous time point of the starting time point into the full data table.

[0122] At the current time point, the full data table includes the historical full data corresponding to each time point before the current time point, and the historical full data corresponding to each time point can be obtained from the full data table.

[0123] The above technical solution stores the historical full data corresponding to each time point in an additional full data table, which facilitates the rapid acquisition of full data.

[0124] Please see Figure 5 In some embodiments, the method further includes:

[0125] S501. Obtain the historical full data corresponding to two adjacent time nodes respectively;

[0126] S502. Perform data mapping processing on the historical full data of the previous time node of two adjacent time nodes to obtain the first data mapping result;

[0127] S503. Perform data mapping processing on the historical full data of the next time node of two adjacent time nodes to obtain the second data mapping result.

[0128] S504. If the first data mapping result and the second data mapping result are inconsistent, determine the node increment data of the later time node relative to the previous time node.

[0129] Retrieve historical full data from two adjacent time points from the full data table, namely the historical full data of the time point preceding and following the two adjacent time points. Join and concatenate the historical full data of the two adjacent time points based on the primary key, and perform encrypted mapping and comparison using a preset encryption algorithm to obtain the node incremental data of the following time point, which is then stored in the incremental database.

[0130] In some embodiments, historical full data for May 5th and historical full data for May 6th are obtained from the full data table. Each field in the historical full data for May 5th and each field in the historical full data for May 6th are associated and mapped based on the primary key. The historical full data for May 5th and the historical full data for May 6th after association and mapping are concatenated with all fields and then encrypted and mapped using the MD5 message digest algorithm. The results are then compared. If the comparison results are inconsistent, the node new data for May 6th is determined and stored in the incremental database.

[0131] In other embodiments, the data mapping processing method also includes a Security Hash Algorithm (SHA) or a Cyclic Redundancy Check (CRC) algorithm, which are not limited here.

[0132] At the current time point, the incremental database includes the incremental data of nodes corresponding to time points before the current time point where new data was added.

[0133] The above technical solution first determines the new data of the node in the next time node based on the full data of two adjacent time nodes, and then writes the new data of the node into the current write data table, thereby reducing the amount of data written to the current write data table and improving operation efficiency.

[0134] In some embodiments, the historical full data corresponding to each time point includes user data of multiple users, and the user data of each user includes field data corresponding to multiple user fields.

[0135] Please see Figure 6 Data mapping is performed on the historical full data of the time point preceding two adjacent time points to obtain the first data mapping result, including:

[0136] S5021. Perform data splicing on the field data corresponding to each user field in the historical full data of the previous time node to obtain the first user spliced ​​data.

[0137] S5022. Perform data mapping processing on the first user's concatenated data to obtain the first data mapping result corresponding to each user;

[0138] Please see Figure 7 The data mapping process is performed on the historical full data of the next time point between two adjacent time points to obtain the second data mapping result, which includes:

[0139] S5031. Perform data splicing on the field data corresponding to each user field in the historical full data of the next time node to obtain the spliced ​​data of the second user.

[0140] S5032. Perform data mapping processing on the second user's spliced ​​data to obtain the second data mapping result corresponding to each user;

[0141] In cases where the first data mapping result and the second data mapping result are inconsistent, the incremental data of the node relative to the previous time node for the later time node is determined, including:

[0142] S5041. If the first data mapping result corresponding to at least one user is inconsistent with the second data mapping result corresponding to at least one user, the user data of at least one user at the later time node shall be determined as the node newly added data at the later time node.

[0143] In some embodiments, the field data corresponding to each user field in the historical full data of two adjacent time nodes are concatenated based on the primary key to obtain the first user concatenated data corresponding to the historical full data of the previous time node and the second user concatenated data corresponding to the historical full data of the next time node. The first user concatenated data and the second user concatenated data are encrypted and mapped using the MD5 algorithm and then compared. If at least one field data corresponding to at least one user is inconsistent with at least one field data corresponding to at least one user, the user data of at least one user in the next time node is determined as the node newly added data in the next time node.

[0144] The above technical solution quickly determines the newly added data of the next time node by comparing the full-field data mapping of two adjacent time nodes.

[0145] In some embodiments, this disclosure can be applied to payment risk control interception scenarios. Setting a set of risk control rules for payment risk control interception scenarios requires the use of user signing indicators. These user signing indicators include multiple related indicators such as user signing ID number, signing mobile phone number, signing bank card, user signing name, user signing occupation, user signing age, and user signing communication address. The data tables involved in these indicators in the business system include: a user basic information table, a merchant basic information table, and a signing details table. Each day, these three data tables are automatically synchronized from the business system database to the data warehouse. The user signing indicators are then cleaned using data warehouse technology (Extract-Transform-Load, ETL). These user signing indicators include the user signing ID number, ... The full data related to multiple indicators such as signed mobile phone number, signed bank card, signed name, signed occupation, signed age, and signed communication address is stored in a full data table. To reduce the bandwidth pressure and write time when writing data from the data warehouse to the risk control application database, the full data of the time node after two adjacent time nodes and the full data of the time node before two adjacent time nodes are linked through the user primary key user_id. After using MD5 to associate multiple fields of multiple risk control indicators, a full-text comparison is performed to obtain the incremental data between two adjacent time nodes, which is then written to the daily incremental data table in the data warehouse.

[0146] Develop an alternating, mutually exclusive task execution device to control the mutually exclusive execution of two read / write tasks every other day. Develop synchronous tasks TASK_A and TASK_B for writing data from the data warehouse to the risk control application library. In the risk control application library, create two initial data tables, A and B, with identical structures. At the initial launch time, initialize and write the historical full data from the previous time point in the full data table into the initial data tables A and B. The risk control application library synchronously performs real-time indicator cleaning based on the real-time data from the three tables in the business system (user basic information table, merchant basic information table, and contract details table) at the two most recent time points. Upon initial launch (the launch time), the historical full data from the previous time point and the real-time data from the two most recent time points (the launch time and the time point before the previous time point) can be merged for query verification.

[0147] Please see Figure 8 Taking a fixed time period of one day as an example, the interval between two adjacent time points is one day. 。At the starting time node, the historical full data (1=a, 2=b, 3=c, 4=d) corresponding to the previous time node is written into two initial data tables, Table A and Table B. At the next time node after the starting time node (i.e., the first day), TASK_B is executed via an alternating mutually exclusive task execution device to obtain the incremental data (2=g, 5=c, 6=f) corresponding to the starting time node in the incremental data table. This incremental data contains user data where the field data of multiple user fields corresponding to the starting time node has changed compared to the field data of multiple user fields in the user data of the previous time node. The incremental data (2=g, 5=c, 6=f) corresponding to the starting time node is written to the write data table B for the first day. The risk control application queries the read data table A corresponding to the next time node after the starting time node through the interface. On the second day, TASK_A is executed through the alternating mutual exclusion task execution device to obtain the incremental data (2=g, 5=c, 6=f) corresponding to the starting time node and the incremental data (3=z, 7=h, 8=i) corresponding to the first day from the incremental data table. The incremental data of the last two days is written to the write data table A corresponding to the second day. The risk control application queries the read data table A corresponding to the next time node after the starting time node through the interface. Retrieve data from data table B; on the third day, execute TASK_B through the alternating mutual exclusion task execution device to obtain the incremental data of the nodes corresponding to the first day (3=z, 7=h, 8=i) and the node incremental data of the second day (9=j, 10=k) in the incremental data table. Write the incremental data of the most recent two days into the write data table B corresponding to the third day. The risk control application queries the write data table A corresponding to the third day through the interface; and so on, using the alternating mutual exclusion execution device, write data to tables A and B alternately and mutually exclusively, staggering the write of data warehouse indicators to the risk control indicator database table and the query of the application interface on the same table to avoid the read-write interlock problem.

[0148] Please see Figure 9 According to a second aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising:

[0149] Data table acquisition module 10 is used to acquire the currently read data table and the currently written data table corresponding to the current time node;

[0150] Data reading module 20 is used to read target data from the currently read data table in response to a data query operation on the currently read data table;

[0151] The data writing module 30 is used to store the newly added data into the current writing data table when there is new data, so as to obtain the writing data table after the new data is added.

[0152] The read / write data table replacement module 40 is used to determine the currently read data table as the write data table corresponding to the next time node of the current time node, and to determine the write data table after adding new data as the read data table corresponding to the next time node of the current time node.

[0153] In some embodiments, the amount of data currently read from the data table is greater than the amount of data currently written to the data table; the newly added data includes newly added data corresponding to the nodes of historical time nodes and newly added data corresponding to the nodes of the previous time node of the current time node; the historical time node is one that is in chronological order before the current time node and adjacent to the previous time node of the current time node.

[0154] Data writing module 30 includes:

[0155] The first node adds data writing unit, which is used to store the node addition data corresponding to the historical time node and the node addition data corresponding to the previous time node of the current time node into the current write data table, so as to obtain the write data table after adding data. The data volume of the write data table after adding data is greater than the data volume of the current read data table.

[0156] In some embodiments, the apparatus further includes:

[0157] The real-time data storage module is used to store the real-time data corresponding to the current time point into the real-time data module at the current time point.

[0158] The real-time data query module is used to retrieve real-time data corresponding to the current time point from the real-time data module in response to a data query operation for the current time point.

[0159] In some embodiments, the apparatus further includes:

[0160] The data table creation module is used to create an initial first data table and an initial second data table at the start time node;

[0161] The first historical full data acquisition module is used to acquire the historical full data of the previous time node of the starting time node;

[0162] The historical full data writing module is used to write the historical full data of the previous time node of the starting time node into the initial first data table and the initial second data table, respectively, to obtain the first data table and the second data table.

[0163] The data table update module is used to determine the first data table as the write data table corresponding to the next time node after the start time node, and to determine the second data table as the read data table corresponding to the next time node after the start time node.

[0164] In some embodiments, the amount of data currently read from the data table is equal to the amount of data currently written to the data table; newly added data includes newly added data for nodes at the start time node;

[0165] Data writing module 30 includes:

[0166] The second node adds a data writing unit, which is used to store the node's new data from the starting time node into the current write data table when new data is added, thus obtaining the write data table after the new data is added; the starting time node is the previous time node of the current time node.

[0167] In some embodiments, the apparatus further includes:

[0168] The business data table acquisition module is used to acquire the business data table involved in the business operation of the previous time node of the start time node in the business system.

[0169] The data cleaning module is used to clean the business data tables to obtain the historical full data of the previous time point of the starting time point.

[0170] In some embodiments, the apparatus further includes:

[0171] The second historical full data acquisition module is used to acquire the historical full data corresponding to two adjacent time nodes respectively;

[0172] The first mapping module is used to perform data mapping processing on the historical full data of the previous time node of two adjacent time nodes to obtain the first data mapping result.

[0173] The second mapping module is used to perform data mapping processing on the historical full data of the next time node between two adjacent time nodes to obtain the second data mapping result.

[0174] The node incremental data determination module is used to determine the node incremental data of the later time node relative to the previous time node when the first data mapping result and the second data mapping result are inconsistent.

[0175] In some embodiments, the historical full data corresponding to each time point includes user data of multiple users, and the user data of each user includes field data corresponding to multiple user fields.

[0176] The first mapping module includes:

[0177] The first splicing unit is used to splice the field data corresponding to each user field in the historical full data of the previous time node to obtain the first user spliced ​​data.

[0178] The first mapping result determination unit is used to perform data mapping processing on the first user's spliced ​​data to obtain the first data mapping result corresponding to each user;

[0179] The second mapping module includes:

[0180] The second splicing unit is used to splice the field data corresponding to each user field in the historical full data of the next time node to obtain the second user spliced ​​data.

[0181] The second mapping result determination unit is used to perform data mapping processing on the second user's spliced ​​data to obtain the second data mapping result corresponding to each user.

[0182] The node incremental data determination module includes:

[0183] The mapping result comparison unit is used to determine the user data of at least one user at a later time node as newly added node data at the later time node when the first data mapping result corresponding to at least one user is inconsistent with the second data mapping result corresponding to at least one user.

[0184] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0185] This application provides a data processing device, which can be a terminal or a server. The data processing device includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement the data processing method provided in the above method embodiments.

[0186] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0187] The methods and embodiments provided in this application can be executed in electronic devices such as mobile terminals, computer terminals, servers, or similar computing devices. Figure 10This is a hardware structure block diagram of an electronic device for a data processing method provided in an embodiment of this application. For example... Figure 10 As shown, the electronic device 900 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 910 (CPUs 910 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 930 for storing data, and one or more storage media 920 (e.g., one or more mass storage devices) for storing application programs 923 or data 922. The memory 930 and storage media 920 may be temporary or persistent storage. The program stored in the storage media 920 may include one or more modules, each module may include a series of instruction operations on the electronic device. Furthermore, the CPU 910 may be configured to communicate with the storage media 920 and execute the series of instruction operations in the storage media 920 on the electronic device 900. Electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input / output interfaces 940, and / or one or more operating systems 921, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0188] The input / output interface 940 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 900. In one example, the input / output interface 940 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 940 may be a radio frequency (RF) module for wireless communication with the Internet.

[0189] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device 900 may also include... Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown.

[0190] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a data processing method in the method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method provided in the above method embodiment.

[0191] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0192] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0193] As can be seen from the embodiments of the data processing method, apparatus, device, terminal, server, storage medium, or computer program provided in this application, this application obtains the currently read data table and the currently written data table corresponding to the current time node. First, the read data table and the write data table corresponding to the current time node are determined. In response to a data query operation on the currently read data table, the target data in the currently read data table is read. If new data exists, the new data is stored in the currently written data table, resulting in a write data table after the new data is added. The data table for reading data and the data table for writing data are separated to avoid read-write interlocks caused by reading and writing data to the same data table, thereby improving data processing efficiency. The currently read data table is determined as the write data table corresponding to the next time node of the current time node, and the write data table after the new data is added is determined as the read data table corresponding to the next time node of the current time node. The write data table corresponding to each time node is different from the write data table corresponding to its adjacent time nodes to ensure that the data in the read data table is updated, making the amount of data read in the next iteration more comprehensive.

[0194] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0195] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device, equipment, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0196] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0197] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Retrieve the currently read data table and the currently written data table corresponding to the current time point; In response to a data query operation on the currently read data table, the target data in the currently read data table is read. If new data is added, the new data is stored in the currently written data table to obtain the written data table after the new data is added. The currently read data table is determined as the write data table corresponding to the next time node of the current time node, and the write data table after adding new data is determined as the read data table corresponding to the next time node of the current time node. At the current time node, the real-time data corresponding to the current time node is stored in the real-time data module. The real-time data corresponding to the current time node is the real-time data between the current time node and the historical time node of the current time. The historical time node is the time sequence before the current time node and adjacent to the previous time node of the current time node. In response to a data query operation corresponding to the current time node, real-time data corresponding to the current time node is obtained from the real-time data module; Get the historical full data corresponding to two adjacent time nodes. The historical full data corresponding to each time node includes user data of multiple users. The user data of each user includes field data corresponding to multiple user fields. Data concatenation is performed on the field data corresponding to each user field in the historical full data of the previous time node to obtain the first user concatenated data. Perform data mapping processing on the concatenated data of the first user to obtain the first data mapping result corresponding to each user; Data concatenation is performed on the field data corresponding to each user field in the historical full data of the next time node to obtain the second user concatenated data. The concatenated data of the second user is processed by data mapping to obtain the second data mapping result corresponding to each user; If the first data mapping result corresponding to at least one user is inconsistent with the second data mapping result corresponding to the at least one user, the user data of the at least one user at the later time node shall be determined as the node newly added data at the later time node.

2. The method according to claim 1, characterized in that, The amount of data currently read from the data table is greater than the amount of data currently written to the data table; the newly added data includes newly added data corresponding to historical time nodes and newly added data corresponding to the previous time node of the current time node; the historical time nodes are those that are chronologically preceding the current time node and adjacent to the previous time node of the current time node. The step of storing newly added data into the currently written data table to obtain a new written data table includes: The newly added data of the node corresponding to the historical time node and the newly added data of the node corresponding to the previous time node of the current time node are stored in the current write data table to obtain the write data table after the newly added data. The amount of data in the write data table after the newly added data is greater than the amount of data in the current read data table.

3. The method according to claim 1, characterized in that, Before obtaining the currently read data table and the currently written data table corresponding to the current time node, the method further includes: At the starting time node, create the initial first data table and the initial second data table; Obtain the complete historical data of the previous time node of the starting time node; Write the historical full data of the previous time node of the starting time node into the initial first data table and the initial second data table to obtain the first data table and the second data table respectively. The first data table is determined as the write data table corresponding to the next time node after the starting time node, and the second data table is determined as the read data table corresponding to the next time node after the starting time node.

4. The method according to claim 3, characterized in that, The amount of data currently read from the data table is equal to the amount of data currently written to the data table; the newly added data includes the newly added data of the node at the starting time node; When new data is added, the new data is stored in the currently written data table to obtain the written data table after the new data is added, including: If new data is added, the new data of the node at the starting time node is stored in the current write data table to obtain the write data table after the new data is added; the starting time node is the previous time node of the current time node.

5. The method according to claim 3, characterized in that, Before obtaining the historical full data of the previous time node of the starting time node, the method further includes: Obtain the business data table involved in the business operation of the previous time node of the starting time node in the business system; The business data table is cleaned to obtain the historical full data of the previous time node of the starting time node.

6. A data processing apparatus, characterized in that, The device includes: The data table acquisition module is used to acquire the currently read data table and the currently written data table corresponding to the current time node. The data reading module is used to read the target data in the currently read data table in response to a data query operation on the currently read data table; The data writing module is used to store the newly added data into the currently written data table when there is new data, so as to obtain the written data table after the new data is added. The read / write data table replacement module is used to determine the currently read data table as the write data table corresponding to the next time node of the current time node, and to determine the write data table after adding new data as the read data table corresponding to the next time node of the current time node. The real-time data storage module is used to store the real-time data corresponding to the current time node into the real-time data module at the current time node. The real-time data corresponding to the current time node is the real-time data between the current time node and the historical time node of the current time. The historical time node is the time node that is before the current time node and adjacent to the previous time node of the current time node. The data query module is used to retrieve real-time data corresponding to the current time node from the real-time data module in response to a data query operation for the current time node. The second historical full data acquisition module is used to acquire the historical full data corresponding to two adjacent time nodes. The historical full data corresponding to each time node includes user data of multiple users, and the user data of each user includes field data corresponding to multiple user fields. The first splicing module is used to splice the field data corresponding to each user field in the historical full data of the previous time node to obtain the first user spliced ​​data. Perform data mapping processing on the concatenated data of the first user to obtain the first data mapping result corresponding to each user; The second splicing module is used to splice the field data corresponding to each user field in the historical full data of the next time node to obtain the second user spliced ​​data. The concatenated data of the second user is processed by data mapping to obtain the second data mapping result corresponding to each user; The mapping result comparison module is used to determine the user data of the at least one user at the later time node as newly added node data at the later time node when the first data mapping result corresponding to the at least one user is inconsistent with the second data mapping result corresponding to the at least one user.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1-5.

8. An electronic device, characterized in that, The system includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the data processing method as described in any one of claims 1-5 by executing the instructions stored in the memory.

Images